Community Catchup #6

The sixth weekly community catchup focused on the next layer of agent work: not just building demos, but packaging agents so they can run safely inside businesses, be evaluated, and connect to real systems without creating a security mess.

Highlights

Adam anchored the session with a demo of Claude managed agents — a way to define agents, environments, tools, memory, sessions, permissions, files, and MCP access in a more controlled runtime. The group immediately connected it to the same questions that keep coming up in production: where does data live, who approves risky actions, how are credentials handled, and how do you know the output is good enough?

Managed Agents Start to Look Like Infrastructure

Adam’s demo moved from simple to serious quickly. A basic agent listed files. Another called a custom weather tool. Then came PDF and spreadsheet tools, workspace memory mounted as markdown files, file outputs through Anthropic’s storage API, and a remote MCP server connected through a credential vault.

The most concrete example was a supermarket price lookup: the agent used a custom skill and remote MCP access to search products, compare prices, and return a useful answer. More important than the shopping example was the shape of the stack: versioned agents, known environments, explicit tool permissions, scoped network access, credential vaults, and human approval before outbound side effects.

Chris on Secure Citizen Development

Chris shared one of the most production-minded patterns so far: a citizen-developer platform for giving staff AI power without handing them unlimited access to internal systems. The starting point was identity, not prompts. Each agent has its own Entra identity, uses existing SSO and federation, and runs inside a harness with public endpoints blocked, cost caps, review gates, and controlled outbound access.

Credentials are not sitting inside the agent. MCP services fetch secrets from Azure Key Vault, everything runs stateless, changes go through protected branches, and higher-risk changes require security review. The first daily workflow already summarizes calendar, email, Slack, department activity, Jira, Confluence, and security signals into a morning briefing.

Security Is Layers of Swiss Cheese

William raised prompt injection directly, and the answer was refreshingly pragmatic: there is no perfect fix. The group talked through layered defenses — evals, regression tests, Semgrep-style checks, proxy guardrails, PII filtering, credential vaults, network restrictions, review gates, and user training before broader rollout.

Rob mentioned proxy patterns like Bifrost for guardrails, token rotation, and filtering before requests hit large models. The consensus was that technical controls matter, but education is another layer. If staff are going to operate powerful agents, they need to understand the risks in plain language.

Evals, Determinism, and the Cost of Confidence

The testing conversation was one of the most useful parts of the session. Jake described the eval harness around Tucky, his travel itinerary app: 50 example inbound emails, expected structured outputs, and model comparisons. It works, but improving the prompt can cost real money in tokens.

Steve framed the broader decision well: sometimes the agent should write deterministic code and then get out of the way. For common booking emails, fixed templates make sense. For PDFs, phone numbers, and known document layouts, traditional parsing, regular expressions, and older computer vision techniques still have a place. The agent does not need to be the runtime for every step.

The Data Locked in PDFs and Legacy Systems

Adrian closed with a pair of deep technical demos. First, a PDF processing pipeline that treats PDF extraction as orchestration: multiple extraction methods, OCR, page-by-page LLM passes, stitching references back to their source content, and careful handling of tables, images, headers, and scanned pages.

Then he showed a database profiling and warehouse-generation tool. It inspects raw tables, calculates cardinality and overlap, infers likely relationships, proposes a Kimball-style model, and generates DBT. The goal is to unlock messy legacy systems where table and column names are meaningless, but the payload and relationships still reveal the underlying business model.

Where This Is Heading

By the end, the pattern was clear: agents are becoming infrastructure. The interesting work is no longer just “can it do the task?” It is whether the agent has the right memory, the right tools, the right permissions, the right review gates, the right evals, and the right humans in the loop.