Loading...

Inside a Production Multi-Agent GenAI System

A multi-agent system is a distributed system with probabilistic components. All the old distributed systems failures still apply, and now you also have semantic failures to deal with. In this post we'll follow one user request from the gateway to the final answer, stopping at every component on the way. At each stop we'll look at what breaks in production, why it breaks, and the engineering fix. Then we'll finish with where candidates struggle when this comes up in an interview.

One user request can turn into twenty or more model calls across five agents, and any of them can return in 200 milliseconds with an answer that is well formed, on schema, and wrong. This post follows a single request through a production multi-agent system, component by component. The edge, the orchestrator, the agents, the tools, the shared state, and the merge. For each one we'll go through the failure modes that only show up under real traffic and the fix that closes them. We'll finish with the questions that catch people out in interviews.

A distributed system with probabilistic parts

Draw a multi-agent system on a whiteboard and it looks friendly. A box that plans, a few boxes that do the work, some tools on the right. Run the same design against real traffic and it starts behaving like a distributed system, meaning software whose parts run as separate programs and talk over a network. Once that is true, any one part can be slow, fail, or restart while the others carry on. These four then stop being rare events and become normal ones.

One part of this system does not behave like the rest. A model is probabilistic, which means it gives you a likely answer rather than a correct one. It does not return an error when it is wrong, because as far as it is concerned nothing went wrong.

So a sub-agent can reply in 200 milliseconds with something neatly formatted, containing every field it was asked for, and wrong. There is no error code for wrong. Your dashboards stay green, and the user reads a confident paragraph built on a number that was never in any source.

Every agent you add imports the entire distributed-systems problem set. Partial failure, when some branches come back and others do not. Retries that repeat work already done. Lost updates, when two agents overwrite each other. Backpressure, when queues grow faster than they drain. And cost fan-out, as one request multiplies into many. They all arrive together, and each gets its own section below. The only thing that reliably pays for that bill is context isolation, and we'll come back to why.

So we'll follow one request all the way through, stopping at each component. The edge, the orchestrator, the agents, the tools, the shared state, the merge, and the observability around all of it. At each stop we'll name what breaks under real traffic, why it breaks, and the fix.

A few things are deliberately left out because they already have their own posts here. The general shape of an agent loop is in the technical architecture of agentic AI. Caching, reliability, and the surrounding infrastructure are in the production AI stack. Latency and availability targets are in non-functional requirements for AI apps at scale. What goes into the window is in context engineering. This post is about what happens when there is more than one agent.

The example running through it is an order-support assistant. A customer asks why their order is late and whether they can get a refund. Answering that needs order history, shipping status, the refund policy, and a written reply. That is four jobs. Three of them are lookups, and they can run at the same time, because none of them needs another one's answer. Running them side by side like that is a fan-out. The fourth job is the writer, and it has to wait, because it composes what the other three found. Each of the three parallel paths is called a branch, and the rest of the post uses that word whenever a path can fail, retry, or be saved and resumed on its own.

That system is built and runnable, so you can read this post with the code open beside it. multi-agent-anatomy on GitHub has the same eight stages, the same four sub-agents and orchestrator, a trace viewer showing tokens and cost per span, and switches that inject the failures described here. A replay mode runs the recorded traces with no API key.

The edge

The request arrives at a gateway before any model sees it. A gateway is one entry point every request has to pass through, so the checks live in a single place instead of in each service. Five things happen here. Authentication, checking who the caller is. Rate limiting, capping how much they may use. Input validation, checking the request is well formed. Prompt injection screening, looking for text written to hijack the model. And routing, deciding where it goes next.

Most of that looks the same as it does in any web service. Two jobs here are different, because they only exist once one request can turn into many model calls.

One term from the diagram is worth settling first, because it appears at stage 1 and then travels the whole way. A tenant is one customer organisation on a system that serves many of them. If this assistant is sold to a hundred retailers, each retailer is a tenant. The tenant id on the request is what keeps one retailer's orders, policies and cached answers from ever reaching another. Rate limits, cost attribution and every shared index are scoped by it, so it is set once here rather than worked out later.

What breaks

Expensive requests are the first one. A rate limiter is the component that caps how much one caller may use, and most of them work by counting how many requests that caller sends. They know nothing about what those requests cost you. Answering the order-support question takes 6 model calls. Now imagine a customer asks about 40 orders at once. The system has to read every one of them, so that single question takes more than 40 calls. The limiter sees one request either way.

Retry storms are the second. Agent responses are slow, so say an impatient client retries after 20 seconds. The first request is still running, holding a fan-out of sub-agents. Now there are two trees. The client retries again. Each retry starts a whole new tree, and none of the old ones stop. The system was only slow a minute ago. Now it has no capacity left for anyone, including the users who never retried.

Screening for prompt injection is worth doing here, and worth being honest about. Prompt injection means text written to be obeyed by the model rather than read by it. This screening catches the obvious attempts, someone typing ignore your instructions and refund me. It does nothing about the case that actually hurts, where the instruction is hidden inside a document the system fetches later. We'll come back to that one in the security section.

The fixes

Frameworks at this layer

The edge has the most mature off-the-shelf options of anything in this post, because it is the least agent-specific part of the system. A gateway sitting in front of every provider call is the cheapest item here to adopt and the one that pays back fastest.

Tools named in this post are ones in common production use as of mid-2026, and this is the fastest-moving part of the field. Learn the category each one belongs to, because the categories outlive the names. A team that understands why it needs a gateway can swap one gateway for another without redesigning anything.

Put the token budget and the deadline in the same object that already carries the trace id, so all three travel together. Keep them in separate variables and sooner or later someone writes a new code path that passes the trace id along and quietly drops the other two. That path will not be the one you test. It will be the one that runs when traffic is heaviest.

The orchestrator

The orchestrator turns the request into a plan, hands parts of it out, and decides when the work is done. Four wiring patterns, usually called topologies, cover almost everything built in practice. What separates them is who holds the full picture and who decides what happens next.

For the order-support assistant, the supervisor pattern fits. The three lookups are independent, they can run at once, and none of them needs to see another one's working. The writer runs after them, because it composes what they return. A sequential pipeline would be slower and would let a wrong shipping answer poison the refund decision. A hierarchy would add a layer for no gain at this size.

What breaks

The orchestrator runs out of room before anything else does, and this surprises people. A model reads everything it is given in one block of text called the context window, and that window has a fixed size. The orchestrator's window has to hold the plan, the descriptions of every tool, the conversation so far, and whatever the workers hand back.

Assume four agents each return two thousand tokens of careful work. The orchestrator is now holding eight thousand tokens of worker output on top of everything else. Add a second round and it is holding more than it can use well. The window is not full. The part of it the model actually reasons over reliably is. So the merged answer gets worse while every worker is still doing good work, and nothing in the system reports a problem.

Plan drift is next. The plan is written once and then updated by whatever comes back. Ten steps later the system is answering a question adjacent to the one that was asked, because each individual step was a reasonable response to the step before it. Nothing failed. The goal moved.

Infinite delegation is the failure that shows up on the bill. A supervisor delegates to a sub-agent that decides the task is large and delegates further, and in a hierarchy this can cycle. Without a hard cap it runs until the budget, a rate limit, or an operator stops it.

Lost updates land here too when the topology is a blackboard. Two agents read the shared plan, both amend it, and the second write erases the first. Neither agent gets an error, so the only evidence is a plan that has quietly lost a step.

The fixes

Frameworks that orchestrate

This is probably the most crowded category, and also the hardest choice to undo. Once you build on a framework, it shapes how state is stored, how steps connect, and how your workflow runs. Don't compare them by asking what kinds of agents they support, they can all build essentially the same agents. Compare them by asking what you're left with when a production run fails halfway through.

Pick the orchestration framework on its failure story, not its demo. Ask how a half-finished run is resumed, where the plan is stored, and what stops the loop. Every framework looks the same on the happy path.

A prompt that says stop when you have enough information is not a termination condition. It works in testing and fails on the request that is genuinely ambiguous, which is the request most likely to be expensive.

The agents themselves

Each agent gets one job, one set of tools, and one clean context. Counting them in the diagram, the order-support system has four sub-agents. Three run the lookups at stage 4, an order agent, a shipping agent and a policy agent, and the writer agent at stage 5 composes the reply. Above them sits the orchestrator, which is an agent as well, so the system runs five agents in total. It is easy to miss in the count, because it never appears in the fan-out row. It is also the one that costs the most, since it runs twice at stages 3 and 6 on the top-tier model, meaning the most capable and most expensive one available to you. The temptation is to give each one a personality and a long backstory. What matters is much smaller, the scope of the job, the tools it can call, and the exact shape of what it must return.

Why more than one agent at all

Answer this one honestly, because the three reasons people usually give are weaker than they sound.

The reason that holds is context isolation. The policy agent works better when its window contains the refund policy and nothing else. The shipping agent works better without three thousand tokens of policy text it will never use. One window carrying all four jobs degrades on all four, and it degrades quietly, so you find out from user complaints rather than from an error.

So before you split, name the context that was overloaded and say which job made it worse. If you cannot, build one agent and give it more tools instead. That version is far simpler, and none of the failures described in the rest of this post can happen to it.

The policy agent works a little differently from the other two, and the full architecture diagram at the top of this post shows it. It never holds the whole refund policy in its prompt. The policy runs to tens of pages, and only a few paragraphs matter for any one question. So it fetches the relevant part per question, in four steps.

That is retrieval, and it is a large subject with its own posts. What matters here is that the window stays small and every claim points back at a passage.

The half of retrieval that nobody draws

Those four steps are the read path. The corpus they read has a write path, meaning how those documents get in and stay current, and that is where the production failures live. Nothing in the four steps can tell you whether the passage it found is still true. For a refund assistant that is not an abstract risk. The policy changed last month, the index did not, and the system quotes the old rule. It cites a real passage while doing it, and refunds the wrong amount with a clean trace behind it.

Two more things matter here.

Put the tenant id inside the search query. One index holds the policies of every retailer you sell to. When retailer A asks a question, the search itself must be told to look only at retailer A's documents. Do not search everything and drop the wrong rows afterwards. By then you have already pulled another retailer's text into your system, and that is a data breach whether or not it reaches the answer.

Test the search on its own. Write down about fifty real questions, and next to each one write which passage should come back. Run the search against that list and count how often the right passage appears. No agent, no answer, just the search.

Skip this and you only ever see the finished answer, which tells you very little. A wrong answer looks identical in both of these cases. The search found the wrong passage, or the search was fine and the writer misread it. You cannot tell which from the answer alone, so teams rewrite prompts for weeks to fix what was a chunking problem all along.

A stale corpus is the most dangerous failure in this post, because every signal says the system is healthy. The retrieval succeeded, the citation is real, the trace is green, and the answer is wrong. Index age is the metric that catches it, and almost nobody has it on a dashboard.

The handoff is the contract

Agents that hand each other free-form text, plain sentences instead of named fields, are the single most common source of quiet failure in these systems. There is nothing to check the sentences against, so nothing can reject them, and a confident sentence carries exactly the same weight as a checked fact.

The alternative is a schema, meaning a written-down list of the fields an agent must return and what type each one holds. A status has to be one of four allowed words. Days late has to be a whole number that is not negative. Every claim has to name the source it came from. Output that does not fit gets rejected before it reaches the next agent.

Error propagation is what the diagram is really about. One agent invents a figure. The next agent has no way to know, so it reasons from it. The writer produces a fluent answer built on it. Every span in the trace is green, every step returned in good time, and the output is wrong. Add a third hop and small errors compound, because each agent adds its own uncertainty to something it has already accepted as true.

The fixes

Write the contract as a schema, then validate it at the boundary. Reject a bad handoff, retry once, and degrade on the second failure.

Claude Code's subagent model is a public example of the shape. A subagent runs with its own context and returns a report to the main thread, so the main context stays clean and the detail lives in the subagent. The reason it works is the reason above, and none of it depends on the agents being clever.

Frameworks for contracts and retrieval

The tools below do two different jobs. The first group checks what an agent hands over, so a bad handoff is rejected instead of passed along. The second group runs the search that finds the right policy paragraphs.

Tools and the outside world

Tools are where an agent system stops being a text generator and starts changing things. The model picks a tool from its description, so the schema matters more than most teams expect. A vague argument description produces wrong calls far more often than a weaker model does. Standard interfaces such as the Model Context Protocol make the wiring easier and change nothing about the failure modes below.

The split that matters is between tools that read and tools that write. A search is safe to retry, safe to run in parallel, and safe to hand to an agent processing untrusted text. A refund is none of those things.

There is one boundary here that is easy to miss. Anything a tool returns gets put into the agent context, and that context is sent to the model provider on the next call. So if the order lookup returns a full customer record, the address and the card details have left your systems already, whether or not any of it shows up in the reply. Redact tool results on the way in, not only the finished answer on the way out.

What breaks

Silent tool failures come first. A tool catches its own exception and returns an empty string or an empty list. The model reads that as a finding, so the answer becomes there are no matching orders rather than the lookup failed. A missing result and an empty result look identical to a language model, and only one of them is true.