Interview question, debug an AI agent that is only sometimes wrong

The hardest bugs in production AI never throw an error. The request returns 200 OK, the dashboard stays green, and the answer is quietly, confidently wrong. This is how production teams see inside a Large Language Model system, and how to answer the observability interview like someone who has actually done it.

A user says "sometimes it hallucinates." There is no stack trace, no error, no way to reproduce it. This is the full map of AI observability. Why it differs from normal monitoring, and how traces become the unit of debugging. How OpenTelemetry captures a Large Language Model (LLM) request, how quality is measured on live traffic, and how a real incident gets investigated end to end.

The bug that never throws an error

Here is the report you will get. "Sometimes it works." "Sometimes it hallucinates." "Sometimes it ignores the document I gave it." No stack trace. No error code. No steps to reproduce. You ask the user for the exact input, you paste it in yourself, and it answers perfectly. The bug refuses to show up when you are watching.

This is the part of AI engineering that catches people off guard. In normal software, a bug is a broken thing. Something throws, something returns the wrong status, a test goes red, and you follow the trail. In an AI system, the request succeeds. The Large Language Model (LLM) returns a fluent, confident, well-formatted answer, and the answer is wrong. Your monitoring sees a 200 OK and a healthy latency and moves on. The failure and the success look identical from the outside.

Three things make these bugs hard, and they are worth naming up front because the rest of this post is really about defeating each one.

• There is no error to catch: a wrong answer and a right answer both return success. Error rate stays at zero while quality quietly falls off a cliff.
• It is not reproducible: the same input can give a different output each time, because of sampling temperature, a changed model, or different retrieved documents. You cannot re-run the failure on demand.
• Correctness is a judgement, not a check: no assertion tells you whether an answer is grounded, helpful, or safe. Someone or something has to score it.

The discipline that beats all three is called observability, and for AI it means something more specific than dashboards. It means recording enough about every single request that you can answer a question you did not think to ask in advance. Months later, from data you already have, you can ask why one answer ignored the policy document. This app has companion pieces on the full production AI stack and on what breaks when a million people use your AI app. This post zooms all the way into one layer of both, how you see inside the box.

Traditional monitoring answers "is the system up and fast." AI observability answers "was the answer any good." A system can be perfectly up, perfectly fast, and perfectly wrong. Only the second question catches that.

A Tuesday at a company that shipped an agent

Picture a mid-size company that added an AI support agent to its product. It answers customer questions from the company help centre. It can look up an order, check a refund policy, and draft a reply. It launched two months ago. The demo was great. Leadership loved it. It handles thousands of conversations a day now.

A support lead forwards a thread to engineering. A customer was told their order qualified for a refund. It did not. The agent had stated the wrong policy with total confidence. Below that email are three more like it from the past week that nobody connected until now.

An engineer picks it up. First move, reproduce it. He copies the customer question into the agent. It answers correctly. He tries again. Correct again. He cannot make it fail. The only record he has is the final text the agent sent the customer, stored in the conversations table. He has the wrong answer. He does not have the prompt that produced it, the documents it retrieved, or which model version was serving at that hour. He is holding the output of a function with none of its inputs.

He checks the dashboards. Latency is healthy. Error rate is zero. The model provider status page is green. Every signal the team has says the system is perfectly fine, and yet it is telling customers things that are false. There is nothing to grep, nothing to bisect, nothing to step through.

By Thursday it reaches leadership as a trust problem, which is what it is. The question in the room is simple and the team cannot answer it. How often is this happening, and why. Nobody knows, because nobody recorded the one thing that would tell them. That gap, the distance between knowing the agent said something wrong and being able to say exactly why, is the thing an AI observability platform exists to close.

This story is a composite, but the shape is common. Almost every team that ships an LLM feature without observability hits this exact wall within a few months, and the fix is never a clever prompt. It is the boring infrastructure that records what happened.

The interview question

This scenario shows up in Senior, Staff, and Principal interviews because it separates people who have shipped AI from people who have only prototyped it. The question usually arrives like this.

Users report that your AI agent is "sometimes wrong." It hallucinates, or ignores retrieved context, but only occasionally, and you cannot reproduce it. There are no errors. How do you design observability for this system so you can detect, debug, and prevent these failures?

What the interviewer is actually testing

They are not looking for a list of tools. They are checking whether you understand why this problem is different from anything in normal backend work, and whether you have a mental model for making a probabilistic system debuggable. A strong answer moves through four moves.

• Name why traditional monitoring fails here: success and failure both return 200 OK, so error rate is blind.
• Make the trace the unit of debugging: capture the prompt, retrieval, model version, tokens, and cost per request.
• Measure quality, since there is no error to catch: score a sample of live traffic for groundedness and helpfulness.
• Close the loop: turn every caught bug into a permanent regression test.

The traps

• Reaching for a vendor name first: naming a tool before explaining what has to be captured and why signals you learned the logo, not the problem. Describe the data model first, then say which tools implement it.
• Treating it as a prompt problem: a better prompt does not help if you cannot see which prompt ran. Observability comes before iteration.
• Forgetting privacy: traces capture full prompts and outputs, which means personal data. If you do not mention redaction and retention, a senior interviewer notices immediately.
• Stopping at detection: finding the bug is half the answer. Rollback, canary, and regression testing are what a Staff engineer adds, because prevention is the actual job.

Why AI observability is a different animal

Observability is an old idea. The classic version has three signals, and every backend engineer knows them.

• Logs: lines of this happened, one event at a time.
• Metrics: aggregated numbers like requests per second and p95 latency.
• Traces: the full path of one request as it moves across services.

This is Application Performance Monitoring (APM), and it is genuinely good at what it does. It just answers the wrong question for AI.

APM was built for a world where correctness is settled in code and caught by tests. The service either returned the right value or it threw. Nobody built a metric for the response was fluent but false, because in normal software that state does not exist. An LLM lives in exactly that state. It fails by being confidently wrong while returning success.

Why logging the final output is not enough

The instinct is to log the answer. It is also not enough, and it is worth being precise about why. An LLM answer is the output of a long chain. A system prompt, the user message, some retrieved documents, maybe a few tool calls, a specific model version, and a sampling setting. When the answer is wrong, the answer text tells you nothing about which link broke. Was the wrong document retrieved? Was the right document retrieved and then ignored? Did a tool return stale data? Did the model version change overnight? The output is the same shape in every one of those cases. You need the inputs, all of them, recorded together.

That last row is the one people miss. In normal software, nothing changes unless you change it. In AI, the provider can update the model behind the same name and your behaviour shifts with no deploy on your side. Observability is how you notice, and traditional monitoring has no place to even look.

The trace is the unit of debugging

Here is the single most important shift. In AI systems, you stop debugging with log lines and start debugging with traces. A trace is the complete recorded path of one request. It is made of spans, and a span is one timed operation inside that request, a model call, a retrieval, a tool execution, a guardrail check. Each span records when it started, how long it took, and a set of attributes, which for AI are the useful part, the model name, the token counts, the cost.

Spans nest. The top span is the whole request. Under it sit the steps, and under those sit their sub-steps. A retrieval span might contain a vector-search span. A model call sits next to the tool call it decided to make. That tree is the shape of what the agent actually did, reconstructed after the fact from data, and it is the thing you open when something goes wrong.

Three ideas hold this together, and they are the same three whether you are tracing a web request or an agent.

• Trace ID: one identifier stamped on every span in the request. It is what lets you pull all the pieces back together, and what ties a metric or a log line back to the same event.
• Span ID and parent span ID: each span has its own ID and records its parent. That parent link is what builds the tree. Without it you have a pile of operations with no shape.
• Context propagation: the trace ID and current span travel with the work, across function calls, threads, and network hops. A step that happens in another service still lands under the same trace instead of starting a new one.

The full request lifecycle, as spans

A single agent answer is not one model call. It is a small pipeline, and every stage becomes a span. Here is the shape of a typical Retrieval Augmented Generation (RAG) agent request.

Read that top to bottom and you can see the whole decision. Which documents came back. Whether the tool succeeded. How long the model took to start streaming. What it cost. When a customer gets a wrong policy, this is the record that tells you, in seconds, whether the policy document was even retrieved. Without it, you are the engineer from Tuesday, holding an output with no inputs.

Here is a useful test for whether your tracing is good enough. Pick any past answer a user complained about and ask whether you can see the exact prompt and the exact retrieved chunks for that specific request. If the answer is no, your observability is not done, no matter how many dashboards you have.

In the interview, say this plainly. The trace is the unit of debugging, and if you cannot pull the exact prompt and retrieved chunks for one specific bad answer, nothing else you built matters.

OpenTelemetry, the standard that makes this portable

You could invent your own trace format. Do not. There is an open standard called OpenTelemetry, usually shortened to OTel, and it is the thing the rest of the industry has agreed on. It gives you libraries to create spans, a defined shape for what a span is, and a wire format called OTLP, the OpenTelemetry Protocol, to ship them. The payoff is simple. Instrument your code once, and you can send the data to any backend that speaks OTLP. You are not married to one vendor.

Mechanically, you wrap each meaningful operation in a span. You start a span before a model call, set attributes on it as you learn them, and end it when the call returns. The library handles the trace ID, the parent links, and context propagation for you, so a span created deep inside a helper still attaches to the right request.

GenAI semantic conventions

A span is only useful if a tool can understand it, and that needs shared names. If your team calls it model_name, mine calls it llm, and a third calls it engine, no dashboard can chart across all three. OpenTelemetry solves this with semantic conventions, agreed attribute names for common operations. There is now a GenAI set specifically for AI. gen_ai.request.model for the model you asked for, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens for token counts, gen_ai.response.model for the version that actually answered. Use them. It means any OTel-aware backend understands your traces with no custom parsing, and it means the token and cost math works without you writing it.

Record the response model as well as the requested one. You ask for a model by a friendly name, but the provider serves a specific dated version behind it, and that version can change. gen_ai.response.model is often the single attribute that solves a quality-dropped-and-we-shipped-nothing mystery.

One more thing OTel gets right. It separates instrumentation from where the data goes. Your app emits spans to a collector, a small service that receives OTLP, and the collector is where you batch, sample, redact personal data, and fan the data out to storage. Your application code does not need to know or care which backend you use, which means you can change backends without touching the agent.

The observability platform, component by component

Put the pieces together and you get a platform. It is not one product. It is a pipeline that starts inside your running app and ends at the places engineers actually look. Here is the whole thing, then each part.

• Instrumented app: the gateway, the orchestrator, the retriever, the tools, and the model client each emit spans via the OTel SDK. This is where trace, prompt, model version, retrieval, tool result, and routing decision are all captured at the source.
• Collector: receives spans over OTLP, batches them, redacts personal data, applies sampling, and fans out to storage. The one place to enforce privacy, so no raw prompt reaches disk unredacted.
• Trace store: holds the spans. This is what you open to debug a single request. Optimised for "show me trace 7f3a in full."
• Metrics store: holds the aggregated time series, latency percentiles, tokens per minute, cost, throughput, cache hit rate. This is what dashboards chart and what fast alerts fire on.
• Eval and quality store: holds the scores, groundedness, hallucination flags, judge ratings, and user feedback, joined back to the trace they came from. This is the AI-specific store that APM never had.
• Dashboards, alerting, replay: the three places engineers look. Dashboards for the live picture, alerting for when a Service Level Objective (SLO) or a quality score slides, and the replay and eval harness for reproducing and fixing a failure.

The important design property is the join. Because one trace ID threads through every stage, a spike on the cost dashboard, a drop on the quality dashboard, and a specific bad answer all point back to the same trace. You can move from quality fell at 2pm to "here are the twenty requests that caused it" without guessing.

In the interview, the join is the detail that marks a senior answer. One trace ID threading metrics, logs, and spans is how you go from quality fell at 2pm to the exact requests that caused it, and most candidates never think to mention it.

Retrieval, tools, and multi-agent chains

The three most common AI failures each live in a specific place in the trace. Learn to read those places and most debugging becomes fast. Start with the most common complaint of all, that it ignored the document you gave it.

Retrieval, where it ignored the context lives

In a Retrieval Augmented Generation (RAG) system, an answer is only as good as what got retrieved and whether the model used it. There are two very different failures hiding behind the same complaint, and the trace tells them apart instantly. Either the right document was not retrieved, which is a retrieval bug, or the right document was retrieved and the model ignored it, which is a grounding bug. The fixes are opposite. One is about your index and your search; the other is about your prompt and your model. Guessing which one you have, without the trace, wastes days.

So the retrieval span has to record what came back, not just that it ran. Log the query, the top-k chunk IDs, their similarity scores, and ideally the chunk text or a hash of it. Now was the policy even retrieved is a five-second lookup instead of an argument.

Tool execution, where stale or wrong data enters

Agents call tools, a database query, an API call, a search. A tool that returns wrong or stale data poisons everything downstream. The model will confidently build an answer on top of the bad input. Each tool call is a span, and it needs to record the arguments the model chose, the raw result, the latency, and whether it errored or retried.

When the answer is wrong, this chain lets you ask the right question in order. Did the model pick the wrong tool or the wrong arguments? Did the tool return bad data? Or did the tool return good data and the model still answered wrong? Three different bugs, three different owners, one trace to tell them apart.

Multi-agent, where the trace becomes essential

Once you have several agents handing work to each other, an orchestrator calling specialised sub-agents, the trace stops being nice to have and becomes the only way to understand anything. A single user request might fan out into a dozen model calls across several agents. The nested span tree is the map of who called whom, where the time went, and where the wrong turn happened.

Record the routing decision as a span attribute. In a multi-agent system, it sent the question to the wrong agent is a whole class of bug, and it is invisible unless you log which route was chosen and why.

Measuring quality when there is no error

Traces let you debug one request. They do not tell you how often the agent is wrong across thousands of requests, because nobody is reading every trace. For that you need to measure quality, continuously, on live traffic. This is the piece that has no equivalent in traditional monitoring, and it is where most teams are weakest.

The idea borrows from testing. Offline, before a release, you run a frozen set of known-good cases and score the new prompt or model against them, and that eval gates the release the way unit tests gate code. But offline evals cannot catch a model that changes under you in production, or a real user question you never thought to test. So you also score live traffic. You take a sample of real requests, run automated scorers over them, and write the scores back next to the trace.

How you score an answer

• Groundedness check: compare the answer against the chunks that were retrieved. Does every claim trace back to a source? This directly catches it ignored the context and hallucination, and it is the single most valuable score for a RAG system.
• Large Language Model (LLM) judge: a separate model prompted with a rubric, scoring the answer for helpfulness, tone, or task success. Cheap and scalable, and the standard way to grade quality at volume.
• Guardrail and safety classifiers: fast checks for leaked personal data, toxic content, or policy violations, run on the output. A blocked output is itself a recorded event, so you can see how often the last line of defence fires.
• Online signals from users: thumbs down, a retry, an edit, an escalation to a human. These are free labels. A spike in any of them is an early smoke alarm, and they feed straight back into your golden set.

An LLM judge is useful but not free of judgement. It can be wrong, and it can be biased toward longer or more confident answers. Validate the judge against human ratings on a sample before you trust its number, and keep a human in the loop on the cases that matter. A judge you never checked is just a second opinion you did not verify.

The reason this matters so much is quality drift. A provider updates the model behind the same name. Your retrieval corpus grows and the good chunk falls out of the top-k. Neither event throws an error. Neither shows on a latency chart. The only thing that catches a slow slide in correctness is scoring live traffic over time and alerting when the score falls. That alert is the difference between finding out from your own dashboard and finding out from an angry customer three weeks later.

In the interview, this is the line that lands. Quality is measured, not asserted, so you score a sample of live traffic and alert on drift, because the model can get worse with no error and no deploy on your side.

Versioning, cost, and the silent model swap

When quality drops, the first question is always "what changed." You can only answer it if everything that can change is pinned to a version and recorded on the trace. In an AI system, more things change than you might expect.

• Prompt version: your system prompt is code. Pin it, version it, and record which version served each request, so any answer traces back to the exact instructions that produced it.
• Model version: record the specific model version that actually answered, not the friendly name you requested. This is the one people forget, and it is the one that catches silent provider updates.
• Agent graph version: for an agent, the shape of the graph, which tools exist, how routing works, what the sub-agents are, is also a version. A change there can shift behaviour as much as a new prompt.
• Retrieval index version: the corpus and the embedding model behind your search change over time. Pin them too, because a re-indexed corpus is a silent behaviour change.

Here is the failure this prevents, and it is the one that makes senior engineers nod in an interview. Quality drops. You check your deploys. Nothing shipped. Your code is byte for byte what it was last week. The cause is that the provider rolled the model forward under the same name, and your behaviour moved with it. If your traces pin the response model version, this shows up immediately as a change in one attribute. If they do not, you will spend days looking for a bug in code that never changed.

Token accounting and cost attribution

Every model call span records input and output tokens, and from tokens you get cost. Summed up, this is how you see spend per request, per user, per tenant, and per feature. It sounds like a finance concern, but it is an observability one, because cost is a signal. A request burning ten times the usual tokens is often a bug. A retrieval that stuffed too much into the context, an agent stuck in a loop, a prompt that ballooned. Cost attribution, tying every token back to who spent it, is what lets you find the one runaway tenant or the one feature quietly costing more than the rest of the product combined.

Token count is also a quality signal in disguise. A sudden jump in output tokens on a feature can mean the model started rambling, and a drop in retrieved tokens can mean your index stopped returning good chunks. Watch the token charts next to the quality charts, not just the cost report.

A real investigation, start to finish

Now put it together on the Tuesday incident, but this time assume the team built the observability first. Watch how the same problem goes from unsolvable to routine.

Step by step, this is what the on-call engineer does.

• 1. The alert fires: not a customer email. The groundedness score on the support agent dropped below its Service Level Objective (SLO) over the last hour, and a drift alert paged. The team knows before the support queue does.
• 2. Open the failing traces: the alert links straight to the sample of low-scoring requests that triggered it. The engineer opens one where a refund policy was stated wrong.
• 3. Inspect retrieval: the retrieval span shows the top-k chunks. The correct policy chunk is right there, retrieved, with a high score. So this is not a retrieval bug. The right document was available.
• 4. Inspect the prompt: the engineer reads the exact prompt sent to the model. The retrieved chunk is in the context, but it is buried near the end of a very long context window, after a large tool result that got stuffed in ahead of it.
• 5. Inspect the tool call: the tool span shows the get_order call returned a big blob that pushed the policy text to the bottom of the prompt. The model anchored on the tool data and skimmed past the policy.
• 6. Root cause: It was a prompt-assembly bug, not a hallucination out of nowhere and not a retrieval miss. An oversized tool result crowded out the retrieved policy, so the model answered from the wrong part of the context.
• 7. Stop the bleeding: a feature flag caps the tool result size and re-orders the prompt so retrieved policy always comes last, closest to the question. No full deploy, just a flag flip, live in seconds.
• 8. Validate and regress: replay the failing request against the fix and confirm the answer is now correct and grounded. Then freeze that exact request as a regression test in the golden set.

Notice what made this possible. Every step read from the trace of one real request. The prompt, the chunks, the tool result, the model version, all recorded. Without that record, steps three through six are pure speculation, and the incident stays open for a week. With it, root cause took one engineer twenty minutes.

Replay, rollback, and closing the loop

Finding the bug is not the end. A Staff engineer is judged on what happens next. Can you fix it safely, prove the fix, and guarantee it never comes back. Three mechanisms do this, and they all lean on the trace you already captured.

Replay

A production system is not reproducible on your laptop, but a captured trace is. Replay means re-running a real failing request against a new prompt or model version, using the exact frozen inputs from its trace, the same prompt, the same retrieved context, the same tool results. You are comparing versions on the real request that failed, not a made-up one. This is how you know a fix actually fixes the reported case, and how you compare a candidate model to the current one before you trust it.

Rollback, canary, and kill switches

When something regresses, you need to change it back fast. That means three habits. Ship changes as a canary deployment, to a small slice of traffic first, watching the quality and cost scores on that slice, expanding only if they hold and rolling back automatically if they slide. Keep a feature flag on every model, prompt, and capability, so you can turn it off in seconds without a deploy. And keep the previous version warm, so rollback is a switch, not a rebuild.

The most useful sentence during an AI incident is turn it off. If turning a bad model off needs a code change and a deploy, your incident is measured in hours. If it is a flag, it is measured in seconds. Build the flag before you need it.

Regression testing, so it never recurs

The last step is the one that compounds. Every confirmed production failure becomes a permanent test. You take the captured trace, freeze the input and the retrieved context, attach the known-good answer, and add it to your golden evaluation set. Now the release gate re-runs that exact case on every future change. A bug that reached a customer once can never ship again, because a specific test would go red first.

This is what turns an observability platform from a debugging tool into a system that gets more reliable over time. Every incident makes the eval suite smarter. The failures teach it. A team that does this ships fewer repeat bugs every quarter, not because they got lucky, but because the loop is closed.