How to Evaluate AI Systems with Real Code

A practical guide to evaluating GenAI systems, from a RAG assistant to a multi-agent researcher, grounded in real code you can open on GitHub, then the offline gate, online rollout, and monitoring lifecycle that keeps them honest in production.

Whatever you build on top of a language model, the same discipline keeps it honest. You never trust an output by default, you score it against examples whose answer you already know. This post walks it step by step with real code, build a golden set, evaluate a RAG app and then an AI agent, wire an automatic release gate, and keep evaluating after launch, plus the interview questions this method answers.

What evaluation means

Evaluation means checking a model output against answers you already know are correct, instead of trusting it because it looks finished. In normal software you would never ship code without a test. An AI answer looks done the moment it appears, so the check is easy to skip. That is the beginner mistake.

The tool for the job is a golden set. That is a small file of example inputs, each paired with the answer you already know is right. You run your system on those inputs and compare what it produces to the answers you wrote down. Build ten by hand and you have started.

A model can score well on a benchmark, a shared public test that many models compete on, and still fail with real users. Real users ask questions your test never included. A tool times out. The data behind an answer changes. Someone edits the prompt. So evaluation is never a one-time check at launch. Run it on every change, for as long as the system is live.

One more idea runs through all of it. Not every mistake costs the same. A made-up fact, called a hallucination, in a medical assistant can hurt someone. The same slip in a brainstorming tool is harmless. So a costly system needs a stricter test and a higher bar to pass before it ships.

The rest of this post is a sequence you can follow in order. Build a golden set, use it to evaluate a RAG app and then an AI agent, turn it into an automatic release gate, and keep evaluating after launch. Every step uses real code you can open on GitHub. Two deeper companions written for senior engineers, the production AI stack and the architecture of agentic AI, pick up once this clicks.

Step 1. Build your golden set

A golden set is small, just ten to a few dozen examples. Each one is an input paired with the answer you already know is right. You write them by hand, in a plain file, and keep the file in version control next to your code.

Where do the examples come from. Start with the questions users actually ask. Then add every failure you find. When something goes wrong in production, write it down as the next example. Over time the set gets sharper exactly where your system keeps getting hurt.

Not every example matters equally. Tag the ones where a wrong answer is expensive and hold those to a higher bar. This is where the cost of a mistake, from the last section, becomes a real rule you can score against.

The golden set is the single most useful thing a beginner can build. Once you have one, every later step, scoring a RAG app, grading an agent, gating a release, is just a different way of running your system against it and reading the score.

Step 2. Evaluate a RAG system

Start with the most common GenAI system, a RAG app. RAG stands for Retrieval Augmented Generation. It answers a question by first fetching relevant documents from a knowledge base, then writing an answer from them. There is no single correct answer to compare against, because two good answers can use completely different words.

So your golden set here is a list of example questions. Each one gets a short reference answer you write by hand, plus what the system produced and which documents it retrieved. Here is a trimmed slice of that evaluation set from the rag-expert-assistant project. It is four lists that line up by position. The question, the answer the system gave, the documents it pulled, and the reference answer.

Even ten examples are enough to start. Now score them. RAGAS is an open-source library for evaluating RAG systems. It gives four scores. Two for retrieval. Two for the generated answer. Low scores tell you exactly where the system is breaking.

• Faithfulness: Is every claim in the answer backed by the documents it was given, or did the model make something up. This is the score that catches hallucinations.
• Answer relevancy: Does the answer actually address the question that was asked.
• Context precision: Of the documents that were fetched, how many were actually relevant.
• Context recall: Did the fetch step find the documents needed to answer at all.

Context precision and context recall grade retrieval. Faithfulness and answer relevancy grade the answer. That split is what turns a low score into a repair instruction. The project prints the mapping right next to the numbers, so you know what to change.

Read those as a lookup table. A low faithfulness score means the model is drifting off its sources, so you tighten the grounding in the system prompt. A low context recall score means the right document never got retrieved, so you raise how many chunks you pull or add keyword search alongside the vector search. The score names the fix. Full file on GitHub, rag-expert-assistant/src/evaluate.py.

The four scores are still measured against a golden set. RAG needs a rich one, an example that carries the question, the answer, the retrieved context, and a reference. Same habit, more columns.

Step 3. Evaluate an AI agent

Move up to an AI agent. An agent is a system that plans, calls tools, and writes over several steps to finish a task, here a multi-paragraph research report with citations. You cannot compare a whole report to one reference answer, and reading every report by hand does not scale. So you use a second model to grade the first one against written rules. This is called LLM-as-judge.

The written rules are called a rubric. A judge is only as good as its rubric, so the care goes into writing the scoring rules down clearly. The ai-agents-project spells out a 0 to 3 scale for each quality it checks. Here is the rubric for factual accuracy.

Notice how specific each level is. A 1 is multiple factual errors. A 3 is every claim verifiable and current. That leaves the judge little room to guess. The same file carries matching rubrics for completeness and for citation quality. Vague rubrics give you vague scores, so spend your time here.

The agent needs its own golden set

The judge needs something to grade. That is a fixed set of test questions, the golden set for the agent. The project keeps ten, and tags each one by type and difficulty. Without the tags, the score drifts toward easy wins.

Five of the ten questions are shown. The tags are the point. If every question were an easy factual lookup, a weak agent would look great. Mix easy and hard, factual and synthesis, and the score reflects what the agent can really do. Big benchmarks do the same thing at scale, and call it stratified sampling.

This is where a fixed test set matters. Run the same ten questions through both designs.

• Single agent: One call does everything.
• Three-agent pipeline: Research, analysis, and writing split across three agents.

Score both with the same rubric. Then compare the results and see if the extra complexity is actually worth it.

That is a real evaluation in one expression. The multi-agent pipeline costs more tokens and more time. So it has to win by a clear margin, here twenty per cent on accuracy, before the code calls it justified. Without the shared test set and the judge, that comparison is just opinion. Full files on GitHub, judge_prompt.py and run_eval.py.

The method has not changed since Step 1. A golden set of questions, a written standard for what good looks like, and a score for every output. The judge and the rubric are how you hold that standard when the answer is too long and too open-ended to check by hand.

Step 4. Gate your releases automatically

Steps 1 to 3 scored one output at a time, mostly by hand. In production the same golden set does three jobs on its own. Here is the whole picture before we walk through it. This section covers the first row, the next section covers the other two.

Offline evaluation means running your golden set automatically, before a change ships, as a test. Freeze the examples in git. On every prompt edit, model swap, or retrieval change, the set runs in CI, the automated system that runs your tests on each change. If a tracked score drops past a line you set in advance, the change does not merge. Re-running the frozen set to catch a break like that is called regression testing. For an agent, a common line is a two point fall in task completion rate.

This is not hypothetical. Here is that gate from an alert-triage agent, trimmed to the decision. It runs a set of golden alerts through the whole system and scores each one two ways. A programmatic check that the outcome matches the known answer, and a separate judge model that scores grounding. The gate lets the change through only when both clear the bar. The comments and the surrounding code are trimmed, shown by the ... markers.

Read the last lines as the gate. The change goes through only when the known-answer match is perfect and the mean judge score clears the threshold. Otherwise it blocks. The function returns 0 to pass and 1 to block, so a CI job runs it as a plain test and fails the build on a regression. A strict check plus a softer judge score, wired into one exit code. That is Steps 1 to 3 turned into a gate that runs on its own.

This is why the golden set is frozen and versioned. If the examples change every week, a pass this week and a pass last week mean different things, and the gate is worthless. The set lives in git, has an owner, and changes only through a reviewed pull request. Every real failure in production becomes a new example.

Step 5. Keep evaluating after launch

The gate in Step 4 runs before a change ships, so it cannot tell you how real users react. That is what online evaluation is for, scoring a change on live traffic after it goes out. You do not switch it on for everyone at once. You use one of three safer rollouts.

• Canary: Send the change to a small percentage of real traffic first and watch the numbers before rolling it out to everyone. A bad change then hurts few users.
• Shadow mode: Run the new version silently beside the live one on real traffic and score what it would have done, with no user seeing its output.
• A/B test: Split traffic between the two versions and compare them side by side, so you can tell which one users do better with.

Whichever you use, watch task completion, latency, cost, and user feedback, and roll back the moment they slip.

One check never stops. Even a change that passed every gate decays over time, because the data and the questions keep moving. That slow fall in quality is called drift. Production monitoring scores a sample of live answers on a schedule to catch it. It also collects the cheap signals users already give you, thumbs up and down, edits, retries, and handoffs to a person.

Don't trust the judge blindly

Step 3 handed grading to a second model, the LLM-as-judge. That is convenient, but the judge is a model too, with the same blind spots as the thing it grades. A judge score you never checked is just another unverified output. You keep it honest the same way you keep any model honest, by measuring it against answers you already trust.

• Calibrate it against humans: Sample around a hundred judged answers, have a person score them, and compare. If the judge agrees with the human less than about 70% of the time, fix the rubric before you trust the number.
• Swap the answer order: When the judge compares two answers, it tends to prefer the one shown first. Run it both ways and average, so position does not decide the winner.
• Watch answer length: Longer answers score higher even when they are not better. Score conciseness as its own criterion, or the judge quietly rewards padding.
• Judge with a different model family: A model tends to favour answers from its own family. When you compare two models, judge with a third from a different family so the score is not home-cooked.
• Keep humans on the high-stakes calls: For anything expensive to get wrong, a medical answer, a legal summary, a payment action, the judge screens and a person signs off. LLM-as-judge is a filter, not the final word.

The 0 to 3 rubric back in Step 3 is what makes calibration possible. A vague rubric drifts, the same answer scores differently week to week, and there is nothing to calibrate against. Anchor each score to a concrete example of what earns it, then check the judge against human scores on a sample.

Metrics that matter to the product

Every score so far measured the model, faithfulness, accuracy, a judge rubric score. A high score there does not mean the feature works, because users do not care about faithfulness, they care whether the thing did what they came for. A real evaluation dashboard tracks a handful of product metrics per feature, not one accuracy number.

• Task completion rate: The share of users who got what they came for, end to end. The headline number. Everything else explains a change in this one.
• Tool success rate: How often the tools the system calls actually return usable results. A perfect model on top of a failing tool still fails the user.
• Hallucination rate: How often the system states something its sources do not support. The RAG faithfulness score from Step 2, watched in production.
• Latency and cost per request: How long an answer takes and what it costs in tokens. A better answer that is too slow or too expensive is not shippable.
• User satisfaction: The direct signal, thumbs up and down, edits, repeat questions, and escalations to a human.
• Business KPIs: The number the feature exists to move, resolved tickets, completed checkouts, retained users. The one your manager actually asked about.

Track five or six of these per feature and you can tell the difference between a model that scores well and a feature that works. Track one accuracy number and you cannot.

The tools teams use

You do not have to build any of this from scratch. A handful of tools cover most of the ground. Here is the short version of what to reach for and when.

The full table, with pricing and trade-offs, is in the architecture of agentic AI. Pick one, wire it into CI, and you have a real gate.

Interview questions and answers

These come up in any AI engineering interview past the junior level. Every answer below is built only from the steps above, so if those make sense, you can already answer them.

How would you evaluate an LLM application before and after launch?

In three modes, and I would name all three.

• Offline, before launch: A frozen golden set of known-answer examples. It runs as a gate on every prompt, model, or retrieval change, and blocks the merge if a tracked metric drops.
• Online, right after launch: Ship to a canary or an A/B slice of real traffic. Watch task completion, latency, cost, and user feedback.
• Monitoring, from then on: Keep scoring a sample of live answers on a schedule to catch drift.

The same golden set runs through all three.

Your model upgrade improved one task and users complain about another. What happened?

A regression the offline gate did not catch. A change that lifts one metric can quietly drop another. That is why the golden set is stratified across task types and frozen, so a win on synthesis cannot hide a loss on factual lookups. The fix is small. Add the failing case to the golden set as a new example, then gate on the whole set instead of the one metric you were chasing.

When would you trust LLM-as-judge, and when not?

It depends on what you are grading.

• Trust it: Subjective quality at scale, so tone, coherence, and format. Calibrate it against human scores on a sample first, and look for agreement around 70% or better.
• Do not trust it: Subtle factual accuracy, and any high-stakes call it would make on its own.

Then name the guardrails. Swap answer order to beat position bias. Watch length bias. Use a judge from a different model family when you compare two models.

What metrics would you put on an AI feature dashboard?

Not one accuracy number. Task completion rate is the headline. Then tool success rate, hallucination rate, latency, cost per request, and a user satisfaction signal. Add the one business KPI the feature exists to move. Task completion tells you whether it works. The rest tell you why it changed.

A model tops the benchmark. Is it ready to ship?

No. A benchmark is a fixed snapshot and production is a moving target. Real users ask questions the benchmark never had. Tools fail. The data shifts under you. Benchmarks calibrate you against the frontier, they do not tell you the feature works. You confirm that with your own golden set offline and a canary on real traffic.

Start this week

You do not need a platform to begin. Pick the one thing you are building right now, a RAG app or an agent, and give it a golden set today.

• Write ten examples: Ten inputs, each paired with the answer you know is correct, by hand, in a plain file. This is your golden set.
• Score against them after every change: A prompt tweak, a new model, a chunking change. Run the ten and watch the number before you ship.
• Weight your mistakes: Decide which failure is expensive, a hallucination, a wrong tool call, a slow answer, and weight your rubric or your gate toward catching that one first.
• Grow the set from failures: Every time something slips through in production, add it as example eleven. The set gets sharper exactly where you keep getting hurt.

Ten honest examples you run often will teach you more than any single accuracy figure. When you want to go deeper, this app has a module for each part of the method.

• Evaluation and Benchmarks: How language-model quality gets measured, including RAG scoring. Open the module.
• Agent Evaluation: Judging multi-step agents with rubrics and LLM-as-judge. Open the module.
• Eval-First Engineering: Building the golden set before the feature, as a working habit. Open the module.

The two senior-level companions, the production AI stack and the architecture of agentic AI, pick up where this post leaves off. Start with ten examples, though. Everything else is built on that.