ML System Design Patterns

Modelling patterns are about the model. ML system design patterns go beyond the model itself. The challenge is that the data your system depends on is never fully under your control. It changes without warning, and other teams use it without telling you. In this post we'll go through the patterns that deal with that, one layer at a time. Features, training, serving, releases and monitoring. Every one of them is applied to the same example, a card fraud detector, so you can see why each choice was made. Then we'll look at the mistakes that quietly break real world systems, and how all of this changes in the age of language models.

Code dependencies are explicit and versioned. Data dependencies are implicit, silently changing, and shared by teams you have never met. Nearly every pattern in production machine learning exists to deal with that one difference. This post maps them layer by layer. Feature stores and point-in-time correctness, training pipelines, the four serving shapes, the release ladder, and the three kinds of drift. We'll finish with the mistakes that quietly break real world systems, and how all of this changes in the age of language models.

The data you do not fully control

Ask most people about machine learning patterns and they will talk about the model. Which algorithm to pick. How to build the features. How to stop it overfitting, which means learning the training data itself instead of the pattern inside it. That work is real and worth learning. All of it is about the model. ML system design patterns are a different list. They cover everything around the model, because that is where a production system actually breaks.

The hard part is the data. One term first, because the rest of the post leans on it. A feature is one input the model uses to make a prediction, such as how many payments this card made in the last hour. Features are not raw data. They are computed from raw data, and that computation is where most production bugs live.

Now compare that data with the code around it. A code dependency is a library your project needs. It is named in a file. You pin it to a version. A diff shows you exactly what changed and when. Data works nothing like that. A column can change meaning overnight, with no release and no warning.

The same feature also gets written twice. A data scientist writes it in Python to build the training set. A backend engineer rewrites it in Java or Go for the live service. Both versions started from the same description in a document. Neither one is wrong on its own. They just slowly stop agreeing, and nobody is comparing them. On top of that, other teams read your data without ever telling you.

That is the whole problem. Nearly every pattern below is trying to make data behave a bit more like code, so you can see what it is, pin which version you used, and put the old one back.

A model is only as good as data you do not own. Every pattern in this post does one of three things. It makes that data visible, or it makes a result repeatable, or it makes a change easy to undo. If a pattern does none of the three, you probably do not need it yet.

One more thing before the patterns. Almost every decision below is the same three-way trade, so it helps to name it now.

Ask these three questions at every layer.

• Freshness: How up to date does this number have to be. A count that is an hour old is fine for a weekly report and useless for catching a card being tested right now.
• Latency: How fast does the answer have to come back. Anything a user is waiting for is measured in milliseconds. Anything that runs overnight is not.
• Cost: How much is one prediction worth. Keeping a value fresh to the second means computing it constantly, whether or not anyone asks for it.

You cannot have all three. Move toward one corner and you pay in another. Most design arguments are two people pulling toward different corners without saying which one they picked.

The loop every ML system runs

Every production ML system runs the same loop, whatever the model is. Data comes in. Features are built from it. A model is trained, evaluated, and served. The results are monitored. Then it runs again.

Each stage fails in its own way, and the failures are specific enough to name. That is why the rest of this post is organised the same way. Every section takes one stage of this loop and lists the patterns that hold it up.

Two things about the loop matter more than the stages themselves.

• It is a loop, not a line: What you serve today shapes what you train on tomorrow. Most of the hardest failures in this post come from that return path rather than from any single stage.
• The stages belong to different people: Data sourcing, features, serving, and monitoring often sit with four different teams. Almost every skew and dependency problem below starts at one of those handovers.

This post assumes the loop and focuses on what goes wrong across it. If you want the stages themselves taught properly, with worked examples rather than a summary, two tracks in this app cover the full path. Start with ML Pipelines for data through training, then MLOps for deployment and monitoring.

One system, followed all the way through

Patterns are hard to judge in the abstract. So from here on we'll apply every one of them to a single system, and you can watch the same six numbers decide each choice.

The system is a card payment fraud detector. It sits inside checkout. For every payment it makes one of three calls. Approve it, send it to a human reviewer, or block it.

Read those numbers before going on, because most of them close off options.

• 80ms at p99: p99 means the slowest 1 request in 100. So 99 out of every 100 payments must be scored within 80 milliseconds. A slow model is out, and so is working a feature out by querying a database while the customer waits.
• A chargeback arriving 5 to 60 days later: A chargeback is the cardholder disputing the payment and the bank reversing it. That is how you find out a payment was fraud, and it turns up weeks later. So you cannot know today whether today's predictions were right, and anything that waits for that answer is running two months behind.
• 0.3% fraudulent: Only 3 payments in every 1000 are fraud. A model that approves everything is therefore right 99.7% of the time and completely useless. Accuracy is the wrong measure here, and the rare case needs deliberate handling.
• Three consumers: A consumer is anything that reads your output. Here that is checkout, the human review queue, and a finance report. Any change you make lands on all three, including the two you were not thinking about.

This is worth doing for your own system before you pick any pattern. Write down what it predicts, its traffic, its latency budget, where the label comes from and how late it is, the base rate, and who reads the output. Most design arguments end quickly once those six numbers are on the table.

Data and feature patterns

Everything else in this post depends on this layer being right. Get it wrong and your numbers will lie to you for months. For the fraud detector, this is the layer that decides one thing. Does the model see the same numbers at checkout that it saw in training.

Feature store

A feature store is one place where a feature is defined, serving two very different readers. You write the definition once. The store then keeps the results in two forms.

• The offline store: Every past value, kept as history. The offline store is read in big scans to build a table of training examples, so it has to be complete rather than fast.
• The online store: The latest value only, for each card or customer. The live service reads the online store in a few milliseconds while the payment is waiting.

Both come from the same definition, so they cannot quietly disagree.

You do not have to buy a product to get this. What you need is one place where the feature is defined, with both the training code and the serving code generated from it.

Take one feature in the fraud detector, cards_seen_from_this_device_24h. It counts how many different cards this device has used in the last day. That count has to mean exactly the same thing in the training set and at checkout. Suppose training counts a full 24 hours and the live service counts only completed payments in the last 24 hours. The numbers now differ by a few, on the exact devices that matter most. The model is scoring something it was never trained on. Nothing errors, and the gap is invisible until someone goes looking. That gap has a name, training-serving skew, and it is the single most common failure this whole layer exists to prevent.

Point-in-time correctness

A training row is one past payment plus the answer, meaning whether it turned out to be fraud. The features on that row have to be the values as they were at the moment of the payment, not the values as they are today.

This sounds obvious and it gets broken constantly, because the easy way to build a training set is to take the feature values you have now and attach them to old outcomes. That one shortcut is the mistake.

This one bites the fraud detector hard. A chargeback arrives up to 60 days after the payment. So when you build a training row today, it is very easy to fold in things that only became known during those 60 days. The model then learns from facts it will never have at checkout. That is called leakage.

The reason leakage deserves its own section is that it makes your numbers look better while making your system worse. Offline accuracy goes up. Everyone is pleased. Live accuracy is worse, and nobody connects the two, because the mistake is in the join, meaning how the feature table and the label table were matched up, not in the model.

If your offline score is much better than your live score and you cannot explain the gap, check the point-in-time join before you touch the model. That means checking that every feature on a training row was computed only from data that existed at the moment of the prediction. This is one of the most common causes, and it is almost never the first thing teams look at.

The rest of the data layer

Four more that are less famous and still earn their place.

• Transform-in-graph: Save the preprocessing inside the model file itself, so the same code that cleaned the training data also cleans the live request. Two separate code paths parsing the same date format will eventually disagree about one of them.
• Repeatable splitting: Decide which rows go to training and which to testing using a hash of a stable key, such as the customer id, rather than a random seed. A hash is a function that turns an id into a number, and it always returns the same number for the same id. Send everything landing in the last tenth to the test set. With a random seed, rows swap sides every time the pipeline runs, so yesterday's test data becomes today's training data and your score slowly stops meaning anything.
• Windowed aggregation: Work out rolling numbers ahead of time, such as a 30-day count, in the data pipeline or a streaming layer. At inference time the model reads a ready-made value instead of calculating one inside an 80ms budget.
• Bridged schema: When a table changes shape, backfill the older records so they match it. You then keep training on years of history instead of throwing away everything from before the change.

Data validation at ingest

Ingest is where data first enters your pipeline, and that is where the checks belong. There are two. Does the data match the schema, meaning the columns you expect holding the types you expect. Do the values look roughly like they usually do. If either check fails, stop the pipeline loudly rather than training on it.

These checks are cheap to write and they catch most real production incidents. A column that silently started arriving empty will not be caught anywhere further down. It will just quietly become a worse model.

Training patterns

The pipeline, not the notebook

Most models start life in a notebook, where every step runs top to bottom in one file. That is fine for exploring and it does not survive contact with production. A workflow pipeline is the same path from raw data to a finished model, broken into separate steps that run in order. Each step can be retried on its own without redoing the others. Each one caches its output, so unchanged steps are skipped next time. And the whole run can be rebuilt later from a single commit, meaning one recorded version of the code.

The value shows up on the day someone asks which data produced the model currently running in production. With a notebook, that question has no answer. With a pipeline, it is a lookup.

Continued training with explicit triggers

Models get stale, so at some point you train a new one. The question is what makes that happen. There are three honest answers, and you pick one deliberately and write it down.

• A schedule: Every week, or every night. Simple, and it retrains whether or not anything changed.
• A data threshold: Once another 100,000 labelled examples have arrived. Ties the work to how much new information you actually have.
• An alarm: When monitoring says the incoming data or the live quality has moved. The most responsive option, and it needs the monitoring layer to exist first.

For the fraud detector a schedule alone is a poor fit. Fraud patterns change when an attacker decides they should, not on the first of the month. What most teams are really running on is a fourth option, retraining whenever someone remembers. That is not a trigger.

Pair whichever you choose with a warm start policy. A warm start means resuming from the previous checkpoint, a saved snapshot of the model partway through training, rather than starting from nothing. It is much faster, so use it for the routine runs. Then retrain from scratch every so often. Small distortions build up across warm starts, and no single run makes them visible.

Shaping the problem before you shape the model

These three change the question you are asking rather than the model that answers it. All three apply to the fraud detector, where 3 payments in 1000 are fraud and plain accuracy tells you nothing.

• Rebalancing: When one outcome is very rare, the model can score well by ignoring it entirely. So you either keep fewer of the common cases, or tell training to weight the rare ones more heavily. How rare the rare case actually is decides which, so measure it before choosing.
• Reframing: Change what the model outputs. Turn a regression, meaning predicting the exact value of an order, into classification over price bands. You get a spread of likely answers instead of one number, and that often fits the decision the product has to make.
• Cascade: Split one problem into a sequence of models, where the first decides which case this is and the second handles it. Powerful and easy to get wrong. The chain has to be trained and scored as one thing, or mistakes in the first model quietly pile onto the second.

Useful overfitting is a real pattern, not a mistake. Sometimes memorising is exactly the goal. Training a small model to copy a big one. Replacing a slow simulation with a model that has learnt its answers. Overfitting a single batch just to check your code runs.

Serving patterns

Serving means running the trained model to get an actual prediction. These four patterns answer one question about it. When do you compute the prediction, and who is waiting while you do it.

For the fraud detector the answer is forced. You cannot work out an answer in advance for a payment that has not happened yet. A customer is standing at checkout, so it has to happen live and fit inside 80ms.

All four sit on top of one idea, the stateless serving function. Stateless means the server remembers nothing about a user between requests. Everything it needs arrives with the request or gets looked up. Any machine can then answer any request, so you handle more traffic by adding machines instead of rewriting the app.

Retrieval and ranking

This is the shape behind every search box, feed, and recommendation list. You cannot score ten million items in 200 milliseconds, so you do it in two passes. A cheap first pass cuts millions of candidates down to a few hundred. An expensive second pass carefully orders the survivors.

This is the one pattern the fraud detector does not need. There are no candidates to narrow down, just one payment to score. It is here because the same shape runs every search, feed and recommendation system, and it comes back later in this post under a different name.

The four that keep serving honest

• Keyed predictions: The caller sends an id along with the request, and that id comes back attached to the prediction. When answers arrive out of order or in a batch, you can match each one to what it was about instead of relying on position.
• Dynamic batching: Hold requests for a few milliseconds so several run through the hardware together. It trades a little tail latency, meaning the wait on the slowest requests, for a lot of throughput, meaning how many predictions the system gets through per second. That is usually the right trade on a GPU, because that hardware is built to run many things at once.
• Graceful degradation: An explicit fallback for when the model is down or slow, whether that is the last known prediction, a heuristic, or a global average. Design it before you need it.
• Prediction logging: Log the inputs, the features, the output, and the model version. That one record is your debugging tool, your monitoring input, and your next training set.

If you build one thing from this section first, build prediction logging. Every monitoring pattern later in this post reads from it, and no amount of clever work recovers a request you failed to record.

Release patterns

A model release is riskier than a code release. Broken code throws an error, an alert fires, and someone gets paged, meaning called out to fix it. A worse model throws nothing. It returns a perfectly valid answer that is just wrong more often than it used to be, and no alert exists for that. The ladder below exists so each rung rules out one specific risk before you climb to the next.

Shadow mode matters more than usual for the fraud detector. A bad model here does not return an error. It declines a real customer at checkout and they go somewhere else.

A canary deployment, serving a small slice of traffic with the new model and ramping only if the metrics hold, is the rung most teams reach for first. But two rungs get confused often enough to be worth separating. In shadow mode the new model sees real traffic and its answers are recorded but never used, so it proves the model runs correctly on real data with no risk to anyone. In an A/B test the new model actually serves some users, so it proves the model is better for the business. Those are completely different claims.

You need both, because scores measured offline and results measured on live users disagree constantly. A model that looks clearly better on your test set routinely loses on revenue or completion rate, and no amount of offline work tells you that in advance.

• Champion and challenger: The model in production is the champion. Any new candidate has to beat it on a fixed, agreed evaluation before it is allowed to replace it. That turns shipping from a judgement call into a rule.
• Multi-armed bandit: Traffic shifts automatically toward whichever version is winning, so you lose as little as possible while learning. Wrong choice when you need a clean measurement, because the split keeps moving underneath you.
• Model registry: A store of model versions that are never edited, each recorded with its lineage, meaning the data snapshot, the code commit, and the hyperparameters it was trained with. Promotion just moves a pointer to the version that serves traffic, so rollback is moving that pointer back.

The registry is what makes rollback boring, and boring rollback is what makes teams willing to ship. If reverting a model means a rebuild, people stop shipping and start arguing instead.

Monitoring patterns

Ordinary monitoring tells you the service is up and answering. An ML system can be up, fast, and steadily getting things wrong. So this layer measures whether the answers are still any good, not whether the server is alive.

Drift, and why one word is three problems

Drift is a catch-all word for the world moving away from what the model was trained on. It hides three different problems, and they need three different responses.

• Covariate shift: Covariate shift means your traffic changed. You launch in a new country and the typical payment amount is different, but a suspicious payment still looks suspicious in the same way. Retraining on recent data usually fixes it.
• Label shift: Label shift means the outcome got more or less common while the inputs still mean what they meant. A fraud ring arrives and your 3 in 1000 becomes 9 in 1000. The usual fix is reweighting, or moving the decision threshold, meaning the score above which you block a payment, rather than retraining.
• Concept drift: The rule itself changed. Attackers work out what you are looking for and change tactics, so the behaviour that used to signal fraud now signals nothing. This is the one retraining cannot fix on its own.

All three look the same on a chart of falling accuracy, which is why they get treated as one thing. Teams then retrain on a schedule and wonder why it does not help. With concept drift, the labels you would retrain on were produced under the old rule, so they teach the model something that has stopped being true.

Skew detection

Compare the feature values coming in live against the ones the model was trained on. If the average payment amount was 40 in training and is 4000 today, something has broken upstream, meaning in the systems that feed you data. You know that without needing to know whether any single prediction was right.

That is why this is the fastest alarm you can build. It never waits for the label, the true answer, to arrive. For the fraud detector that matters more than anywhere else, because the label is a chargeback up to 60 days away. A broken upstream pipeline shows up here in minutes and in your accuracy numbers in weeks.

The rest of the monitoring layer

• Continuous evaluation: Score past predictions against the real outcomes as those outcomes arrive. Read the results knowing they are both late and skewed toward the cases a human bothered to review.
• Slice-based evaluation: One overall number hides a badly broken group. Accuracy holding at 94% while mobile users in one region sit at 60% looks perfectly healthy on the dashboard. Decide the slices up front, by device, region, tenant, or customer size, and alert on each one separately.
• Human-in-the-loop escalation: Send the cases the model is least sure about to a person, and keep their answers. The review queue then becomes labelled training data, which is a rare case of monitoring paying for itself.

If you want this layer in much more depth, including how a real incident gets investigated end to end, the observability post picks it up from here.

The anti-patterns

Most of these come from the paper Hidden Technical Debt in Machine Learning Systems, which is a decade old now and still describes the failure modes better than anything since. None of them are modelling mistakes. They are all structure, and they all get worse quietly.

CACE is the one worth learning first. It stands for changing anything changes everything. Drop one feature from the model and every remaining weight shifts to make up for it, so a change you thought was local moves results everywhere. That is why you test one change at a time, and why adding just one more signal is never as small as it sounds.

• Correction cascades: A second model added to fix the first model's mistakes, then a third to fix that one. Each layer buys a week and makes the stack harder to unpick. The real fix is retraining the base model, which is exactly the work every patch was avoiding.
• Pipeline jungles and glue code: The codebase turns into scrapers and adapters with a model somewhere in the middle. It grows a little at a time, so nobody ever decides to build it, and the only defence is deleting paths rather than adding them.
• Unstable data dependencies: You use a feature another team owns, and they redefine or retrain it on their own schedule without telling you. Snapshot the version you depend on, or accept that your model changes whenever theirs does.
• Configuration debt: Config files nobody reviews or tests, carrying as much logic as the code. In ML systems the thresholds and the feature list often live there, so the config holds the real decisions.

The fraud detector has a textbook example of a fifth problem, the feedback loop. Block a payment and it can never produce a chargeback, so a blocked fraudster and a blocked innocent customer look identical in your data forever. The model decided what it would later learn from. These loops are the hardest failure here to see, because the system looks like it is improving. Recommenders get the same thing, where you only learn about the items you chose to show. More data makes it worse rather than better. Breaking the loop means deliberately showing or allowing a small share of what the model would not have picked, so the next training set is more than a record of your own past decisions.

The LLM-era additions

Systems built on large language models, or LLMs, look like a separate world with its own vocabulary. Mostly they are these same patterns under new names, plus a few genuinely new pieces. Lining the two lists up makes the new stack much less mysterious.

Three of these carry a warning worth stating plainly.

• RAG quality is retrieval quality: If the wrong passage reaches the prompt, no model writes a right answer from it. Chunking, meaning how you split the documents, and the retrieval that searches them decide the result far more than the model does. Tuning prompts while retrieval is broken wastes weeks.
• A judge has to be checked against people: Using a model to score your answers only works if you first score a sample by hand and confirm the model agrees. Skip that and you get a confident number that measures nothing.
• The tool boundary is untrusted: When a model can call tools in a loop, treat every call as a separate system that can be slow, fail, or return rubbish. Timeouts, retries, and a hard limit on how many steps the loop may take. The runaway loop is the new failure mode.

If this is the layer you are working in, the technical architecture of agentic AI and how to evaluate AI systems with real code go a long way further into it.

How to choose

The list above is a catalogue, not a plan. Building all of it before you have a single model in production is the most expensive mistake in this space. It is also a common one, because buying the catalogue is much easier than building the judgement.

Start with the boring version. Write the rules by hand, ship them, and instrument them, meaning measure how well they actually do. This is the heuristic benchmark, and in the fraud detector it might be three thresholds and a blocklist of known-bad cards. Two useful things then happen. You have a score every model has to beat, and quite often you find out no model was needed.

After that, add each pattern when a specific pain turns up.

• Offline and live scores disagree: That is skew. Adopt a feature store.
• Nobody can rebuild last month's model: That is the notebook. Adopt a pipeline.
• Quality fell and nobody noticed for weeks: Adopt drift and skew monitoring.
• A release makes everyone nervous: Adopt shadow mode and a registry you can roll back from.

Every pattern costs real maintenance, so each one should be paying off a problem you have already had.

Most production ML problems are data dependency problems rather than modelling problems. That is why the fix is almost never a better architecture.

If you want the modelling side of this instead, the tracks in this app cover it directly.

• Feature Engineering: How features are built, and where the definitions that cause skew come from. Open the module.
• Model Versioning: Immutable versions, lineage, and promotion, which is what makes rollback boring. Open the module.
• Monitoring in Production: Drift, skew, and the alarms that catch a silent decline. Open the module.
• Eval-First Engineering: Building the evaluation before the feature, as a working habit. Open the module.

Sources

The primary references behind the patterns and anti-patterns above.

• Sculley, Holt, Golovin et al., Google: Hidden Technical Debt in Machine Learning Systems, NIPS 2015. The origin of CACE, correction cascades, undeclared consumers, glue code, pipeline jungles, and configuration debt. Link
• Lakshmanan, Robinson and Munn: Machine Learning Design Patterns, O'Reilly 2020. The 30-pattern catalogue behind much of the data, training, and serving sections here, including transform, bridged schema, cascade, reframing, keyed predictions, useful overfitting, and heuristic benchmark. Link
• Breck, Cai, Nielsen, Salib and Sculley: The ML Test Score, a rubric for ML production readiness and technical debt reduction, IEEE Big Data 2017. The basis for treating data validation and skew detection as testable requirements. Link
• Breck, Polyzotis, Roy, Whang and Zinkevich: Data Validation for Machine Learning, SysML 2019. How schema and distribution checks are run at ingest at Google scale, as part of TFX. Link
• Chip Huyen: Designing Machine Learning Systems, O'Reilly 2022. Reference for the drift taxonomy, continuous evaluation, and degenerate feedback loops. Link
• Google Cloud Architecture Centre: MLOps, continuous delivery and automation pipelines in machine learning. The maturity levels behind the pipeline pattern, and the retrain triggers of schedule, new data, performance degradation, and drift. Link