Architectural patterns for building maintainable, testable software when an LLM sits on the critical path. The worked example throughout is a live-streaming platform that uses a model to moderate content as it airs.

​

Key takeaways

• An LLM breaks three things you normally rely on: the same input can return a different output, a wrong answer looks identical to a right one, and every call costs time and money. So don't wire it straight into your code.
• Treat the model as a slow, flaky external service. Put it behind a boundary and keep the nondeterminism on one side of that line.
• Nothing else reads raw model text until it passes a schema check. On a bad reply, re-prompt with the exact validation errors; after a few retries, fall back.
• Prompts belong in version control, reviewed in pull requests and tied to the eval run that justified them. Changing a prompt is changing your system's interface, even though it reads like prose.
• Tests and evaluations catch different failures. Tests check the deterministic machinery on every commit; evaluation scores the model against a labelled set on a schedule and gates prompt changes.
• Failure is the normal case here, so design for it. Fall through a chain of cheaper options (retry, smaller model, regex blocklist, default) down to a per-stream default that still beats the broadcast deadline — a verdict that lands after the clip airs is useless.
• Keep the model off the critical path. Run it on a side branch so its latency and outages never reach the viewer.

The problem in production

We added content moderation to our live-streaming product. The idea was simple enough: feed the stream's transcript and the chat into a language model, and let it flag anything that breaks the rules: slurs, doxxing, harassment. We pointed it at a big folder of old recorded clips first. It caught what we wanted it to catch. So we turned it on for live streams.

Live is where it got weird.

One streamer got cut off and dropped to a holding slate. Why? He'd said a sponsor's name out loud during a paid read, and the model decided that was spam. We blacked out the one segment that was literally paying for the broadcast.

Another night, a slur went out, and the model said nothing. The speech-to-text had garbled it into something slightly misspelt, and the model never connected it back to the real word, the way anyone half-listening would have.

And then there was the worst kind of ticket: the ones the on-call engineer couldn't reproduce. Same transcript, run again by hand, came back clean. No flag, no error, nothing to point a finger at.

Here's the part that gets under your skin. We hadn't changed the code. It was the same moderation logic that sailed through every test we threw at it. The only thing that was different was that the feature now leaned on something that doesn't behave like code.

Think about what you expect from a normal function. Same input, same output. And when it can't do its job, it throws an error so you know it broke. The model gives you neither. Ask it the same thing twice, and you might get two answers; it can stall for a second or two before it replies, and when it's wrong, nothing in the answer tells you so.

So that's the problem I want to dig into: how do you get back the testing and debugging you take for granted everywhere else, when one piece of the system answers differently every time you ask the same question?

Treating the model as an unreliable dependency

It's worth being specific about which assumptions the model breaks, because each one comes back later as a pattern you'll need.

There are three. The first is that the same input won't reliably give you the same output. Sampling temperature, a quiet update on the provider's end, things you never touched yourself, any of them can shift the answer between one call and the next.

The second is that the model can hand you something that reads perfectly and is simply wrong. There's nothing in the response to mark it as wrong; it looks exactly like a correct one. The third is that calls are neither fast nor free. A single one can take a few seconds, and every one of them lands on the bill.

Wire a feature straight into the model, and all three of these leak into the rest of your code. Test coverage quietly falls apart because you can't assert on an output that varies from run to run. Error handling stops making sense, because the failures aren't exceptions you can catch, they're believable wrong answers. And latency and cost go dark, buried inside what looks like an ordinary function call.

So don't treat it like part of your code. Treat it like what it is: a slow, flaky external service. Put it behind a boundary and keep the mess on one side of that line. None of these tools are new. Schema validation, retries, fallbacks, mocking, logging, the same stuff you'd reach for with any unreliable dependency. We're just aiming it at a language model.

A quick note on the example I'll keep coming back to. The platform pipes the live transcript (whatever speech-to-text produces) and the chat through a model that's doing two jobs at once: moderating content and generating captions and chapter markers. Moderation is the one I care about here, because it's the one that can do something you can't undo, like killing a live stream. The model also sits next to the video path rather than inside it, which matters for reasons I'll get to later when we cover where to put it.

Isolating nondeterminism

The whole game is to make the unpredictable part as small as you can and wall it off from everything else, so the rest of the system stays boring and predictable.

In moderation, everything after the verdict lands is predictable in exactly that way. Flipping MediaLive over to a slate, dropping a warning into chat, writing the audit record, checking whether a confidence score clears your threshold: all of it runs on fixed rules.

You can write ordinary tests against these, and they'll come back the same every single time. The model lives outside this part. It produces a verdict and hands it off; a thin boundary layer either turns that reply into a typed value the rest of the code can work with or throws it out.

That split is what makes the feature testable at all. People ask how you're supposed to unit-test something that sits on top of a model, and the answer is that you don't, at least not in those tests. You swap in a stub that returns a fixed verdict and test everything around it.

Does a cut_to_slate verdict actually fire the right actions? Does a low-confidence one get ignored the way it should? When the boundary rejects a reply, does the fallback kick in? Whether the model itself is any good is a separate question, and that's what evaluation is for, not unit tests.

Validating model output

We ask the model to read a chunk of transcript, classify it, and return JSON. It returns JSON the same way it returns everything: as plain text, with no guarantee the text actually parses.

Most of the time it does. When it doesn't, it's usually that the model added a line of explanation first, or fenced the JSON in ​```json, and json.loads throws on what should have been a routine path. The replies that parse but are still wrong cause more trouble. The action field comes back as cut rather than the cut_to_slate we defined.

Confidence arrives as "high" where the code expects a float. A category shows up that was never in the allowed set. None of them throw, so the object looks well-formed, the control plane acts on it, and whatever breaks later gives no obvious sign that the model was the cause.

We handle this by refusing to let any other code read the reply until it passes a schema check. The schema requires an action from a fixed set (pass, blur, warn_chat, cut_to_slate), a category from a second fixed set, and a confidence that parses as a number.

A reply that misses any of those is rejected, and we re-prompt the model with the exact validation errors so it can correct itself. That repeats up to a small number of retries. If nothing valid comes back by then, we stop and fall back. The diagram below shows a failed first attempt and the retry that cleared validation.

The validator is the only part of the system that deals with raw model text. After that, everything works with a typed ModerationDecision, so the rest of the code never needs to know what string the model produced.

Versioning and reviewing prompts

The moderation prompt spells out the actions the model is allowed to take, what counts as spam, and a handful of examples that show where the lines are. That block of text is what decides whether a stream gets cut. Which makes it part of your system's interface, every bit as much as the code is, even though it reads like prose and tends to get a fraction of the scrutiny.

Editing the prompt is just as risky as editing code. Tighten the hate-speech instruction a notch too far, and the system starts cutting streams over sarcasm, and you find that out hours after the build went green, when the streamers start complaining. So prompt changes go through the same pipeline as everything else.

They live in version control. They get reviewed in a pull request. Each one is tied back to the evaluation run that earned it its place. When moderation starts behaving differently in production, the first question is always which prompt version is live, and you should never be in a spot where you can't answer that.

Testing versus evaluation

Two separate checks guard this feature. They catch different kinds of failure, and neither one stands in for the other.

A green CI run tells you the machinery around the model holds together. It confirms the cut fires when it should, that malformed replies land in the fallback, and that the confidence threshold gets applied. What it doesn't tell you is whether the model is still catching slurs. A prompt change that quietly halves your recall will sail straight through CI, because nothing in there is measuring recall in the first place.

That gap is what evaluation fills. You take the real prompt, run it against a labelled set, score it, and hold the result up against thresholds you committed to ahead of time: recall on hate speech north of 0.95, a false-cut rate under half a percent. The two checks aren't interchangeable. They run on different schedules, they measure different things, and in production you want both.

Failure handling

Model calls time out. They hit rate limits. They come back with something you can't use. And a live stream doesn't get to pause while you sort it out. Failure isn't an edge case here; it's the norm, so you plan for it from the start instead of bolting something on after the first incident.

The way through is a chain of fallbacks, each one faster and dumber than the one above it:

• the validated model verdict, when everything works;
• if parsing fails, one retry with the schema errors fed back into the prompt;
• if that retry fails or times out, a smaller and faster model;
• below that, a blocklist: plain regular expressions over known slurs and patterns, no model involved;
• and finally a default, for when even the blocklist can't decide in time.

The default at the bottom is the one to think carefully about, and the right answer depends on the channel. On a kids' channel you probably want it to cut whenever it can't decide. A news feed is the reverse: pulling a live broadcast automatically is worse than letting a borderline clip through, so the safe default there is to leave the stream up and route the clip to a human queue.

Whatever you choose, it has to respect the deadline. A fallback only helps if it returns before the clip airs. A correct verdict that arrives afterward is useless, because the thing it was meant to stop has already gone out to viewers.

Logging and observability

When the model cuts a sponsor read and the streamer asks why, you're reconstructing the decision after the fact, and a live pipeline gives you very little to reconstruct it from. A normal request would still be sitting in a queue or a log you could replay, but a live stream has already moved on, and there's nothing to re-run. The only record of what happened is whatever you logged at the time.

For each verdict, the system records the transcript window and any frame context that went in, the prompt and its version, the raw model output, the decision that came out of parsing (or the fallback, if one took over), the action taken, and the latency and cost of the call.

The two fields people cut first when storage gets tight are the raw output and the input window. Those happen to be the two you need to tell a model error apart from a parsing error, so dropping them leaves an incident review with no way to decide which one it was actually looking at.

Deciding where the model fits

Up to now, most of this could apply to any system that calls a model. The streaming context changes that, and it forces two decisions: whether a model belongs in this feature at all, and if it does, where in the pipeline it should sit. Latency forces your hand on placement, so start there.

The live relay encodes each frame, packages it, and delivers it to viewers on a budget measured in milliseconds. Low-latency HLS targets three to five seconds from camera to screen. Amazon IVS real-time targets under three hundred milliseconds. A model call doesn't fit inside numbers like that.

On a good day it returns in a fraction of a second; on a bad day, several seconds; and sometimes it doesn't return at all. Put it directly in the relay and two things follow: the stream's latency is now bounded by the model's worst response time, and if the model's API gets throttled, the stream stalls with it.

So the model runs on a side branch. It processes transcripts and segments a few seconds behind the live edge, after the relay has already delivered them. Its verdicts go to the deterministic management plane, which acts on the ones that arrive in time and ignores the ones that come late or not at all.

The model can still trigger real actions, like cutting a stream or flagging a clip. It just isn't something the video has to pass through on its way to viewers.

The failure mode to avoid is putting the model anywhere its own latency and outages turn into the viewer's. The test is simple: if a feature can't send the next segment until the model replies, it's on the critical path and has to come off.

That leaves the first question, the one it's tempting to skip: does this feature need a model at all? Moderation does, because the task is open-ended. A blocklist can't read tone or context, and it won't catch a slur deliberately misspelt to slip past it. But plenty of decisions in a streaming stack aren't open-ended, and for those a rule or a table lookup is faster, cheaper, more predictable, and incapable of hallucinating.

Adding a model in one of those spots buys you cost and a new failure mode and nothing else. Two things worth settling before you reach for one: is the task genuinely fuzzy, and can you tolerate the occasional wrong answer? If a rule covers it, use the rule.

In this design the model is just one dependency, and most of the work is in the code around it that keeps it contained:

• The validator is the only place raw model text ever exists; it either returns a typed ModerationDecision or rejects the reply.
• After that, the management plane is fully deterministic, so unit tests cover it with the model mocked.
• Evaluation measures model quality on a labelled set, and gates whether a prompt change is allowed to ship.
• Failure handling falls through a chain of cheaper options, down to a per-stream default.
• Logging keeps enough of each call to reconstruct it later.
• And placement keeps the model on a side branch, off the video path, where its latency and failures can't reach the relay.

A few things this deliberately leaves out:

• The speech-to-text layer that produces the transcript is a separate system with its own failure modes.
• Cost depends too much on your traffic and choice of model to say anything useful in the abstract.
• The hardest part was never the architecture. It was defining "content that breaks the rules" precisely enough to score a model against, which is mostly a policy problem, not an engineering one.

Summary

The architecture here is fairly stable. The prompts and thresholds aren't, so keep both in version control and run them against an evaluation set on a regular schedule.