System health and answer quality
Every piece of production software requires someone to watch how it runs: metrics, logs, alerts, a person who reacts. That did not start with models and will not end with them.
The difference with a model is one of degree, not of kind. In deterministic software, a result is largely machine-checkable — you compare it with what was expected, the test passes or it does not, and the operational signals tend to move with correctness. In a system with a model, most results have no such test. An answer can look right, pass every check on format and schema, and still be wrong — and the only check that catches it is somebody reading it.
That is the whole difference: monitoring keeps the system working; it does not establish that what came back makes sense.
Three sources of uncertainty
We should distinguish three sources of uncertainty: sampling, natural-language input and provider updates. None of them arrives automatically with the others, and each calls for a different response.
**Sampling.** The same request can produce a different answer on the next run — behaviour is a distribution, not a function. This applies where sampling is switched on; at a temperature near zero and with a fixed answer schema the problem is small, though it never disappears: a distribution is not a point.
**The scope of the input.** Natural-language input admits cases the design has not anticipated. It applies where the input is free text; behind a strictly validated form the range is as narrow as in any other program.
**Change without a change in your code.** The provider updates the model version your application uses. Your code is unchanged, there has been no new deployment, and the program behaves differently. This applies to deployments where the model comes from an outside provider. With the weights running on your own hardware this problem disappears — at the price of owning the updates, the quality and the security yourself.
What tests establish
A test suite over deterministic code establishes something about the cases checked, within the limits of the specification; it does not prove that the whole behaviour is correct. Over a model it establishes exactly the same thing, with two differences. First, the result belongs to a particular model version, and that version is part of neither your repository nor your deployment — unless the model runs on weights you hold. Second, with sampling on, the same input may come out differently on the next run, so one green test describes a sample of the behaviour rather than the behaviour itself.
The shape of failure changes
In the old world, a rare input triggered a bug inside code whose intent we still knew: we knew what the program was supposed to do, and we could see where it went wrong. Here failures arrive as behaviour nobody anticipated. A phrasing the designer never imagined while writing the instructions. A judgement that sounds credible and is wrong. A case for which the design has no place.
You cannot review code for a case you are unable to conceive of. Such cases come out of live use: in production, in tests with users, in a conversation that shows people are using the program differently from what we assumed.
Why a cadence, not just reaction
Work on a system with a model shifts from "repair when it breaks" to "read the output while it runs": watching the live application, capturing what actually happened, folding those cases back into the test or eval set, and repeating.
Reacting to a failure is not enough, because with a model the failure does not announce itself. There is no exception, no error log, no dip that draws the eye — there is an answer that looks right and is wrong. Finding it means reading outputs, and reading outputs only means something if it happens regularly: a one-off review tells you about the day you did it.
The cadence is not universal and there is no single correct one. It is set by the damage a wrong answer does and by how long its consequences take to become visible. It will differ for a system that suggests something to an operator and for one that sends documents out on its own.
The scope of that change is narrower than it sounds. Routing, storage, retries, contracts and the tools the model calls stay deterministic and can still be tested using the existing methods. What needs oversight is above all the model's own behaviour and the way the rest of the application consumes its answers — because the deterministic shell looks like the whole program.
The honest counter-argument
Each of the three sources of uncertainty has a partial remedy, and none resolves it completely.
Temperature zero, constrained decoding and response caching narrow the spread of answers, but they do not narrow the input space. Replay logs let you reproduce a specific case, not gain certainty about the next one — and with sampling, a replay does not always return the same output. Hard fallback procedures on critical paths protect the deterministic shell, not the model.
None of these techniques settles the problem of the provider changing the model: your repository stays untouched and the behaviour shifts.
We still have to decide who reads the output. Part of the monitoring and the preparation of test cases can be delegated to models — but that replaces one question with a harder one: whether the agent deciding what counts as a fault is trustworthy enough to decide it alone.
What this means for your company
If you budget an AI deployment as a one-off cost, that budget has no room for oversight of the outputs — the work that does not end at delivery. It is not an addition to the project; it is a line that remains after the project closes.
Settle two things, one a standard and one a person. The standard: which answer is unacceptable, and what happens when the system returns one — does it stop the process, or only write an event to a log. The person: who reads that, and who has the authority to stop the system when the results stop adding up.
Without those two answers, oversight is a declaration. The system runs until somebody notices something by accident — and the somebody who notices is usually a customer, not the team.