A demo is a controlled environment. The inputs are clean, the state is fixed, the awkward edge cases are kept off-stage, and a human is standing by to nudge anything that wobbles. Under those conditions almost any capable model looks finished. Production removes every one of those supports at once — messy data, shifting APIs, no one watching at two in the morning — and a large share of the agents that dazzled in the demo simply fall over.
The reflex is to read the failures as a capability gap and reach for a bigger model. That is the wrong diagnosis. The model is not the variable that changed between the demo and the outage. What changed is that the system now has to absorb the model’s volatility — its tendency to take a different path on the same input — without a human catching every wrong turn. That is not a prompt you can write. It is a system you have to engineer.
A demo proves a model can. Production asks whether you can govern what it does.
Two kinds of debt
Conventional software fails predictably. Messy code is messy the same way every time you run it: you reproduce the bug, write a regression test, and the test holds. A probabilistic agent breaks that contract. The intent is stated, but the execution is generated fresh on each run, across a wide space of constraints — so the same request can succeed on Monday and fail on Tuesday in a way no prior test anticipated.
That volatility shows up on the books as two distinct costs.
The first is a stock: accumulated technical debt. Prompts, memory representations, tool schemas, and orchestration graphs pile up faster than anyone can validate them. Like all debt, it sits on the balance sheet quietly and compounds.
The second is a flow: the recurring operational tax of keeping probabilistic behavior inside bounds — retries, escalation paths, latency management, and the constant maintenance of guardrails. It never reaches zero. Every request pays it.
Plot the two against rising complexity and the curve bends the wrong way. Early on the burden is flat and easy to dismiss. Then the accumulated debt and the per-request tax compound together, and the cost of keeping the system reliable climbs into a zone where it outruns the value the agent produces.
Messy code is predictably messy. A messy agent states its intent and improvises the execution.
None of this is fixed by a better model. It is contained by better engineering, on three fronts at once: the architecture the agent runs inside, the governance that bounds what it may do, and the evaluation that tells you whether it is still working.
Architecture: build the control plane
Start with the architecture, because the most expensive failures are architectural.
Take the cheapest one to understand: the runaway loop. An agent calls a tool, the call fails to parse, and the agent reflexively retries the same prompt. The model returns the same kind of error. Nothing in the loop has any reason to stop, so it doesn’t — it just bills. An unmonitored loop on a frontier model can drain a meaningful amount of money per hour; left running over a weekend, it can reach into the thousands before anyone notices on Monday.
The instinctive fix is a cap in the application — max_iterations = 100. It doesn’t hold. A top-level agent that spawns sub-agents simply distributes the loop across children, each under its own limit, and the aggregate runs unbounded. You cannot reliably constrain a loop from inside the thing that is looping.
The control has to sit underneath. A single enforcement point — a gateway every model call passes through, beneath whatever framework the team happens to use — can detect recursive loops, hold a hard budget ceiling, and apply semantic guardrails, without rewriting the application above it. This is not a product to sell or a latency contest to win. It is an internal chokepoint whose only job is to make autonomy interruptible.
You cannot fix a runaway loop from inside the thing that is looping. The control has to sit beneath it.
Architecture also decides the shape of the work. The instinct is to wire steps in a line: intake, then a vendor check, then a policy check, then execute. It reads cleanly and it is fragile. Every handoff adds latency, a single failure anywhere snaps the whole chain, and any change forces you to revalidate the entire sequence end to end.
The resilient shape is a graph, not a line. Independent specialists run in parallel and converge on a validation gate before anything executes. It is cheaper, it is modular, and — most importantly — it contains the blast radius of a localized failure instead of propagating it. One specialist failing degrades one branch; it does not take down the run.
A chain fails at its weakest link. A graph routes around it.
There is a quieter architectural decision hiding in the step count. Look at how far deployed agents actually run before a human takes over, and the number is small — most hand back control within a handful of steps. That is easy to read as immaturity. It is better read as design. Teams are deliberately trading open-ended capability for controllability.
The pattern that makes it work is a bounded sandbox: the agent runs autonomously inside an explicit boundary — intake, processing, validation, a domain step or two — and then hits a human-in-the-loop checkpoint before any consequential action leaves the box. The point is not to maximize how much the agent can do alone. It is to keep each autonomous run short enough to stay legible, and to put a person at the boundary where the irreversible actions live.
Governance: graduated trust, not a switch
The second front is governance, and most organizations get it wrong the same way: they treat it as a binary. An agent is either locked down — read-only, useless — or fully trusted with production. Both settings fail. Over-restrict the simple agents and people route around them; shadow IT fills the gap. Under-restrict the autonomous ones and a probabilistic system gets the keys to things it can damage.
The fix is to stop treating trust as a switch and start treating it as a ladder. Authority is granted in graduated tiers, each adding controls as the scope widens:
- Observe — read-only display. Basic scoped access, authentication, usage logging.
- Advise — read-only plus generative recommendations, like drafting code or an email. Adds hallucination testing and domain-specific evaluation.
- Act with approval — may write data or send communications, but a human is in the loop. Adds workflow audit trails and incident-response procedures.
- Fully autonomous — independent execution, earned last and fenced hardest: strict guardrails, continuous red-teaming, automated rollback, circuit breakers.
A second test runs orthogonal to the tier, applied per action rather than per agent. Green actions are low-impact and reversible — routing a ticket, classifying a document — and can run autonomously. Yellow actions carry moderate impact and execute only inside hard limits, downgrading to propose-only the moment a limit is exceeded. Red actions are high-risk and high-blast-radius — writing to a production database, changing master data — and require explicit human authorization every time.
The mandate underneath both is one sentence: do not scale an agent’s autonomy faster than you can govern its authority. Autonomy and scope are assessed independently — a highly autonomous agent with a tiny blast radius is fine; a low-autonomy agent pointed at production master data is not.
Don’t scale an agent’s autonomy faster than you can govern its authority.
Evaluation: telemetry over intuition
The third front is knowing whether any of it is working — and here the default is to trust a feeling. The output looks right, so it ships. With a probabilistic system, “looks right” is not a measurement; it is a hope.
Replace it with telemetry. Instrument the agent at the level of individual spans — every retrieval, every tool call, every generation — and score each against explicit dimensions rather than a gut read. Does the answer stay faithful to the context it was given, or does it wander past it? Does it actually address what was asked? Were the retrieved chunks relevant, and did they include everything the answer needed? Open evaluation frameworks formalize exactly these dimensions; the discipline is to hold each above a floor and alert when it slips.
The floors have to sit high, because a threshold that sounds strict can still be far too lax. A faithfulness bar that tolerates even modest drift will let a meaningful share of unsupported claims slip through — an unacceptable rate for anything consequential. And automated scoring is a complement, not a replacement: across deployed systems, the large majority still keep a human as the primary reviewer, with model-based judges running alongside them rather than instead of them.
Two adjacent decisions belong to the same front. The data underneath has to be clean and current, because retrieval quality is a ceiling on answer quality — fixed-size chunking that cuts ideas in half, paired with vector-only search, will quietly cap how good the system can ever be, where semantic chunking and hybrid retrieval lift that ceiling. And retrieval itself should be routed, not uniform: classify the query first, send simple lookups down a fast path, hard multi-step questions down a heavier one, and questions that fall outside the knowledge base to the model’s own parametric knowledge — skipping retrieval entirely rather than forcing weak context into the answer, which produces fewer hallucinations, not more.
The right mental model for all of this is onboarding, not installation. You do not hand a new hire root access on day one. You widen their scope as they earn trust, and you keep reviewing the consequential work. An agent deserves the same.
“Looks right” is not a measurement. It is a hope.
The whole system
Put the three fronts together and the result looks less like a clever model and more like a piece of industrial infrastructure — which is the right intuition. A power grid does not try to make its energy source less volatile. It engineers a chain of controls around the volatility, so that what reaches the load is stable.
The same chain maps onto a governed agent. Clean, de-siloed data is the energy source. Adaptive retrieval is the transformer, routing and conditioning the supply. The enforcement gateway is the circuit breaker, tripping on loops and runaway budgets before they cascade. Tiered governance is the control room, defining who may pull which levers. And continuous evaluation is the bank of telemetry dials, monitoring the output without interrupting it. No single element makes the system reliable. The chain does.
Agentic engineering is shifting from specifying deterministic logic to governing probabilistic action.
What this asks of you
This is the discipline the durable deployments share, and it is unglamorous on purpose. It accepts that the model is volatile and refuses to pretend otherwise. It spends its effort not on coaxing a better answer out of a prompt, but on building the structure that makes a volatile component safe to run in production.
The teams that internalize it end up in a small minority — and a handful of well-governed agents tends to be worth more in production than a sprawling estate of clever, fragile, over-permissioned ones. The lesson compresses to two instructions. Architect for resilience. Design for oversight.