An agent demo needs to work once. An agent in production needs to fail safely, repeatedly, for years, while someone who was not involved in building it is on support.
Three things separate the two, and none of them are about the model.
1. A defined failure behaviour
The dangerous output is not an error – it is a confident wrong answer. An agent asked to extract a total from a document will return a number even when the document does not contain one.
So the agent must be able to decline. In practice that means a structured response with an explicit confidence or “not found” branch, and a rule about what happens next: escalate to a human, retry with a different prompt, or fail the job. Choose deliberately. The default – proceeding with whatever came back – is how an agent quietly corrupts a fortnight of records.
2. An audit trail of decisions
When a client asks why their invoice says what it says, “the model decided” is not an answer. Log, per invocation: the input, the model and version, the raw output, the parsed result, the confidence branch taken, and who approved it if anyone did.
This is unglamorous and it is the thing that makes an agent defensible in a review. It also makes the agent improvable – you cannot evaluate accuracy you never recorded.
3. A hard cost ceiling
Token spend is a runaway risk, not a line item. A retry loop that triggers on an edge case at three in the morning can produce a bill you find out about at the end of the month.
Cap it in three places: per invocation, per user or account per day, and globally. When a ceiling is reached the agent stops and alerts rather than degrading silently. This is straightforward and routinely omitted.
Human approval, by default
Our default is that anything which commits money, sends an external message, or changes a customer record requires a person to approve it. The agent drafts, summarises, extracts and proposes. The human confirms.
That sounds like it removes the benefit. It does not – the expensive part of the work was reading and composing, not clicking approve. What it removes is the class of incident where an agent sent four hundred emails before anyone noticed.
Where a rules engine wins
Worth saying plainly: if the task is deterministic, a rules engine is cheaper, faster, testable and does not hallucinate. Language models earn their cost on genuinely unstructured input. We have talked clients out of AI features more than once, and the projects were better for it.