Every team deploying agents reaches the same compromise: "the agent drafts, a human approves." It feels responsible. It ships past the risk-review meeting. And it contains a scaling trap that most teams discover only after the queue is on fire.
Human-in-the-loop fails as a permanent architecture because human attention scales linearly while agent output scales with compute — and because review, as usually practiced, produces decisions but not lessons.
The queue math
Say your agent handles 2,000 tasks a week and a reviewer needs three minutes per task. That's 100 hours of review — two and a half full-time humans doing nothing but supervising one agent. Double the agent's volume and you're hiring. The economics that justified the agent evaporate into the review layer. Worse, the review layer becomes the latency floor: the agent answers in seconds, then waits hours for a human, and users experience the human's response time, not the agent's.
Attention decay
The deeper problem isn't cost — it's that review quality collapses under volume. When 95% of items are fine, reviewers learn that approving is almost always correct, and attention drifts. This is well-documented in every domain that's tried it (radiology queues, content moderation, code review) and agents are no exception: the catch rate on rare bad outputs falls as the queue grows. You end up paying full price for review while receiving a fraction of the protection, with an audit trail that says "a human approved this" on the exact items where the human was rubber-stamping.
Review without learning
Here's the failure I care most about, because it's the fixable one. A reviewer catches the agent misquoting the refund policy. They fix the response, click reject, move on. Next Tuesday a different reviewer catches the same mistake. And the Tuesday after. Each rejection was a correct decision and a wasted lesson — the knowledge of what went wrong lived in the reviewer's head for ten seconds and went nowhere. The org is paying humans to repeatedly patch the same hole instead of paying once to close it.
That's not a staffing problem; it's a missing-feedback-loop problem wearing a staffing costume.
What humans in the loop should actually do
Move humans up one level of abstraction. Instead of reviewing outputs, review changes to the system that produces outputs:
- Escalations: the agent flags its own low-confidence or high-stakes cases — a small, information-rich queue instead of a firehose.
- Audit samples: a random 2–5% of runs, scored properly, keeps an honest baseline and catches what self-flagging misses.
- Improvement approvals: when the system clusters failures and proposes a prompt or policy fix, a human reviews that — one decision that eliminates a failure class, validated by regression gates before it ships.
Ten thousand micro-approvals become fifty meaningful reviews a week. Humans do what they're uniquely good at — judgment about direction — and the loop does what it's good at: making sure nothing gets reviewed twice.
Keep the training wheels for as long as trust requires. Just make sure that while the human is in the loop, the loop is actually learning from them — otherwise you haven't deployed an autonomous system, you've hired a very fast intern with amnesia and assigned your best people to follow them around.