How an AI Agent Works at Scale · Hanh D. Brown

Ask a production agent one question and a stopwatch starts. Two seconds. That is the whole budget, and a caller notices anything longer as an awkward silence. Artificial Intelligence (AI) systems built for that window look almost nothing like the chat box most of us picture.

Inside those two seconds, a great deal happens at once.

How many models does an AI agent use in one conversation turn?#

Ten to fifteen. Not one. That number surprises almost everyone, including people who buy these systems for a living. The picture in our heads is a single clever machine, thinking hard about a single question. It is the wrong picture. Here is the shape of one turn instead.

The calls split roughly three ways. About a third are big reasoning models. They do the heavy thinking in one or two passes. Another third are small sorters that decide what kind of request this even is. The rest are guesses made early, in case they turn out to be needed.

Each family costs differently and fails differently. The spread is deliberate. It is also why the real cost question is per token rather than per hour, because a system that sent every task to the most expensive model would price itself out of the market by lunchtime.

Here is the part worth holding onto. You see none of it. One tidy reply lands on the screen. The ten or fifteen calls behind it leave no trace.

Why does a voice agent have to run models in parallel?#

A person on a phone will not wait. One to two seconds is the honest ceiling for a spoken answer, and past that the silence stops feeling like thought and starts feeling like a dropped call.

That one constraint reshapes everything upstream. A coding assistant can take its time and nobody minds. A voice agent cannot. So work that a slower system would do in order has to happen at the same time.

So the parallelism is not just across tasks. It runs across listening, thinking, and talking. While the agent is still hearing you, it is already drafting. While it is talking, it is listening for you to cut in.

Four things run at the same time inside one turn.

• Listening. The agent hears you and drafts its reply in the same breath.
• Speaking. It listens for you to cut in.
• Guessing. It fetches an answer it may never use.
• Hearing twice. Two transcribers run side by side, and the system trusts whichever one is strong where it counts.

The doubling up is not waste. The best model for clear speech invents words during silence. The model that handles silence well is weaker on speech. So run both.

Like a kitchen where the sauce goes on before anyone orders it, the waste is the point. You trade a little spoiled work for a table that never waits.

How do you make an AI agent reliable at scale?#

You stack checks. Reliability here is not a hunt for one model that is finally correct. It is an admission that no model is. Then you build the machine around that fact.

The arithmetic is the argument. Take a step that is right about ninety percent of the time. Check it. Then check the checker. Failures multiply downward instead of piling up, which is the whole reason a chain of unreliable parts can end up more trustworthy than any single part inside it.

You land near three or four nines. Three nines alone is one bad outcome per thousand calls, rather than one per ten.

Research points the same direction. Sample a problem several times, take the answer that keeps recurring, and accuracy rises. The compute bill rises too. Recent work therefore spends those extra samples only where they change the outcome (Reliability-Aware Adaptive Self-Consistency, 2026).

Then there are the monitors. Picture a floor supervisor who never sleeps. These evaluators read every conversation and flag the handful that need a person. A person now reads five conversations instead of ten thousand, and reads the right five.

Simulations complete the picture. Before a change ships, it gets run against many personas, several languages, and deliberately difficult callers. This is the same instinct behind treating the agent like an employee, except the performance review runs before the first shift.

How do AI agents handle payment data safely?#

Card numbers never touch the model. That is the whole rule. It is not caution, it is compliance. No large language model provider carries the certificate needed to handle payment card data, so the number takes a different road.

In practice a separate certified system runs the payment while the agent runs the conversation. Two rooms. One locked door. The agent knows a payment happened, and it never once sees the card that made it happen.

National guidance points the same way. The National Institute of Standards and Technology (NIST) is the United States agency that writes technical standards. Its generative AI profile lists twelve risk areas and more than two hundred suggested actions (NIST AI 600-1).

Two hundred actions sounds like bureaucracy until you read a few. Most carry the same instinct as the payment rule. Decide in advance what the model is never allowed to hold.

I have sat in the meeting where somebody asks whether the model could just handle the card. The answer is no. The reason is not judgment.

How does outcome pricing work for an AI agent?#

You pay per result. Not per seat at a desk, not per conversation, but per resolved problem or completed sale. That quietly rearranges whose interests point where.

Think about what that does to the vendor. A vendor paid per seat wants more seats. A vendor paid per resolution wants your customer off the phone happy, because an unresolved call earns that vendor nothing at all. The incentive flips.

It also flips the cost conversation. One call might save a sale worth a hundred dollars. It might save a relationship worth a thousand. Against that, squeezing pennies out of the model choice is the wrong fight. Quality beats cost whenever the outcome is genuinely valuable.

There is a sharp edge in that logic. Say it plainly. If paying for outcomes sounds like a bad deal to you, the honest reading is that your outcomes may not be worth much.

Why do most multi-agent systems disappoint?#

Splitting the work usually starves it. The instinct feels right. One agent per job looks tidy on a whiteboard, and tidy whiteboards are how most of these decisions get made.

What happens instead is context loss. A triage agent hands work to a task agent. Each one now knows less than a single agent would have known. The seams are where the quality goes.

There is a sharper version of the diagnosis. Teams often build multiple agents because they have multiple teams, so the system ends up shaped like the organization chart instead of the problem. That is a familiar failure wearing new clothes, and it is close to the blind spot in most orchestration pitches.

Sometimes several agents genuinely fit, and the condition is strict: the jobs have to be truly separate, with each agent carrying its own context from the first step through to the last. For quality alone, though, better context engineering usually beats splitting the work.

Like a family dinner cooked by three people in three kitchens, more hands did not help. Somebody still has to know what dinner is.

I think about that at my own table. The useful help was never the person who took one task off my list. It was the person who understood the whole evening, and saw what the list was missing. The agent is not the model. It is the orchestra around it. Somebody has to hear the whole piece.

Source: Zack Reno Wedeen, Head of Product at Sierra, on the Max Agency podcast by LangChain.