The Pipeline: ASR, LLM, and TTS in Real-Time

Every time someone says a sentence to an AI voice agent, three systems have to transcribe it, reason about it, and speak a reply before the silence feels wrong. That entire chain has to finish in under a second, because human conversation tolerates roughly one second of quiet between turns before it starts to feel broken, according to NICE's technical breakdown of how these systems work.

The pipeline runs in five overlapping stages. Audio arrives through telephony (SIP or a contact-center platform), streaming speech recognition (ASR) converts it to text using partial hypotheses rather than waiting for full sentences, and a large language model reasons about what was said, ideally grounded in your business knowledge so it doesn't fabricate answers. Then orchestration kicks in: the agent takes action through tool calls, checking a calendar or issuing a refund. Finally, neural text-to-speech streams the response back, with modern TTS engines producing first audio in as little as tens of milliseconds.

Here's the part most buyers miss: these stages can't run one after the other. A system that waits for each stage to finish, as NICE puts it, "would answer seconds too late to feel human." Everything has to stream and overlap, which is why latency is a systems property, not a model property. It's determined by architecture choices across the whole stack, not by which LLM the vendor happens to use.

The numbers have improved fast. End-to-end response latency has dropped substantially since 2024, with 2026 benchmarks typically reporting a median near 680ms and the best systems well below that, per aggregated industry benchmarks. Sub-800ms is generally considered the threshold for natural dialogue. One caution: NICE openly notes that latency figures in vendor marketing are often engineering frameworks rather than measured results, so measure end-to-end latency yourself during any proof of concept.

Barge-in handling is where usable tools separate from frustrating ones. When a caller interrupts, the agent must halt its TTS playback "within a fraction of a second," and it must also distinguish real interruptions from backchannels like "mm-hm" and thinking pauses from finished turns. Get this wrong and the agent either talks over people or hangs on dead air.

The benchmarks worth demanding from any vendor:

• End-to-end response latency under 800ms, measured live, not quoted from a datasheet
• Barge-in halt time well under a second, tested with real interruptions
• Speech recognition accuracy of 97%+ for English across accents and background noise
• Grounding and retrieval built in, since it's the primary defense against fabricated answers

At AIQ Labs, we've found this is exactly where the work lives: our voice agents handle interruptions, mid-call actions like transfers and holds, and off-script moments because the pipeline underneath was engineered for them, not bolted on. A better LLM doesn't fix a badly engineered pipeline, and most real-world quality problems trace back to latency, turn-taking, and integration long before they trace back to the model.

Benchmarks That Matter: Latency and Accuracy

Ask a vendor for their latency number and you'll usually get a marketing figure, not a measurement. That distinction matters more than almost anything else on a spec sheet, because latency is the single biggest reason callers hang up on an AI voice agent platform that otherwise works fine.

Here's the benchmark context. Human conversation tolerates roughly one second of silence between turns before it starts to feel laggy, according to NICE's technical breakdown of how voice agents work. Against that ceiling, the industry has made real progress: median end-to-end response latency has fallen sharply since 2024, with 2026 figures typically reported near 680ms and the best production systems faster still, per 2026 benchmark data. The practical threshold for buyers is sub-800ms end-to-end latency, which is where dialogue stops feeling mechanical.

Notice the phrase "end-to-end." That's the part vendors blur. An AI voice agent is a pipeline: speech recognition (ASR) feeding an LLM feeding text-to-speech (TTS), all running as overlapping streams over telephony. A vendor might quote TTS time-to-first-audio while staying quiet about LLM reasoning time or telephony overhead. NICE is unusually candid on this point: their own latency figures are an engineering framework, not measured benchmarks, and they recommend buyers insist on measuring end-to-end response latency during any proof of concept. Do exactly that. Call the agent yourself, interrupt it mid-sentence, and time the silence.

Interruption handling deserves its own test. Barge-in, where a caller talks over the agent, must halt TTS playback within a fraction of a second. A system that keeps talking over you fails the demo, no matter what the latency dashboard says.

On accuracy, the key shift is what you measure. Word error rate is the wrong yardstick. ASR accuracy is now largely solved, typically 97%+ for English, but a caller doesn't care whether the agent transcribed them perfectly; they care whether the appointment got booked. The better metrics are:

• Task success rate: call resolution accuracy for well-configured agents typically runs in the low-to-mid 90s percent, and that's the number that maps to business outcomes.
• Intent detection: baseline accuracy improves meaningfully once the agent is grounded in a well-trained knowledge base.
• Failure behavior: ask what the caller experiences in each failure mode and what automatic fallbacks exist. The answers reveal operational maturity far better than uptime marketing.

One caution on the numbers themselves: several widely cited benchmarks trace back to vendors measuring their own platforms, so treat them as directional rather than gospel. The defensible move is to run a proof of concept on your own call traffic and measure task completion, latency, and abandonment against these published ranges. Agents that answer within two seconds keep abandonment in the low single digits; let callers wait 30 seconds and it climbs past 20%. That gap is the whole business case for caring about benchmarks, and it's the same standard we hold our own voice deployments to at AIQ Labs, where we measure resolution and latency on live call volume rather than demo traffic.

Choosing the Right Platform Tier for Your Business

The AI voice agent market has sorted itself into three clear tiers, and picking the wrong one is the most common way buyers waste money. The structure matters more than any individual vendor, because each tier assumes a very different level of technical resources on your side.

According to MarketsandMarkets, the market has crystallized into three distinct categories:

• Developer infrastructure platforms like Vapi, Retell AI, and Bland AI. These are API-first and self-serve, built for teams with engineering staff. If you have developers on payroll, this tier gives you maximum control.
• Managed enterprise platforms like PolyAI, Parloa, Cognigy, and NICE. These deliver turnkey deployments with months-long implementations and high containment claims. They suit large organizations with complex compliance needs and the budget for a formal rollout.
• No-code platforms like Synthflow and Goodcall. These let small businesses deploy a working agent in hours without writing code. The tradeoff is less customization and thinner integration depth.

So which fits you? Two questions settle it. First, do you have engineering resources in-house? If not, the developer tier will stall in a proof of concept that never ships. Second, what's your call volume and complexity? A dental office handling appointment calls has very different needs than a bank handling account actions, and research shows that scalable, cost-effective platforms have now made voice AI accessible to businesses that were priced out even two years ago.

There's also a deeper shift to understand before you buy. Early voice AI was pitched as IVR replacement: a nicer menu tree that routed calls. That era is over. The current generation of agents runs an agentic loop (perceive, ground, plan, act, observe) that lets them handle entire calls and take real actions like refunds, bookings, and account updates. They can recover when a tool call fails and decide when escalation to a human is the better move. When you evaluate vendors, ask whether the platform actually closes the call or just transfers it. The cost difference is stark either way: voice AI typically runs well under a dollar per interaction versus several dollars for a human agent.

This is where a partner model earns its keep. Businesses without engineering teams often get the best results from a managed approach, which is why AIQ Labs builds and trains voice agents as done-for-you AI Employees — receptionists, SDRs, dispatchers, intake specialists, and 99+ other roles — rather than handing over raw infrastructure. The platform tier you choose should match who's going to maintain it, because the one certainty is that today's best model stack won't be next year's.

If you want to hear a managed voice agent handle real call volume before committing to a tier, take the two-minute Fit Check and book a free 30-minute strategy call with AIQ Labs.

The Buyer's Checklist: Vetting Vendors Beyond the Demo

Every vendor demo sounds great. The agent answers instantly, handles interruptions gracefully, and books the appointment on the first try. The problem is that demos are staged, and vendor selection, not adoption timing, is what analysts call the most critical business challenge in voice AI today.

So pressure-test the platform against real conditions. Here's the checklist that separates mature vendors from polished marketing.

• Model flexibility. There is no single best ASR, LLM, or TTS. The right components vary by language, latency tier, cost envelope, and use case, so ask which speech, reasoning, and synthesis models the platform supports and how quickly it can swap them. A vendor locked to one stack today will struggle when a better model ships tomorrow.
• Failure behavior. Ask three questions: What does the caller experience during each failure mode? What automatic fallbacks exist? What appeared in the last incident review? The answers reveal operational maturity far better than any uptime guarantee on a sales deck.
• Grounding. Retrieval-augmented grounding is the primary defense against fabricated answers. Ask exactly what knowledge the agent can draw from, how live data gets pulled at runtime, and what happens when a caller asks something outside that scope.
• Measured latency, not marketing latency. Insist on measuring end-to-end response latency during your own proof of concept, because published figures are often engineering frameworks rather than benchmarks. Use real yardsticks: sub-800ms for natural dialogue, with the current median typically reported around 680ms as of late 2026. Also test barge-in, which should halt speech within a fraction of a second.
• Task success over word accuracy. A 97% speech recognition score means little if the agent books the wrong appointment. Agree on task-level success metrics before the pilot starts.

Compliance deserves its own line of questioning. Voice interactions carry PII and biometric voiceprints, and the regulatory picture is fragmented across the FTC, EU AI Act, TCPA, and HIPAA. Ask how the vendor handles call recording, data retention, and audit trails for your specific industry. Beware of overstated containment claims, too. Ask for the actual percentage of calls resolved without human help, and how it was measured.

Finally, think about ownership. Multi-provider stacks that hand off between separate speech, reasoning, and synthesis vendors add latency and compliance burden at every seam, and renting a closed platform forever means you never own the asset. That's why some buyers prefer custom-built agents they own outright — the approach behind AIQ Labs' AI Operating Systems, built in phases and owned by the client, with full audit trails for regulated industries.

Give vendor evaluation the time it deserves. Analysts estimate the extra effort up front pays off many times over when you go to market, and a structured pilot against the criteria above will tell you more in two weeks than any demo ever will.

Want to see how these criteria hold up under real call volume? Book a free 30-minute strategy call with AIQ Labs and put the platform through your hardest calls, not a scripted demo.