Voice AI Grows Up: Engineering Production Speech Systems in 2026

July 6, 2026 · 9 min read · By Dr. Yochai Edlitz, Founder & CEO, MLAIA

Updated October 2026

Voice AI has matured to the point where the hard engineering problems are no longer "can we make this work?" but "how do we make this reliable, fast, and economically viable at scale?" Yet most production systems engineering teams have had to learn the hard lessons themselves: latency that felt acceptable in a demo becomes intolerable at 500 concurrent calls; multilingual models that scored well on benchmarks collapse on regional accents; and the per-call economics that looked attractive at low volume don't hold when you're running 50,000 interactions a day.

This post is a production engineering guide, not a market overview. It covers the architectural decisions that define whether a voice AI system delivers or disappoints — from the fundamental choice between cascaded and end-to-end architectures, through the latency physics of real-time conversation, to the specific challenges of deploying voice AI in multilingual environments like Israel.

1. The Architecture Decision That Defines Everything

When you set out to build a production voice AI system, the first and most consequential decision is architecture. There are two fundamentally different approaches, and the choice between them shapes every downstream engineering decision.

The cascaded pipeline: STT → LLM → TTS

The cascaded pipeline is the dominant pattern in production today. Speech-to-text (STT) converts the user's audio into a text transcript. A language model processes that transcript and generates a text response. Text-to-speech (TTS) synthesizes the response back into audio. Each stage is a discrete, independently optimizable component.

The advantages are substantial: each component can be swapped, fine-tuned, or replaced independently; failure modes are isolated and debuggable; the LLM operates on text, which means it benefits directly from the full sophistication of modern language models without any voice-specific adaptation; and mature tooling exists for monitoring each stage. Most enterprise voice AI systems running in production today are cascaded pipelines, and for good reason.

The disadvantage is latency. Each stage introduces processing time, and the stages are largely sequential. In a well-optimized cascaded pipeline, the STT model begins generating partial transcripts as the user speaks (streaming), the LLM begins generating tokens before the transcript is complete (speculative completion), and the TTS begins synthesizing as soon as the first sentence of the LLM response is ready (sentence-level streaming). Even with these optimizations, the physics of sequential processing impose a floor of roughly 300–600ms for the first audio byte out, depending on model sizes and infrastructure.

Speech-to-speech: the emerging alternative

Speech-to-speech (S2S) models — systems like OpenAI's gpt-realtime (generally available since August 2025 (opens in a new tab)), Google's native-audio Gemini models, and Kyutai's open Moshi (opens in a new tab) — bypass the text representation entirely. They process audio input and generate audio output as a single end-to-end model, theoretically capturing paralinguistic information (tone, emotion, speaking rate, hesitation) that text transcripts discard.

The promise is compelling: sub-200ms response latency, more natural prosody, and the ability to respond to vocal affect rather than just word content. The production reality in 2026 is more nuanced. S2S models are significantly larger and more expensive to serve than cascaded equivalents. They are harder to fine-tune on domain-specific content because there's no text intermediate to adapt. Debugging a failure in an S2S system is substantially harder than debugging a failure in a cascaded pipeline where you can inspect the transcript and text response independently. And hallucination monitoring — critical in regulated applications — requires a text representation that S2S systems don't natively provide.

The practical recommendation for 2026: if your latency requirement is above 400ms first-audio-byte and your use case requires domain adaptation or compliance monitoring, build on a cascaded pipeline with aggressive streaming optimization. If you're building a consumer conversational product where naturalness is paramount and latency is sub-400ms, S2S is worth prototyping — but plan for limited debuggability and higher infrastructure costs.

2. The Sub-500ms Latency Budget

Human conversation has a remarkably narrow tolerance window. Cross-language studies of conversation find that the gap between one speaker finishing and the next starting is typically around 200ms, with only small differences between languages (Stivers et al., PNAS 2009 (opens in a new tab)). As a rule of thumb, voice AI systems can get away with 400–500ms before users consciously register awkward pauses, and up to 700ms before the interaction starts to feel robotic. Beyond 1 second, the experience degrades sharply and callers begin to disengage.

Breaking down a typical cascaded pipeline latency budget (target: 450ms first audio byte):

• Network transit (user audio to server): 20–60ms depending on geography and network conditions
• VAD (Voice Activity Detection) endpoint detection: 50–100ms — the model must detect that the user has finished speaking without cutting off trailing words
• STT transcription (streaming partial): 30–80ms for the final segment after streaming begins; streaming models can start before the user finishes speaking
• LLM first-token latency (TTFT): 100–200ms for a well-provisioned 7B–13B model; this is often the dominant contributor
• TTS first sentence synthesis: 60–120ms for a streaming TTS model generating the first audio chunk
• Network transit (audio to user): 20–60ms

Total: 280–620ms. The range is wide because each component has high variance depending on model size, hardware, and optimization level. The components that most teams underinvest in are VAD endpoint detection (which controls how quickly the system decides the user has finished speaking) and LLM first-token latency (which is dominated by model size and quantization choices).

3. The Economics: About $0.25 vs $8+ Per Call

The business case for voice AI in customer service and telephony applications is built on a stark cost differential. Gartner's 2019 survey of customer service leaders put the average cost of a live-channel contact (phone, live chat, email) at $8.01, against about $0.10 for self-service (Gartner (opens in a new tab)). A well-engineered voice AI system handling a routine call costs roughly $0.15–0.30 at current list prices.

Here is an estimate for a 3-minute automated call in Hebrew or English, with list prices checked in October 2026. Assumptions: the agent speaks about half the call (roughly 225 words, or 1,300 characters), and the conversation has 8 turns, each sending about 3,000 input tokens (system prompt plus history) and receiving about 100 output tokens.

• STT (Deepgram Nova-3 streaming, $0.0077/min monolingual or $0.0092/min multilingual, Deepgram pricing (opens in a new tab)): ~$0.02–0.03
• LLM inference (a small, fast model; e.g. Claude Haiku 4.5 at $1/$5 per million input/output tokens, or Claude Haiku 5.5 at $0.10/$0.50, Anthropic pricing (opens in a new tab)): ~$0.003–0.03
• TTS (ElevenLabs Eleven v3, which covers Hebrew, at $0.08 per 1,000 characters, ElevenLabs API pricing (opens in a new tab)): ~$0.10
• Telephony (e.g. Twilio, $0.0107/min to receive calls on an Israeli local number, Twilio (opens in a new tab)): ~$0.03
• Compute and infrastructure overhead (our allowance for orchestration, logging and monitoring): ~$0.05

The total lands at roughly $0.25 with a premium Hebrew-capable voice, and closer to $0.15 with cheaper voices and a very small LLM. TTS is typically the largest single cost component, and also the one with the most variance: ElevenLabs Eleven v3 at $0.08 per 1,000 characters versus Google's Chirp 3 HD voices at about $0.03 per 1,000 (Google Cloud pricing (opens in a new tab)) is a 2.7× difference in TTS cost alone. The LLM, by contrast, has become a small line item for short turns on a small model. The right TTS choice is a function of voice quality requirements, language coverage and call volume.

One important caveat: the 95th percentile of calls handled by AI is not the same as the 95th percentile of calls that humans handle. Voice AI systems today still struggle with highly ambiguous requests, strong emotional states, and multi-party scenarios. The practical model is a hybrid: AI handles the routine, well-structured interactions; human agents handle the tail. The effective per-interaction cost for the hybrid model is substantially lower than pure-human, even though the AI portion isn't fully autonomous.

4. The STT Landscape: Open-Source vs Cloud

Speech-to-text has seen more competitive disruption in the past two years than any other component of the voice AI stack. OpenAI's Whisper — released as open weights — established a new benchmark for accuracy and multilingual coverage, and the derivative ecosystem (Faster-Whisper, WhisperX, Distil-Whisper) has made high-quality ASR deployable on commodity hardware. At the same time, commercial STT providers have responded with models that significantly outperform vanilla Whisper on real-time streaming, speaker diarization, and domain-specific vocabulary.

Whisper and its derivatives

OpenAI's Whisper large-v3 remains a strong baseline for offline and near-realtime transcription, particularly for languages beyond English. The open-weights model runs on a single A10G GPU and handles most European and Middle Eastern languages including Hebrew with reasonable accuracy. Faster-Whisper (CTranslate2 backend) is up to 4× faster than the original implementation at the same accuracy (faster-whisper (opens in a new tab)) — this is the go-to for on-prem deployment where controlling costs and data residency matters. Distil-Whisper large-v3 is about 6.3× faster than large-v3 and stays within about 1.5% WER on short-form audio (model card (opens in a new tab)), but it is English-only, so it does not help with Hebrew.

Deepgram Nova-3

Deepgram's Nova-3 is consistently the strongest choice for English-first production applications where streaming latency is the primary constraint. Nova-3 delivers word-level timestamps, real-time partial transcripts with low word error rate, and purpose-built speaker diarization, at $0.0077/minute for monolingual streaming ($0.0092 multilingual; $0.0043 for pre-recorded audio). Its streaming latency (time from word spoken to word in transcript) runs approximately 200–300ms in production, which is lower than Whisper streaming variants. The tradeoff is multilingual support: Nova-3's Hebrew and Arabic accuracy lags behind English by a meaningful margin.

Google Speech-to-Text v2 and Azure Speech

Google's STT v2 and Azure Cognitive Services Speech both offer strong multilingual coverage including Hebrew, Arabic, and code-switching between them. Both provide streaming APIs with competitive latency and built-in telephony integrations (8kHz G.711 support without upsampling). For deployments that are already cloud-committed to GCP or Azure, the native integrations simplify infrastructure. Google's newer chirp_3 model, in public preview since August 2025, lists Hebrew among its preview languages (Google Cloud docs (opens in a new tab)); check its current launch stage, and whether features such as diarization cover Hebrew, before committing.

5. The Israeli Voice AI Ecosystem

Israel has a disproportionately large voice AI ecosystem relative to its market size, driven by a combination of strong NLP research (Hebrew is a morphologically complex language that generated significant academic work), defense and intelligence applications that pushed voice recognition development decades before the commercial market, and a dense startup culture in the audio and speech space.

Verbit

Verbit is the most prominent Israeli company operating at the intersection of voice AI and transcription, with a focus on the legal, education, and media verticals. Its hybrid AI-plus-human correction model has been its commercial differentiator — AI handles the bulk transcription, human specialists handle the accuracy-critical corrections. Its funding passed $550M with its 2021 Series E, a figure that includes secondary transactions (TechCrunch (opens in a new tab)). For enterprises evaluating transcription infrastructure, Verbit's API is worth assessing against Deepgram and Rev.ai for accuracy on accented or domain-specific speech.

Hyro

Hyro builds conversational AI for healthcare, specifically targeting patient-facing voice interactions in hospital and clinic settings. Its architecture is domain-specific: rather than a general LLM, Hyro uses a hybrid approach with a knowledge graph representing the healthcare facility's specific information (department locations, appointment types, provider availability) combined with NLP for intent classification. This makes it significantly more accurate and hallucination-resistant for its target domain than a general-purpose LLM-based voice agent, at the cost of requiring structured knowledge ingestion. Active in Israel and the US healthcare market.

Hi Auto

Hi Auto focuses on drive-through and quick-service restaurant (QSR) voice AI — one of the most demanding acoustic environments for speech recognition due to background noise, multiple simultaneous speakers, and the need for near-perfect order accuracy. Its system is deployed at US chains including Bojangles and Checkers & Rally's (NoCamels (opens in a new tab)), and its acoustic preprocessing pipeline for high-noise environments is a notable engineering differentiator. For any voice AI application in challenging acoustic environments (factory floors, open-plan offices, outdoor settings), Hi Auto's acoustic preprocessing approach is a useful reference architecture.

Voiceitt

Voiceitt works at the intersection of voice AI and accessibility, focusing specifically on users with non-standard speech patterns — people with ALS, cerebral palsy, stroke survivors, and others whose speech deviates significantly from the training distribution of standard STT models. The company's approach of training personalized acoustic models on small amounts of user-specific audio is a technically interesting counterpoint to the trend toward large universal models. For any voice AI application targeting accessibility or serving user populations with diverse speech patterns, Voiceitt's personalization architecture is worth studying.

6. Hebrew and Multilingual Engineering Challenges

Hebrew presents a distinctive set of engineering challenges for voice AI that are worth understanding in detail, both because Israeli enterprises deploying voice AI will encounter them directly and because they illustrate the class of problems that emerge in any morphologically complex language.

The code-switching problem

Israeli speech in professional and commercial settings almost never stays in pure Hebrew. Technical terms, brand names, English-origin words, and switches to full English sentences are constant. A customer service interaction about a software product might contain 60% Hebrew, 30% English technical vocabulary, and 10% borrowed words that could be transcribed in either language. Standard ASR models are trained on "clean" single-language audio and degrade significantly on code-switched input. The production solution requires either a model specifically trained on code-switched Hebrew-English audio or a multi-language hypothesis approach where the transcription engine maintains competing hypotheses in both languages and selects based on downstream language model probability.

Morphological complexity

Hebrew is a root-and-pattern morphological system where prefixes, suffixes, and infixes carry grammatical meaning that is critical for semantic interpretation. The word "מהבנק" (from the bank) contains a preposition, a definite article, and a noun fused into a single orthographic unit. STT models that operate at the word level rather than the morpheme level lose this information. For downstream intent classification and entity extraction tasks, morphological segmentation as a post-processing step significantly improves accuracy compared to treating Hebrew words as atomic tokens.

TTS quality for Hebrew

Naturalness in Hebrew TTS still lags English noticeably. Standard cloud Hebrew neural voices (for example Google's and Azure's) are intelligible but often not conversational — they sound formal and slightly robotic compared to equivalent English models. ElevenLabs' Eleven v3 model, released in 2025, lists Hebrew among its supported languages (ElevenLabs (opens in a new tab)), and its Hebrew voices represent the current state of the art for naturalness, though at higher cost. For production deployments where voice quality is a significant user experience factor, voice cloning with a custom Hebrew voice actor recording roughly 2–4 hours of prompted speech is a viable path using current voice cloning technology. The resulting custom voice substantially outperforms off-the-shelf Hebrew TTS on naturalness metrics.

Right-to-left considerations in integration

This seems obvious but is frequently missed: any system component that logs, displays, or processes Hebrew text must handle right-to-left text direction. This includes logging infrastructure, transcript viewers, quality review interfaces, and any customer-facing text that includes Hebrew. The failure mode — Hebrew characters displayed in the wrong order or mixed with LTR content incorrectly — creates confusing transcripts that impede human review of AI behavior. UTF-8 encoding is universal in modern infrastructure, but RTL rendering support is not universally tested.

7. Production Monitoring and Quality Metrics

Voice AI systems have a monitoring problem that most teams underestimate: the primary failure mode is quality degradation that doesn't trigger any error or alert. A cascaded pipeline where the STT model is transcribing correctly, the LLM is generating coherent responses, and the TTS is synthesizing intelligibly can still be delivering a bad user experience if any of the following are true: the transcription is accurate but the LLM's response is factually wrong; the LLM response is accurate but the TTS prosody is awkward at a particular phrase; or the system is technically functioning but failing to handle a class of requests that has grown in frequency.

The monitoring stack for a production voice AI system needs to track four distinct quality dimensions:

Transcription accuracy (STT quality)

Track word error rate on a sample of calls with ground-truth transcripts. This requires either human review of a random call sample or a secondary high-accuracy STT pass as a reference. Particularly monitor for accuracy degradation on product-specific vocabulary, proper nouns, and domain jargon — these are the words that matter most for intent classification and where standard models are weakest. A 2% overall WER improvement that comes entirely from common words but leaves domain vocabulary at 15% WER has no practical benefit.

Conversation completion and handoff rate

The fraction of interactions that reach successful completion without escalation to a human agent is the most direct measure of system effectiveness. Track this at the intent level: a system that completes 95% of simple FAQ interactions but escalates 40% of billing dispute interactions is telling you something specific about a capability gap. Handoff rate alone is not a sufficient metric — a system that never escalates also doesn't exist, and a system with an artificially low escalation threshold is hiding bad interactions behind human labor.

Latency percentiles (p50, p95, p99)

Median latency is not the right metric for voice AI because users experience the tail, not the average. A system with 300ms median first-audio-byte latency but 1.8 second p99 latency will generate complaints from a predictable fraction of users who encounter the tail. Monitor p95 and p99 latency broken down by call phase — the STT streaming lag, LLM TTFT, and TTS synthesis latency should all be individually observable. Latency spikes in production often have identifiable upstream causes (LLM provider throttling, network congestion, VAD misconfiguration on noisy audio) that are only diagnosable with per-component telemetry.

LLM response quality scoring

Automated quality scoring for LLM responses in voice contexts uses a secondary LLM as a judge, evaluating each response for factual accuracy, relevance, appropriate tone, and adherence to guardrails. This adds cost (on the order of half a cent per call for a ~3,000-token transcript with a small judge model such as Claude Haiku 4.5) but provides the only automated signal about response quality at scale. For regulated applications, this automated scoring plus a human review queue for flagged interactions is the minimum viable compliance framework.

Voice AI is the first AI application where users form an opinion within the first three seconds. A model that takes 700ms to respond and then says something slightly awkward loses the user in a way that a chatbot or document AI never does — because the failure is instant, embodied, and interpreted as incompetence. This is why production voice AI engineering is as much about p99 latency as it is about accuracy.

8. Where Voice AI Is Heading in the Next 18 Months

The trajectory of production voice AI over the next 18 months is shaped by three forces that are already visible in the current generation of systems.

First, speech-to-speech will cross a production viability threshold. OpenAI's realtime voice models and Kyutai's Moshi have demonstrated that S2S latency and quality can reach a bar that users find acceptable. The remaining barriers — cost, debuggability, and fine-tuning — are engineering problems that will be solved, not fundamental limitations. By late 2027, the most naturalistic consumer voice AI applications will likely be S2S rather than cascaded.

Second, voice cloning will become a mainstream deployment pattern. Custom voices trained on a specific organization's brand voice or a real person's recorded speech will replace off-the-shelf TTS voices in most enterprise deployments. The quality gap between a custom cloned voice and a generic neural TTS voice is already large enough that it's a meaningful differentiator in customer experience. The technology to create a good custom voice from 2–4 hours of audio is available today.

Third, real-time call analysis will become standard infrastructure. A voice AI system that only monitors completed calls is missing most of its operational signal. Real-time side-channel analysis — a separate model observing the conversation as it happens and flagging escalation triggers, compliance risks, or upsell opportunities — is emerging as a standard capability in enterprise voice deployments. The architecture is already well-understood; what's lagging is the deployment of the monitoring models alongside the conversation models.

For Israeli enterprises, these trends layer on top of the multilingual challenge: the models that will define the next generation of voice AI are predominantly trained on English and Mandarin data, and the gap in Hebrew voice AI quality relative to English will widen before it narrows unless organizations invest in Hebrew-specific voice data and fine-tuning. This is the constraint that most Israeli voice AI deployments are currently navigating — and where partner organizations with deep audio AI and Hebrew NLP expertise provide the most concrete value.

About MLAIA, the team behind Ultrabot

MLAIA — Machine Learning & AI Approach — is the Israeli data-science company that builds Ultrabot, with production ML work across signal processing, audio AI, medical, defense, and enterprise domains. Led by Dr. Yochai Edlitz (PhD, Weizmann Institute), MLAIA builds voice AI pipelines that work in production: low-latency cascaded architectures, Hebrew and multilingual STT optimization, custom voice cloning, and real-time monitoring infrastructure. Based in Yavne, Israel; serving clients globally.

Related engineering guides: noise reduction, echo cancellation and ANC for cleaner input audio, on-device audio AI for wake words and streaming models within latency and power budgets, and microphone array beamforming for far-field pickup.