The voice AI stack has historically been a pipeline: speech recognition converts audio to text, a language model reasons over the text, and speech synthesis converts the response back to audio. Each stage is mature, well-studied, and steadily improving. The pipeline works, and most production voice agents today are built this way.
It also has a ceiling, which the industry is now running into. The next generation of systems, speech-to-speech models that process and generate audio natively without an intermediate transcript, promises to break through that ceiling. We believe they have the capacity to. But it is worth being clear-eyed about why this is a hard problem, and why the promise and the current reality should be evaluated separately.
Why the pipeline has a ceiling
The pipeline's weakness is structural: everything the language model knows about the user arrives through a transcript, and a transcript is a lossy channel. Hesitation, tone, pacing, emphasis, and emotional prosody are stripped out at the first stage. The reasoning layer literally cannot respond to signals it never receives.
Consider a banking agent asking whether you recognize a potentially fraudulent charge. A confident "yes" and a hesitant, drawn-out "yes" produce identical transcripts and demand different responses. A pipeline system treats them the same. A human would not. The same limitation runs in the output direction: a synthesis stage that receives only text has to guess at the appropriate delivery, because the reasoning that would inform tone happened in a modality where tone does not exist.
Speech-to-speech architectures address this in principle. The model hears the actual audio, so paralinguistic signal can inform the response, and it generates audio directly, so delivery can be shaped by context rather than guessed at. In principle.
Why native audio is so much harder than it looks
Going end-to-end does not just move the same problem into one model. It changes the nature of the problem in at least three ways.
First, the model must learn perception and expression jointly. In a pipeline, recognition and synthesis can be optimized independently against well-defined targets. A native audio model has no such decomposition. Understanding what a hesitant voice means and knowing how to respond to it are entangled in the same weights, and progress on one does not automatically produce progress on the other. It is entirely possible to build a system that hears emotion well but responds woodenly, or one that speaks beautifully while missing the emotional context it should be speaking to.
Second, new failure modes appear that the pipeline world never had. When a model generates its own voice turn by turn rather than rendering text through a fixed synthesis engine, questions arise that never existed before. Will it sound like the same person at minute twelve that it did at minute one? Will its persona stay stable as conversational context accumulates? Will its full-duplex behavior, the natural give-and-take of interruption and backchanneling, feel human or chaotic? These properties are emergent rather than engineered, which makes them hard to guarantee and easy to break.
Third, the training signal problem gets harder. Pipelines could ride on decades of paired data: audio with transcripts, text with recordings. Conversational appropriateness has no such natural supervision. What does the "correct" response to a frustrated, trailing-off sentence sound like? There is no reference answer sitting in a dataset, only human judgment about whether a given response was fitting. Supervising that quality at scale is an open problem for the entire field.
Coupled understanding and expression is the actual frontier
The deeper point is that understanding and expression are not two features to be checked off independently. They form a loop. Responding appropriately requires hearing what was actually communicated, and communicating effectively requires shaping delivery to what was heard. Building systems that close this loop is, in our view, the central open challenge in voice AI, and speech-to-speech is where that challenge lives.
This is also why the category is so hard to evaluate. A speech-to-speech model cannot be meaningfully scored with a transcription metric plus a synthesis metric, because its distinctive successes and failures live in the interaction between the two. Evaluation has to be conversational, multi-turn, grounded in human judgment, and conducted under realistic conditions: accents, noise, emotional complexity, and length. Anything less measures the parts and misses the system.
What builders should do now
For teams weighing pipeline versus speech-to-speech architectures, the honest framing in mid-2026 is that it depends on which failures you can afford. Pipelines are predictable and emotionally deaf. Native audio systems are emotionally capable in principle and unproven in the ways that matter most for production: consistency, stability, and appropriateness over long, messy, real conversations.
That makes rigorous measurement the deciding input, not an afterthought. Before committing a roadmap to this category, teams need a way to assess where a candidate system actually sits on each capability: emotional understanding, response appropriateness, expressiveness, identity stability, robustness. Architectural promise is not a substitute for evidence, and demos are not evidence.
We expect speech-to-speech to define the next chapter of voice AI. The architecture is right and the trajectory is taking shape. But the gap between promise and production will close unevenly, capability by capability, and the teams that navigate it well will be the ones measuring rather than assuming.
If you are betting part of your roadmap on this category and want a rigorous, human-grounded way to assess candidate systems, talk to us before you place the bet.
