How realtime simultaneous interpretation works: three-stage pipeline vs end-to-end speech models

By the Parley Voice team · Last updated 26 October 2026 · We make Parley, so this is not an independent review.

1. Route one: ASR → MT → TTS (three-stage)

The three-stage idea splits one hard job into three: hear clearly, translate well, speak it out.

Flow (schematic): source speech → ASR → MT → TTS → target speech

Step 1: ASR turns speech into text

ASR (automatic speech recognition) takes microphone audio, finds words, sentences and pauses, and outputs source-language text. Modern ASR may itself use Transformer or Conformer models—so “three-stage” does not mean outdated recognition.

Step 2: MT translates the text

MT (machine translation) takes that text and produces the target language, for example English. Neural and LLM-based MT can use context for grammar, sense and domain terms.

Step 3: TTS speaks the translation

TTS (text-to-speech) turns the translated text into playable audio. Modern TTS includes acoustic modeling, voice control and neural vocoders; some stacks add voice cloning.

Three-stage is not a low-quality design. It is mature and modular: you can swap ASR without rewriting MT, or change a voice without retraining translation. For realtime simultaneous use it hits three familiar limits.

1. Latency stacks. ASR needs enough audio; MT needs context; TTS needs to synthesize. Strict sequencing adds wait. Streaming ASR, incremental MT and streaming TTS can overlap, but cross-module coordination stays complex.

2. Errors cascade. If ASR mishears a name or term, MT usually works on the wrong text; polished TTS cannot restore lost meaning.

3. Prosody may drop. Emphasis, hesitation, pauses and emotion do not all survive in a transcript. If TTS only sees a plain sentence, listeners may get the words without the tone.

Cascades still fit audited text, glossary control, specific language pairs, or independent module swaps. For continuous two-way spoken conversation they are not the only option.

2. Route two: end-to-end speech models (S2ST)

End-to-end speech translation is often called speech-to-speech translation (S2ST). Instead of wiring external ASR, MT and TTS as the core path, a unified or jointly trained system does cross-language understanding and speech generation—without first completing a full transcript and final text hand-off to a separate TTS service.

Flow (schematic): continuous input speech → audio feature encoding → cross-lingual semantic modeling → streaming speech generation → continuous translated audio

That diagram is a teaching abstraction, not a claim that every commercial model shares one internal layout.

1. The model “hears” audio, not only a transcript

Raw audio is framed into features such as mel spectrograms, continuous vectors or discrete audio tokens. Encoders learn phonemes, words, rhythm, context cues and some speaker traits, mapping them into representations that support cross-lingual understanding. How much tone and emotion survive depends on architecture, data and generation.

2. Transformers and attention for context

Attention helps decide when a turn of phrase is safe to translate. Speakers often reverse course mid-sentence; realtime models must weigh heard context, history and what still needs to arrive. Some systems use streaming Transformer encoders/decoders; others use multimodal backbones that jointly train understanding and speech. Google’s 2025 realtime S2ST notes and Meta’s SeamlessStreaming (EMMA) are public examples of streaming generation without waiting for a full utterance.

3. Generate target speech from semantic representations

After understanding the source, the model can predict target acoustic features or speech tokens and decode them to waveform. Translation and speaking can stay tightly coupled inside one system. Research such as Qwen3-Omni’s Thinker–Talker layout shows how understanding and speaking can connect—without proving every commercial LiveTranslate build uses that exact internal graph. Speaker features can also support live voice cloning.

4. Simultaneous experience needs listen-while-speak

Even a strong model is poor “simultaneous interpretation” if it waits for a whole paragraph. Streaming and incremental generation matter: short chunks in, decide when enough is known, emit confirmed target audio while still listening, and balance accuracy, wait and playback rate. Word-order differences may force a short extra wait. “Full duplex” in research (emit while still receiving) is related to, but not identical with, product “two-way meeting translation.”

A clarification

“End-to-end” does not mean zero text, one network component, or a single matmul. Training may use transcripts; products may emit captions; internals may use text tokens, audio tokens, decoders and vocoders. The product distinction is that the main speech-to-speech path need not serialize independent ASR, MT and TTS services.

3. Trade-offs

Neither route wins every job. Offline captions, heavy review, or strict text rules still favor cascades. If the goal is for people who do not share a language to keep talking like a normal meeting, end-to-end speech models usually match the product direction better.

4. Why ParleyVoice chose end-to-end

We built ParleyVoice meeting translation for a practical question: can two people keep speaking naturally without stopping every few seconds for translation? We integrate models aimed at realtime speech-to-speech rather than making three separate services the core path.

1. Keep translation closer to conversation rhythm. Long pauses break business calls, interviews and customer chats. End-to-end coupling of understanding, timing and speech generation is a better base for listen-while-speak—not “zero delay.” Capture, network, play buffering and routing still add time.

2. Preserve “who is speaking” when the model allows. Managed Qwen and Doubao paths support live voice cloning without a pre-recorded enrollment sample. Likeness depends on input quality, speaker, language direction and model limits.

3. Design for real two-way meetings, not only a demo. Your mic → translated speech into a virtual mic for Zoom, Google Meet, Microsoft Teams and similar; meeting audio → translation in your headphones; bilingual captions and optional SRT export; usually only one side installs Parley. Desktop Windows and macOS; see also supported in-person modes.

Models today: Qwen LiveTranslate focuses on multilingual realtime SI (public figures: many input languages, fewer spoken outputs—hearing a language ≠ speaking it). Seed LiveInterpret (Doubao) focuses on Chinese ↔ English with listen-while-speak and live cloning. Provider-reported lag figures (for example Qwen LAAL ~2.3 s; Seed speech-output latency ~2.53 s) use different metrics—not a head-to-head speed contest or a Parley full-path SLA. Check languages and voices.

5. Where this is heading

AI simultaneous interpretation is moving from “hear a sentence, translate a sentence” toward long-context understanding, speaker attribution, adaptive wait, and natural delivery. End-to-end models let those goals share one optimization target. Terms, overlapping talk, noise, word order and critical numbers still deserve human double-checks.

ParleyVoice chose end-to-end not because three-stage lost all value, but because our goal is natural, low-wait, two-way spoken conversation across languages.

Try it on the next cross-language call: parleyvoice.com, meeting translation, or download and first use.

Further reading

1. Google Research: Real-time speech-to-speech translation (2025)
2. Meta AI: Seamless Streaming Speech Translation
3. Qwen3-Omni Technical Report
4. ByteDance Seed: Seed LiveInterpret 2.0
5. Qwen: Qwen3.8-LiveTranslate
6. Google Research: Translatotron 2