Speech Recognition Architecture: CTC vs RNN-T vs Attention Encoder-Decoder

Most teams pick an automatic speech recognition (ASR) model by reading a leaderboard, then discover in production that the leaderboard measured the wrong thing. A model that wins on clean read speech can invent sentences during silence, lag two seconds behind a talker, or fall apart on a part number spoken over a compressor. The cause is rarely the model size. It is the speech recognition architecture underneath: how audio becomes frames, how frames become tokens, and what the decoder is allowed to assume about time.

This matters now because the field has split into two camps that look similar from the outside. Whisper-style attention encoder-decoders dominate offline transcription, while transducer and CTC systems built on the Conformer encoder dominate low-latency voice agents, call-centre analytics, and on-device dictation. Choosing between them is an architectural decision, not a benchmark lookup.

This article walks the whole stack, from the log-mel front end to the serving tier, and gives you a way to choose a decoder family from your latency, accuracy and failure tolerance. What this covers: audio front ends, encoders, the three decoding families with their loss functions, streaming mechanics, WER and its traps, Whisper facts, hallucinations, biasing, diarization, edge deployment, and a latency budget.

Context and Background

Classical ASR split the problem into an acoustic model, a pronunciation lexicon and a language model, glued together by a weighted finite-state transducer. Training each part separately required forced alignments and hand-built lexicons. End-to-end neural models collapsed that pipeline into one network trained on pairs of audio and text, and three families emerged: Connectionist Temporal Classification (CTC), the recurrent neural network transducer (RNN-T), and attention-based encoder-decoders (AED). Each answers the same hard question differently: audio has far more frames than the transcript has tokens, so how do you align them without labelled timings?

CTC was introduced by Graves and colleagues in 2006 and answers with a latent monotonic alignment and a “blank” symbol. The transducer was proposed by Graves in 2012 (arXiv:1211.3711) and adds a label-history network so output tokens condition on earlier tokens. Attention encoder-decoders, popularised by Listen, Attend and Spell, drop the explicit alignment entirely and let a decoder attend across the whole encoded utterance. The Conformer encoder, introduced by Gulati and colleagues at Google in 2020 (arXiv:2005.08100), combined self-attention with convolution and became the default acoustic backbone for the first two families.

Two later developments shaped 2026 practice. First, OpenAI’s Whisper (arXiv:2212.04356) showed that weak supervision at scale, 680,000 hours of web audio in the original paper, buys robustness that curated corpora do not. Second, NVIDIA’s FastConformer (arXiv:2305.05084) redesigned the Conformer’s downsampling and reports roughly 2.8 times faster processing than the original, which is why the Parakeet and Canary families in NVIDIA NeMo build on it.

If you have read our posts on LLM tokenization with BPE, SentencePiece and tiktoken, you already know that the output vocabulary shapes everything downstream. ASR is no different: a 1,024-piece SentencePiece vocabulary changes the decoder’s compute, its alignment behaviour and its failure modes compared with a character or a 50,000-token vocabulary. Audio is also one modality in the larger fusion picture covered in multimodal AI architecture for vision, language and audio.

For primary sources, the Whisper repository documents model sizes and licence, and the Conformer paper remains the best single description of the encoder block.

The Reference Architecture: From Waveform to Tokens

A modern ASR system is a pipeline with one learned core. Audio is sampled at 16 kHz, converted to a log-mel spectrogram at 100 frames per second, shrunk by convolutional subsampling to 12.5 to 25 frames per second, encoded by a Conformer or Transformer stack, and decoded into subword tokens by a CTC, RNN-T or attention decoder. Everything else is engineering around that core.

Figure 1: The end-to-end speech recognition architecture. Frame rate drops at the subsampling step, which is the single largest lever on compute and latency.

The figure shows the stages in order. Raw 16 kHz audio is cut into overlapping frames, transformed with a short-time Fourier transform, pooled through a mel filterbank, log-compressed and normalised. A strided convolution stack then reduces the frame rate before the expensive encoder runs. The decoder, whichever family it belongs to, consumes the encoder’s output and emits tokens. The rest of this article zooms into each box and compares the three decoder choices.

The audio front end: log-mel and frame rates

Whisper’s front end is the clearest public reference. According to the paper, audio is resampled to 16 kHz and converted to an 80-channel log-magnitude mel spectrogram using 25 millisecond windows with a 10 millisecond stride. The large-v3 checkpoint changed this to 128 mel bins, up from 80 in earlier large models, as stated on its model card. Most Conformer systems use 80 or 128 mel bins with the same 25 ms window and 10 ms hop.

The arithmetic is worth doing once. A 10 ms hop gives 100 feature frames per second. Whisper processes fixed 30-second windows, so its input is 30 x 100 = 3,000 frames. Its first convolution layers include a stride-2 step, which halves the sequence to 1,500 encoder positions per window, or one position per 20 ms. That figure also explains why Whisper’s timestamp tokens are quantised to 20 ms, a detail the paper confirms.

Why log-mel at all? The mel scale spaces filters roughly the way the cochlea does, with fine resolution below about 1 kHz and coarser resolution above, which matches where speech carries information. Log compression turns multiplicative loudness differences into additive ones, so a quiet and a loud speaker produce similar feature shapes. Per-utterance or per-window mean and variance normalisation removes channel offsets from different microphones. Learnable front ends exist, but log-mel remains the default because it is cheap, stable and well understood by every toolkit.

A practical consequence for streaming: a 25 ms window with a 10 ms hop means the feature extractor already needs 15 ms of lookahead beyond the current hop, a small but real contribution to latency that is easy to forget when budgeting.

Subsampling: the cheapest compute you will ever save

Speech is oversampled relative to text. At 100 frames per second, a ten-word sentence of three seconds has 300 frames but perhaps 15 subword tokens. Self-attention costs grow quadratically with sequence length, so running a deep Transformer at 100 frames per second is wasteful. Convolutional subsampling at the encoder input reduces the rate by a fixed factor before attention sees anything.

The original Conformer used a 4x reduction, giving 40 ms per encoder frame. FastConformer uses 8x, giving 80 ms per frame, and the downsampling change is the main design difference behind its reported speedup (an inference from the paper’s framing; the NVIDIA NeMo docs page could not be fetched to confirm the exact factor). The cost is resolution. An 80 ms frame can still represent a token, since average speech runs at several tokens per second, but it constrains how short a token’s alignment can be. For transducers this interacts with the blank symbol: if there are fewer frames than tokens in a fast burst, the model needs multiple emissions per frame, which standard RNN-T allows but CTC does not.

The rule of thumb is that a CTC model needs at least as many encoder frames as output tokens (more if repeated tokens appear, since a blank must separate them). With a 1,024-piece vocabulary and 80 ms frames this is almost never violated for natural speech, but it can bite with a character vocabulary combined with 8x subsampling on fast talkers.

The Conformer encoder, block by block

A Conformer block is a “macaron” sandwich: a half-step feed-forward module, multi-head self-attention with relative positional encoding, a convolution module, a second half-step feed-forward module, and a final layer norm. The convolution module uses a pointwise convolution with a gated linear unit, a depthwise convolution, batch normalisation, a Swish activation, and another pointwise convolution. Attention captures global context such as speaker style and long-range agreement. The depthwise convolution captures local patterns such as formant transitions over a few frames.

Why does this beat a plain Transformer for speech? Acoustic events are locally structured, and convolution gives that structure for free with far fewer parameters than attention would need to learn it. The combination also trains well on moderate data. The Conformer paper presented ablations supporting each piece, and later work confirmed the pattern broadly, though specific reported WER figures depend on corpus and decoding setup, so check the paper rather than quoting a number from memory.

For streaming, the encoder needs a restriction on what each frame may see. Full self-attention over the entire utterance cannot start until the audio ends. The two standard fixes are limited left and right context in attention, and causal convolution that pads only on the left. The first controls the algorithmic latency directly: with 80 ms frames and a right context of two frames, the model waits 160 ms before committing a frame. We return to this in the streaming section.

FastConformer and the long-form trick

FastConformer changes the downsampling stack (depthwise-separable convolutions at 8x) and trims the encoder for speed. The paper also reports that after training with global attention, you can swap in limited-context attention and recover accuracy by fine-tuning with a global token, enabling transcription of audio up to 11 hours long according to the abstract. That is a useful reminder that attention cost, not parameter count, is what limits long-form inference.

The Three Decoding Families Compared

All three families share an encoder and differ in how they turn encoder frames into tokens. The differences show up in the loss function, the alignment assumption, the streaming behaviour and the failure modes. The one-sentence summary: CTC assumes frames are independent given the audio, RNN-T adds a label history network so tokens depend on previous tokens, and attention encoder-decoders drop alignment entirely and generate autoregressively.

Figure 2: Three decoder heads on the same encoder. CTC works per frame, RNN-T sums over a time-label lattice, and attention decodes autoregressively with cross attention.

Read the figure left to right. The CTC head is a single linear layer and softmax per frame, followed by blank collapsing. The RNN-T joint network combines each encoder frame with the prediction network’s summary of previous labels. The attention decoder cross-attends to all encoder frames and generates one token at a time. The sections below give the maths and the consequences.

CTC: alignment as a sum over paths

CTC adds a special blank symbol to the vocabulary. The network emits a distribution over vocabulary plus blank for every encoder frame. A path is one choice of symbol per frame, and a collapse function removes repeats and then blanks, so the path “h h blank e l l blank l o” collapses to “hello”. The CTC loss is the negative log of the total probability of all paths that collapse to the reference transcript:

L_CTC = -log sum over paths pi in B^-1(y) of prod over t of p(pi_t | x)

Computing that sum naively is exponential, but a forward-backward dynamic program over the blank-augmented label sequence makes it O(T x U) where T is the frame count and U the label length. The key assumption is conditional independence: given the audio, each frame’s output is independent of the others. That is what makes CTC decoding trivially parallel and fast, and also why it has no internal language model. A CTC model will happily spell “recieve” if the acoustics say so, because nothing in the head knows English spelling.

Strengths: simple, non-autoregressive, low memory, easy to deploy, naturally streaming when the encoder is. Weaknesses: weaker raw accuracy without an external language model, spiky and delayed token emissions (the model often emits a token late, after seeing enough context), and difficulty with repeated phrases needing blank separation.

RNN-T: monotonic alignment with label memory

The transducer keeps CTC’s monotonic alignment but removes the independence assumption. It has three parts: an encoder over audio, a prediction network (an LSTM or stateless embedding) over previously emitted non-blank labels, and a joint network that combines one encoder frame with one prediction state to produce a distribution over vocabulary plus blank. Emitting a non-blank token advances the label index u and stays on the same frame; emitting blank advances the frame index t. The probability of a transcript is the sum over all monotonic paths through the T x U lattice.

Two properties matter in practice. Because blank advances time, the model decides frame by frame whether it has more to say, which is exactly what a streaming system needs. And because the prediction network conditions on previous labels, the model learns an implicit internal language model, usually giving better accuracy than CTC at equal encoder size.

The cost is memory in training. The joint network produces logits for every (t, u) pair. A worked example: batch 16, T = 250 encoder frames (about 20 seconds at 80 ms), U = 60 labels, vocabulary V = 1,024. That is 16 x 250 x 60 x 1,024 = 245,760,000 logits, about 0.98 GB in fp32 for the logits alone, before gradients. Common remedies are pruned transducer losses, function merging (computing the loss without materialising the full tensor), and a smaller vocabulary. NVIDIA’s Token-and-Duration Transducer (TDT) adds a duration prediction so the model can skip multiple frames per step, which is another route to speed; check NeMo’s documentation for current model availability rather than assuming a specific checkpoint.

Attention encoder-decoders: Whisper’s family

An attention encoder-decoder trains a decoder to maximise p(y_i | y_<i, x) with cross-entropy, teacher-forced on the reference. There is no alignment variable at all. The decoder cross-attends to the whole encoder output at every step, so the model can reorder, repeat or skip as it sees fit. This is the same recipe as a text translation model with audio as the source.

That freedom is both the strength and the hazard. The decoder is a full conditional language model, so it learns spelling, punctuation, casing, and cross-utterance context. Accuracy on difficult, noisy audio is typically the best of the three families. But nothing forces monotonicity or coverage, so the decoder can loop on a phrase, skip a segment, or generate fluent text unrelated to the audio. And since attention needs the full encoder output, vanilla AED cannot stream without surgery such as chunked or monotonic attention.

Decision matrix

Criterion	CTC	RNN-T / transducer	Attention encoder-decoder
Alignment	Latent, monotonic, frame-independent	Latent, monotonic, label-conditioned	None, learned via cross attention
Internal language model	None	Implicit via prediction network	Strong, full decoder
Streaming	Natural with a streaming encoder	Natural, the standard choice	Needs chunking or special attention
Decoding cost	Lowest, parallel	Low to moderate, sequential over frames and labels	Moderate to high, autoregressive over tokens
Training memory	Low	High (T x U x V joint)	Moderate
Typical failure	Misspellings, late or spiky emissions	Deletions on fast speech, over-reliance on label history	Hallucination, repetition loops, skipped speech
Best fit	Edge, keyword-heavy, cheap batch	Live captions, voice agents, dictation	Offline transcription, noisy multilingual audio

None of the rows is absolute. A strong CTC model with a good external language model can beat a weak transducer, and joint CTC-attention training, used widely in ESPnet-style recipes, gets many of the benefits of both. Treat the table as a prior to be overwritten by measurement on your own audio.

Streaming vs Offline: Where the Latency Actually Comes From

Streaming ASR latency has three independent components: algorithmic latency from the encoder’s lookahead, compute latency from running the model, and endpointing latency from deciding the speaker has finished. Teams routinely optimise the first two and lose a second on the third.

Figure 3: A streaming ASR session. Partial hypotheses flow back while audio is still arriving; the endpointer decides when a result becomes final.

In the diagram, audio arrives in small frames, a voice activity detector (VAD) gates speech into chunks, the encoder processes each chunk with a cache of left context, and the decoder emits partial text that may be revised. Only a silence-based decision produces the final result. The sections below take each delay in turn.

Chunking, lookahead and the cache

Chunked streaming splits audio into fixed chunks, for example 160 to 640 ms. Within a chunk the encoder may attend to all frames in the chunk plus a limited right context (lookahead) and a bounded left context carried over as a cache. Cache-aware streaming, as implemented in NVIDIA NeMo for streaming Conformer and FastConformer models (unverified in detail here), stores the intermediate activations of the previous chunks so nothing is recomputed. That turns per-chunk cost into a constant, independent of how long the session has run.

The latency budget follows directly. If a chunk is 320 ms and the right context is one 80 ms frame, a token can be committed at the earliest about 400 ms after the audio it represents arrived, plus compute. Shrinking the chunk reduces latency but also reduces the context available for each decision, which costs accuracy, especially for homophones that need the following word to disambiguate (“recognise speech” vs “wreck a nice beach”). Many modern systems train a single model with multiple chunk configurations so the operator can pick the latency-accuracy point at inference time.

Transducers are a natural fit because they emit tokens as soon as the encoder produces enough evidence, and blank emission handles silence. CTC streams equally well but, because it emits late and without label history, usually needs a larger right context for the same accuracy.

Whisper is the notable exception. Its encoder expects a fixed 30-second window. “Streaming Whisper” implementations work by repeatedly re-running the model on a growing or sliding buffer, then using agreement between successive hypotheses to decide which prefix is stable. That works, but each pass is a full 30-second encoder forward, so the compute per second of audio is much higher than a true streaming transducer, and the latency is lumpy. If you need sub-300 ms partials, pick a streaming transducer or a streaming CTC model, not a Whisper wrapper.

Endpointing: the hidden second

Endpointing decides when the user has stopped talking. The simple method is a silence timer: if the VAD sees no speech for N milliseconds, finalise. Set N to 300 ms and you cut people off mid-thought during natural pauses; set it to 1,000 ms and every turn of a voice agent feels sluggish. Better systems combine acoustic silence with decoder state (an end-of-sentence probability, or a transducer that has emitted nothing for several frames) and with semantic cues from the downstream dialogue manager. In a voice-agent latency budget the endpointer routinely costs more than the acoustic model.

A useful engineering distinction: partial results are for display and can be wrong; final results are for action and must be stable. Never trigger an irreversible action from a partial.

Latency budget, worked example

The following is an illustrative budget for a streaming voice assistant, not a measured benchmark. Your numbers will differ with hardware and model.

Stage	Illustrative cost
Feature window lookahead	about 15 ms
Encoder right context (two 80 ms frames)	about 160 ms
Chunk wait (average half of a 320 ms chunk)	about 160 ms
Model compute per chunk on GPU	tens of ms
Network round trip	20 to 100 ms
Endpointing hangover	300 to 700 ms

Sum these and you see the structure: roughly 350 ms of unavoidable algorithmic delay before compute, and the endpointer then dominates the time to a final transcript. Measure time-to-first-partial and time-to-final separately, because they are controlled by different components.

Whisper in Depth: Facts, Sizes and Behaviour

Whisper is a Transformer encoder-decoder trained on a multitask format. The paper reports 680,000 hours of labelled audio, of which 117,000 hours cover 96 non-English languages and 125,000 hours are X-to-English translation data. The decoder is controlled by special tokens: start of transcript, a language token, a task token (transcribe or translate), a no-timestamps flag, and timestamp tokens quantised to 20 ms. The same weights therefore do transcription, translation, language identification and timestamping, depending on the prompt.

Sizes, versions and the turbo variant

Model	Layers (enc and dec)	Width	Parameters	Notes
tiny	4	384	39M	English-only variants exist
base	6	512	74M	English-only variants exist
small	12	768	244M	English-only variants exist
medium	24	1024	769M	English-only variants exist
large	32	1280	1550M	v1, v2, v3 share this size
turbo	decoder reduced from 32 to 4	1280	809M	pruned and fine-tuned large-v3

These figures come from the Whisper paper and the GitHub README. The README lists approximate VRAM (about 10 GB for large, about 6 GB for turbo) and relative speed (turbo about 8x versus large at 1x, measured on an A100 transcribing English; the README itself cautions that real-world speed varies).

Large-v3, according to its model card, moved to 128 mel bins, was trained on about 1 million hours of weakly labelled audio plus about 4 million hours of audio pseudo-labelled by large-v2, supports 99 languages including a Cantonese token, and shows a 10 to 20 percent error reduction over large-v2 across many languages. The turbo model card describes it as large-v3 with decoder layers cut from 32 to 4, then fine-tuned, trading a minor quality loss for speed; it is not trained for translation, and the README notes it returns the original language if asked to translate. The turbo card gives no new training-data details, so none should be assumed.

Why does cutting only the decoder help so much? Autoregressive decoding runs the decoder once per output token, each time cross-attending to 1,500 encoder positions, while the encoder runs once per window. For a 30-second window containing 80 spoken words, the decoder runs perhaps 100 to 120 steps (an estimate: it depends on tokenisation and language). Reducing 32 layers to 4 shrinks the per-step cost by roughly eight times for the decoder stack, while the encoder cost is unchanged. That is a sound mechanism-level explanation of the speedup, though end-to-end gains depend on how much of the runtime the decoder accounted for in your setup.

Licence: the GitHub repository states the code and weights are MIT-licensed. Individual Hugging Face model cards may list different licence metadata (the large-v3 card lists Apache-2.0 while the turbo card lists MIT at the time of writing), so confirm on the specific checkpoint you ship.

Running Whisper: a faster-whisper snippet

The faster-whisper project reimplements inference on CTranslate2. Its README claims up to 4x speedup over the reference implementation at equal accuracy with lower memory, and reports, for large-v2 on 13 minutes of audio on an RTX 3070 Ti, 2 min 23 s for openai/whisper fp16 versus 1 min 03 s for faster-whisper fp16 and 59 s for int8. Those are the project’s own measurements on one machine, so treat them as indicative.

from faster_whisper import WhisperModel, BatchedInferencePipeline

# int8_float16 cuts VRAM on GPU; use device="cpu", compute_type="int8" on CPU
model = WhisperModel("turbo", device="cuda", compute_type="int8_float16")
pipe = BatchedInferencePipeline(model=model)

segments, info = pipe.transcribe(
    "meeting.wav",
    batch_size=8,
    vad_filter=True,                 # Silero VAD skips non-speech
    vad_parameters=dict(min_silence_duration_ms=500),
    condition_on_previous_text=False,  # reduces repetition loops
    word_timestamps=True,
)
print(f"language={info.language} p={info.language_probability:.2f}")
for seg in segments:  # generator: transcription runs here
    print(f"[{seg.start:7.2f} -> {seg.end:7.2f}] {seg.text}")

Note segments is a lazy generator, so nothing runs until you iterate. The VAD filter and condition_on_previous_text=False are the two settings that most directly attack hallucination, discussed next.

Hallucination and Other Failure Modes

Attention decoders fail in a way CTC and transducers rarely do: they generate fluent, plausible, entirely wrong text. Whisper’s own model card warns that predictions may include text that is not actually spoken in the audio, and that repetition is a known issue, more likely in lower-resource languages.

Why decoders hallucinate

During training the decoder sees web-scale transcripts, many of which include credits, subtitle boilerplate or sponsor lines. Given ambiguous input, such as silence, music or noise, the decoder falls back on its language prior, which is exactly what an autoregressive language model does. The most common symptoms in practice are phantom phrases during silence, repetition loops where one phrase repeats until the window ends, and drift when condition_on_previous_text carries an earlier error forward into the next window.

The original Whisper paper describes its mitigations for long-form audio: temperature fallback that increases temperature from 0 in steps of 0.2 when the average log probability falls below -1 or the gzip compression ratio exceeds 2.4, and a no-speech probability threshold of 0.6 combined with the log-probability threshold. The compression-ratio trick is clever: repetitive text compresses extremely well, so a high ratio is a cheap detector of a loop.

Mitigations that work in production

Gate the audio first. Running a VAD before the decoder removes most silence-induced hallucination at almost no cost. Cap the per-window output length relative to the audio’s duration, since 80 tokens from 2 seconds of audio is implausible. Disable conditioning on previous text for noisy or multi-speaker sources. Monitor the compression ratio and average log probability per segment and re-decode or flag outliers. For safety-critical use (medical, legal, industrial command), prefer a model class that cannot invent text unsupported by audio, or require a confidence threshold with a human in the loop.

Transducers and CTC fail differently: deletions on fast or mumbled speech, substitution of acoustically similar words, and out-of-vocabulary names turned into near-homophones. These are visible in WER and easier to bound, which is an under-appreciated argument for them in controlled domains.

Measuring It Honestly: WER and Its Pitfalls

Word error rate is the edit distance between hypothesis and reference at the word level, divided by the number of reference words: WER = (S + D + I) / N, where S is substitutions, D deletions, I insertions, and N the reference word count. Because insertions are not bounded by N, WER can exceed 100 percent.

A worked example. Reference: “turn the pump on at six” (N = 6). Hypothesis: “turn pump off at six”. Aligning them, “the” is deleted (D = 1), “on” is replaced by “off” (S = 1), and there are no insertions. WER = (1 + 1 + 0) / 6 = 33.3 percent. Notice that the single wrong word, “off”, is the one that matters operationally, while the deleted article is harmless. WER weights both equally.

def wer(ref: str, hyp: str):
    r, h = ref.split(), hyp.split()
    # d[i][j] = edit distance between r[:i] and h[:j]
    d = [[0]*(len(h)+1) for _ in range(len(r)+1)]
    for i in range(len(r)+1): d[i][0] = i
    for j in range(len(h)+1): d[0][j] = j
    for i in range(1, len(r)+1):
        for j in range(1, len(h)+1):
            sub = d[i-1][j-1] + (r[i-1] != h[j-1])
            d[i][j] = min(sub, d[i-1][j] + 1, d[i][j-1] + 1)
    return d[-1][-1] / max(len(r), 1)

print(wer("turn the pump on at six", "turn pump off at six"))  # 0.3333...

In real evaluations use a maintained library such as jiwer and, crucially, a documented text normaliser.

The pitfalls

Normalisation changes the score. Casing, punctuation, numerals (“six” vs “6”), contractions and spelling variants all alter WER. The Whisper paper built a text normaliser through iterative manual inspection, released its code, and showed that choosing one normaliser versus another (they compared against the FairSpeech normaliser) moves scores noticeably on some datasets. Comparing a number from one paper with a number from another is meaningless unless both used the same normaliser.

Benchmarks leak. Web-scale training sets can contain test-set transcripts or near duplicates of public corpora. Prefer an in-house test set of audio the model has never seen, recorded the way your users record it.

Averages hide the tail. A 5 percent mean WER can hide a 40 percent WER on a specific accent, device or noise condition. Report WER by slice. For Whisper-style systems, also report hallucination rate (segments with text but no speech) and for transducers, deletion rate, because those are not equally penalised by one scalar.

WER ignores meaning. Entity error rate, command accuracy or slot-filling accuracy usually track business value better than WER for voice agents. Measure both.

Streaming adds dimensions. Report time-to-first-partial, time-to-final, partial stability (how often the displayed text changes), and WER at final. A model is a point on a latency-accuracy curve, not a single number.

Biasing, Diarization, Edge Deployment and Serving

A transcript that is accurate on average can still be useless if it mangles your product names or cannot say who spoke. These four topics decide whether a model survives contact with a real workload.

Language model fusion and contextual biasing

External language model fusion adds a score from a separately trained text model during beam search. Shallow fusion interpolates log probabilities at each step with a tunable weight; it is simple and works with all three decoder families. For transducers, density-ratio and internal-LM-estimation methods subtract the model’s own implicit language model before adding the external one, which avoids double-counting. CTC benefits the most from fusion because it has no internal language model.

Contextual biasing handles the long tail: names, SKUs, drug names, equipment tags. Common approaches are boosting hotword n-grams during beam search with a trie or finite-state graph, attention-based biasing that encodes a list of phrases and lets the decoder attend to it, and prompting. Whisper supports an initial prompt, a text prefix that conditions the decoder, and teams use it to supply domain vocabulary. The caveat is that prompting is a soft nudge: it can fix spelling of a known term and it can also cause the model to insert that term where it was never spoken. Test biasing for false insertions, not only for hits.

The diarization pipeline

Diarization answers “who spoke when” and is usually a separate pipeline rather than part of the recogniser. The standard modular recipe is: VAD to find speech, segmentation into speaker-homogeneous windows, a speaker-embedding network producing a vector per window, clustering of those vectors, and optional re-segmentation. The ASR output, with word timestamps, is then aligned against speaker turns by overlap.

The weak point is overlapping speech. Clustering assigns each window to one speaker, so crosstalk is mislabelled, and timestamp error in the ASR (Whisper’s native timestamps are quantised to 20 ms and can drift on long files) moves words across turn boundaries. Research systems such as the DiCoW paper (arXiv:2501.00114) condition Whisper directly on diarization output to attack target-speaker transcription; treat that as research direction rather than a drop-in product. For most teams, a modular pipeline with forced alignment for word times is the pragmatic choice.

Edge and on-device deployment

On a phone, a headset or an industrial gateway, memory and power decide the model before accuracy does. Quantisation is the first tool: int8 weights roughly quarter the storage of fp32 and run faster on CPUs and NPUs with integer units, and faster-whisper’s int8 mode illustrates the effect on a desktop CPU (its README reports 1m42s versus 6m58s for the reference implementation on one benchmark machine). For genuinely small footprints, choose a small streaming CTC or transducer model with a few tens of millions of parameters rather than shrinking a large Whisper checkpoint. For a deeper treatment of model sizing for edge, see our post on small language models and edge architecture.

Edge design also changes the evaluation target. A wake-word plus command grammar needs a few hundred milliseconds and tiny memory. Free dictation needs a bigger model and a language model. A pragmatic hybrid runs a small streaming model locally for instant partials, and optionally sends audio to a larger server model for a corrected final, giving the user speed first and accuracy second.

Serving architecture and the end-to-end budget

Figure 4: A production ASR serving path. The batching scheduler and the endpointer are the two components that most often set tail latency.

Clients stream audio over WebSocket or gRPC to a gateway that pins each session to a worker, because streaming models hold per-session state (the encoder cache and decoder hypotheses). A VAD and endpointer gate what reaches the model. A scheduler batches chunks from many sessions into one GPU call; larger batches raise throughput and increase waiting time, so set a maximum batching delay in milliseconds, not only a batch size. Post-processing adds fusion, punctuation, inverse text normalisation (“six” to “6”) and diarization before the transcript and metrics are emitted.

Capacity planning starts from real-time factor (RTF), processing time divided by audio time. If one GPU processes a batch of chunks at an RTF of 0.01 per stream at the batch size you run, it can serve around 100 concurrent real-time streams on paper; the precise figure comes from your own load test, because it depends on model, batch shape and memory for per-session caches. Session state is the usual limit: a stateful encoder cache per stream consumes memory proportional to context length, so concurrent streams per GPU are bounded by memory before compute.

Trade-offs, Gotchas, and What Goes Wrong

Accuracy and latency pull against each other, and training cannot hide it. A streaming model with 80 ms lookahead will lose to the same encoder with unlimited context on offline audio. Do not buy an offline-best model for a live product and expect to bolt streaming on afterwards.

Whisper is not a streaming model. Sliding-window wrappers work for captions where a second of lag is tolerable, but they multiply compute and produce unstable partials. Doing 30-second encoder passes every few hundred milliseconds is paying for the same audio many times.

The decoder you choose determines the failure you inherit. Hallucination is a quiet failure, since the output looks right. Deletions and misspellings from CTC or transducers are noisy failures that downstream systems can often detect. For high-consequence commands, prefer noisy failures.

Domain shift beats model size. A bigger model on out-of-domain audio frequently loses to a smaller model fine-tuned or biased on in-domain data. Collect a few hours of real, labelled audio before spending money on scale.

Normalisation drift. If the model emits “6” and your reference says “six”, you will spend a sprint chasing phantom accuracy bugs. Fix text normalisation in both training targets and scoring.

Multilingual and code-switched speech. Language identification errors on short or accented utterances send the decoder into the wrong language, producing a fluent translation-like output. Pin the language when you know it.

State leaks. Stateful streaming workers that are not cleanly reset between sessions leak context from one caller into the next. Treat the encoder cache and prompt as session-scoped data with the same care as any other user data, particularly for recordings that contain personal information.

Practical Recommendations

Start from the product constraint, not the model. If a human reads the text afterwards and a second or two of lag is fine, use an attention encoder-decoder such as Whisper large-v3 or turbo, run it through an optimised runtime, put a VAD in front, and monitor hallucination signals. If the system acts on the words in real time, use a streaming transducer or CTC model on a Conformer or FastConformer encoder and budget the endpointer explicitly. If the target is a constrained device, choose a small streaming CTC model with a domain language model, and accept that it will not match a large model on open-ended speech.

Whatever you choose, build the evaluation harness first. A small, representative, held-out set with a fixed normaliser is worth more than any leaderboard figure.

A checklist before launch:

Define separate targets for time-to-first-partial, time-to-final and WER on your own slices.
Fix one text normaliser for training targets, references and hypotheses.
Gate audio with a VAD and set explicit endpointing thresholds you have tuned on real conversations.
For Whisper-style models, log average log probability and compression ratio per segment and alert on outliers.
Test contextual biasing for false insertions, not just recoveries.
Load-test concurrent streams with realistic session lengths and measure memory, not only compute.
Confirm the licence on the exact checkpoint you ship.
Plan a fallback: a second decoder or human review for high-stakes utterances.

Frequently Asked Questions

What is the difference between CTC and RNN-T in speech recognition?

Both use a blank symbol and a monotonic alignment, but CTC predicts each frame independently while RNN-T adds a prediction network that conditions on previously emitted tokens. That gives RNN-T an implicit language model and usually better accuracy, at the price of a larger training memory footprint from the time-by-label joint network. CTC is simpler and faster to decode and often needs an external language model to compete on accuracy.

Is Whisper a streaming ASR model?

Not natively. Whisper encodes fixed 30-second windows with full attention and decodes autoregressively, so it needs the whole window before producing reliable text. Community wrappers approximate streaming by re-running the model on a growing buffer and keeping the prefix on which successive passes agree. That is usable for captions but costly and unstable for sub-second voice interaction, where a streaming transducer or CTC model fits better.

What is a Conformer encoder?

A Conformer is a neural network block that combines multi-head self-attention, which models global context, with a depthwise convolution module, which models local acoustic patterns, inside a macaron layout of two half-step feed-forward layers. It was introduced by Gulati and colleagues in 2020 and became the standard encoder for CTC and transducer ASR. FastConformer is a faster variant using 8x downsampling.

Why does Whisper hallucinate?

Whisper’s decoder is an autoregressive language model conditioned on audio. When the audio is silent, noisy or ambiguous, the language prior dominates and the model emits plausible text that was never spoken, or loops on a repeated phrase. Its model cards warn about this. Voice activity detection, temperature fallback, compression-ratio checks and disabling conditioning on previous text are the standard mitigations.

How do you calculate word error rate?

Align the hypothesis to the reference with word-level edit distance, count substitutions, deletions and insertions, and compute (S + D + I) divided by the number of reference words. For example, “turn pump off at six” against “turn the pump on at six” has one deletion and one substitution over six words, so WER is 33.3 percent. Always normalise casing, punctuation and numerals consistently before scoring.

Which is better for real-time ASR: RNN-T or attention encoder-decoder?

RNN-T, or a streaming CTC model, is generally the better fit for real-time use because it emits tokens incrementally and has a bounded, tunable latency. Attention encoder-decoders usually give better accuracy on difficult offline audio but need the full utterance, or chunking workarounds, and can hallucinate. Many teams run a streaming transducer for live partials and an attention model for offline cleanup.