Neural Text to Speech Architecture: VITS, Codec Language Models and Flow-Matching TTS (2026)

Every modern voice assistant, audiobook narrator and AI dubbing tool rests on a deceptively simple contract: take a string of characters and return a waveform that sounds like a person meant it. Ten years ago that contract was met by stitching recorded fragments together. Today neural text to speech meets it with networks that have never been told what a phoneme duration or a pitch contour is, and the field has split into three architectural families that make very different bets about what a speech model should predict.

This matters now because the choice is no longer about quality alone. It is about latency budgets, voice-cloning risk, licensing, and whether your model hallucinates words. A conditional variational autoencoder, a language model over audio tokens and a flow-matching transformer fail in completely different ways, and the failures show up in production, not in demos.

You will leave with a working mental model of how each family is built, what the training objective actually optimizes, the numbers that matter, and a decision framework you can apply to your own product.

What this covers: the classic cascade and its limits, VITS and end-to-end variational synthesis, neural codecs and codec language models, flow-matching and diffusion TTS, a comparison matrix, failure modes, and a practical selection checklist.

Context and Background

Speech synthesis has moved through four eras. Concatenative systems stitched pre-recorded units. Statistical parametric systems (HMM-based) generated vocoder parameters, with a characteristic buzzy sound. Then WaveNet, from DeepMind in 2016, showed that an autoregressive neural network predicting raw audio samples one at a time could close most of the naturalness gap, though at a cost of thousands of sequential steps per second of audio. The fourth era, which we are in, is defined by learned representations end to end.

The canonical neural cascade that dominated 2017 to 2021 had two stages. An acoustic model, such as Tacotron 2 or FastSpeech 2, mapped text or phonemes to a mel spectrogram, a time-frequency picture of the sound on a perceptual frequency scale. A neural vocoder, such as WaveGlow or HiFi-GAN, then turned that spectrogram into a waveform. The split was practical: spectrograms are compact, and vocoders could be trained and swapped independently. It also had a hidden cost, which is that the acoustic model was trained against a spectrogram loss that does not match human perception, and the vocoder was trained on ground-truth spectrograms but fed predicted ones at inference. That train-test mismatch leaves audible artifacts.

Three ideas broke the cascade open. The first was to train the whole text-to-waveform path jointly, which gave us VITS. The second was to stop predicting continuous spectrograms and instead predict discrete tokens from a neural audio codec, which let speech borrow the entire language-model toolkit and gave us zero-shot voice cloning from a few seconds of audio. The third was to treat the spectrogram as a continuous generative target and sample it with flow matching or diffusion, which removed the need for explicit duration modeling.

If you want to see how these ideas show up in a commercial product, our analysis of ElevenLabs Eleven v4 and its expressive TTS architecture covers the production side, and the mirror-image problem of turning audio back into text is covered in streaming ASR architecture across CTC, RNN-T and attention encoder-decoders. For the generative math behind the third family, see flow matching and rectified flow versus diffusion.

Two external anchors are worth bookmarking. The original VITS paper, “Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech” (arXiv 2106.06103), remains the clearest description of end-to-end variational synthesis. The F5-TTS paper (arXiv 2410.06885) is a good entry point to the flow-matching family and its simplifications.

A note on evidence. Throughout this article, numbers taken from papers are attributed to them, and anything that depends on your hardware, text or voice is labeled illustrative. Speech synthesis benchmarks are unusually easy to game, and I will say where.

The Three Architectures of Neural Text to Speech

Short answer: neural text to speech in 2026 comes in three families. VITS-style models learn a latent variable and decode it to audio directly, giving fast and stable synthesis. Codec language models predict discrete audio tokens autoregressively, giving expressive zero-shot cloning. Flow-matching models sample a mel spectrogram from noise conditioned on text, giving robust, parallel generation.

Figure 1: The three routes from text to waveform. The solid path is the classic cascade (acoustic model, mel spectrogram, vocoder). Dashed routes show VITS, which collapses everything into one jointly trained network, and the codec language model route, which predicts discrete tokens and decodes them with a codec.

Figure 1 summarizes the topology. The shared front end is text normalization (expanding “Dr.” and “10/30” into speakable words) and, for many systems, grapheme-to-phoneme conversion (G2P). The back ends differ in what the central model emits: a spectrogram, a latent variable, a token stream, or a spectrogram sampled from noise. That one choice determines nearly everything downstream.

The shared problem: one-to-many alignment

Speech synthesis is a hard sequence-to-sequence problem for a specific reason. A sentence of 80 characters becomes perhaps 300 spectrogram frames, and nothing in the text tells you which character maps to which frames. Worse, the mapping is one-to-many: the same sentence can be spoken in thousands of valid ways, differing in speed, pitch, emphasis and pause placement. A model trained with a mean-squared-error loss on a spectrogram will average over all those valid renditions and produce a flat, over-smoothed “average” voice.

Each family answers these two problems differently. Alignment is handled by attention (Tacotron), by an external duration predictor (FastSpeech), by monotonic alignment search learned during training (VITS, Glow-TTS, Matcha-TTS), by the autoregressive structure itself (codec language models), or by not modeling alignment at all and letting the network infer it (E2 TTS and F5-TTS). The one-to-many problem is handled by a latent variable (VITS), by sampling from a categorical distribution over tokens (codec models), or by sampling from a learned continuous distribution (flow matching and diffusion).

Why the representation is the design decision

The choice of intermediate representation sets the compute profile. A mel spectrogram at a typical 24 kHz setup with a 256-sample hop produces roughly 94 frames per second, each with 80 to 100 mel bins. A neural codec such as EnCodec at 24 kHz produces 75 frames per second, each with up to eight discrete codes. A newer low-rate codec such as Mimi, from Kyutai’s Moshi project, runs at 12.5 frames per second. A raw waveform at 24 kHz is 24,000 samples per second. Every autoregressive model pays its cost per element of that representation, which is why WaveNet was slow and why codec researchers obsess over frame rate.

Evaluation: what the numbers actually measure

You will see four recurring metrics. Mean Opinion Score (MOS) is a human rating from 1 to 5 of naturalness, and its limits are well known: confidence intervals are wide, listener pools differ, and ground truth often scores below 5. Word error rate (WER) is computed by running an automatic speech recognizer on the synthesized audio, so it measures intelligibility and hallucinated or dropped words. Speaker similarity (SIM) is the cosine similarity between speaker embeddings of the prompt and the output, typically from a WavLM-based verification model. Automatic MOS predictors such as UTMOS estimate quality without humans.

Each metric has a blind spot. WER can be low while the prosody is robotic. SIM rewards acoustic match, not speaking style. Automatic MOS is trained on particular datasets and can be fooled. When a paper reports a single leading number, ask which metric, on which test set, with which recognizer.

VITS: Variational End-to-End Synthesis

VITS, from Kim, Kong and Son (ICML 2021), is the architecture that made single-stage neural text to speech practical. Its name expands to Variational Inference with adversarial learning for end-to-end Text-to-Speech, and the expansion is the whole recipe: a conditional variational autoencoder (CVAE), a normalizing flow, adversarial training and a stochastic duration predictor, all trained jointly from text and waveform pairs.

Figure 2: VITS training graph. The posterior encoder maps a real spectrogram to a latent z. A flow transforms z so it can be matched to a text-conditioned prior. The decoder reconstructs audio from z, and a discriminator pushes the output toward realism.

Training path versus inference path

During training, VITS has access to the real audio. A posterior encoder, built from non-causal WaveNet-style residual blocks, takes the linear-scale spectrogram of the ground-truth waveform and produces a distribution over a latent variable z at the frame rate. A decoder, essentially the HiFi-GAN generator, turns z into a waveform. Because the decoder only sees short windowed segments of z during training, the model is memory-efficient.

The text side supplies a prior. A transformer text encoder reads phonemes and outputs a prior distribution over latents per phoneme. A normalizing flow then makes the prior more expressive: it applies invertible transformations (affine coupling layers) so a simple Gaussian can be warped into a complex distribution that better matches the posterior. The training loss includes a KL-divergence term that pulls the posterior toward the flow-transformed prior, a reconstruction term on the mel spectrogram of the decoded waveform, adversarial and feature-matching losses from the discriminators, and a duration loss.

At inference, there is no real audio. The model samples z from the text-conditioned prior, runs the flow in reverse, and decodes with the HiFi-GAN generator. All of this is non-autoregressive, so a short sentence is synthesized in one parallel pass.

Monotonic alignment search

The text has far fewer elements than the latent sequence, so something must decide how many latent frames each phoneme owns. VITS borrows Monotonic Alignment Search (MAS) from Glow-TTS. MAS is a dynamic-programming algorithm that finds the most likely monotonic, non-skipping alignment between the text prior and the latent, maximizing the likelihood of the data. It runs inside the training loop with no external aligner, and the recovered durations become the targets for the duration predictor.

The monotonic constraint encodes a fact about speech: you never read the words of a sentence out of order. That restriction removes the attention failures that plagued Tacotron, where the model would skip or repeat words. This is the main reason VITS-family systems are rarely accused of hallucinating text.

The stochastic duration predictor

Fixed durations make speech sound mechanical, because real speakers vary the length of every syllable. VITS therefore models duration with its own flow-based generative model, conditioned on text. Sampling different noise gives different rhythms for the same sentence. This is the cheap, targeted way VITS captures the one-to-many nature of speech: rhythm varies through duration sampling, and the rest of the variation lives in the latent z.

What VITS gets right and where it stalls

The paper reported naturalness close to ground truth on the single-speaker LJ Speech dataset, with MOS around the mid-4s on a five-point scale, ahead of the two-stage systems it compared against, and it extended to the multi-speaker VCTK dataset with speaker embeddings. Those are paper-reported numbers on clean, studio-recorded corpora, and I would not extrapolate them to noisy or expressive data.

The strengths are practical. VITS models are small, often tens of millions of parameters, run well faster than real time on a CPU for short utterances, and are straightforward to fine-tune for a new voice with tens of minutes of clean audio. That is why VITS descendants sit inside many open source offline and edge TTS toolchains, including the Piper ecosystem.

The limits are equally concrete. VITS is trained on a modest amount of transcribed studio speech, so it generalizes poorly to unseen speakers without fine-tuning. Its latent is a single vector per frame trained for reconstruction, so controllable expressiveness (a whisper, a laugh, a shout) is weak unless you add conditioning. Training is also notoriously finicky: GAN plus flow plus VAE terms must be balanced, and instability is a common complaint. Later work such as YourTTS added zero-shot multi-speaker capability on top of VITS, and the StyleTTS 2 line replaced the flow with a style diffusion latent while keeping a fast decoder. The lightweight open model Kokoro (82 million parameters, Apache-licensed weights per its model card) builds on StyleTTS 2 with an inverse-STFT-based decoder, and illustrates how far a small, fast, non-autoregressive design can go for fixed, curated voices.

Neural Codecs and Codec Language Models

The second family starts from an observation that sounds almost too convenient: if you can turn audio into a short sequence of discrete tokens and back without much loss, then speech synthesis becomes next-token prediction, and everything learned from large language models transfers.

Neural audio codecs and residual vector quantization

A neural audio codec is an autoencoder with a quantized bottleneck. SoundStream (Google, 2021) and EnCodec (Meta, 2022) are the reference designs. A convolutional encoder downsamples the waveform to a low frame rate. A quantizer then converts each frame into integer codes, and a decoder reconstructs the waveform. Training uses reconstruction, multi-scale spectral and adversarial losses.

The quantizer is the clever part. A single codebook with a million entries would be needed to represent a frame precisely, which is impractical. Residual vector quantization (RVQ) solves this by stacking codebooks. The first codebook quantizes the frame embedding as well as it can. The second codebook quantizes the residual error left over, the third quantizes the remaining error, and so on. The sum of the chosen codebook vectors approximates the original embedding, and each codebook contributes progressively finer detail.

Concrete numbers help. EnCodec’s 24 kHz model produces 75 frames per second with codebooks of 1,024 entries, which is 10 bits per code. With eight codebooks, the bitrate is 75 times 8 times 10, or 6,000 bits per second, which is the 6 kbps operating point. Using only the first two codebooks gives 1.5 kbps at lower fidelity. A ten-second clip at 8 codebooks therefore becomes 750 frames times 8 codes, or 6,000 integers. That is the sequence a language model must learn to produce.

Two properties of RVQ shape the downstream model design. The first codebook carries coarse content: much of the phonetic structure, rough pitch and speaker identity. Later codebooks carry fine acoustic detail. Second, the codes are strongly dependent across the stack, so models must decide whether to predict all codebooks jointly, one after another, or in a staggered pattern.

VALL-E: speech as conditional language modeling

Microsoft’s VALL-E (January 2023, arXiv 2301.02111) reframed TTS as conditional codec language modeling. It was trained on about 60,000 hours of English speech from the LibriLight corpus, orders of magnitude more than typical single-speaker TTS sets, with transcripts generated by an ASR system. Its headline claim was zero-shot voice cloning from a three-second enrolled recording, which it treated as an in-context learning problem, mirroring how large text models follow a prompt.

Figure 3: Two-stage codec language model generation. An autoregressive model predicts the first codebook token by token, then a non-autoregressive model fills in the remaining codebooks in parallel, and the codec decoder produces the waveform.

The generation process in Figure 3 has two stages. An autoregressive (AR) transformer takes the phoneme sequence and the codes of the three-second acoustic prompt, and predicts the first-codebook tokens of the target one at a time. A non-autoregressive (NAR) transformer then predicts codebooks two through eight, each conditioned on the codebooks already produced, with every frame in a layer generated in parallel. The result is passed through the EnCodec decoder.

The conceptual leap is the prompt. Because the AR model sees the codes of a real speaker’s voice as a prefix, it continues in that voice, including its room acoustics and recording channel. Nothing is fine-tuned, and no explicit speaker embedding is required.

The cost of autoregression: speed, drift and hallucination

Autoregressive generation is also where this family hurts. Speed is the obvious cost: producing 75 tokens per second of audio one at a time means a ten-second sentence needs about 750 sequential AR steps for the first codebook alone. Optimized implementations with key-value caching make this tractable, and streaming is natural because tokens come out in order, but the latency floor is real.

Robustness is the subtler cost. Unlike VITS, a codec language model has no monotonic alignment constraint, since text and audio are simply concatenated into one sequence. The model can therefore skip words, repeat phrases, loop on a silence, or stop early, especially on long or unusual text. Sampling makes it worse: top-k or nucleus sampling gives natural variability, but a single unlucky token can derail the rest of a sentence. VALL-E 2 and several follow-ups address this with repetition-aware sampling and grouped code modeling; the broader lesson is that the failure modes of language models, repetition loops and hallucination, carry over to audio.

Variations on the theme

The design space around VALL-E is large, and a few distinctions are worth knowing.

Single-stream versus multi-stream. Instead of AR plus NAR, some systems flatten the codebooks into one long sequence, or use a delay pattern that staggers codebooks so a single transformer can predict several per step. Meta’s MusicGen popularized the delay pattern for audio. Kyutai’s Moshi uses a smaller, low-rate codec (Mimi, 12.5 Hz) so that one temporal transformer plus a small depth transformer handle the codebooks per frame.

Semantic versus acoustic tokens. Acoustic codec tokens capture everything, including noise. Semantic tokens, derived from self-supervised models such as HuBERT or from a supervised ASR encoder, capture mostly content. Two-stage designs such as AudioLM and SPEAR-TTS first model semantic tokens (what is said and how), then map them to acoustic tokens. CosyVoice from Alibaba uses supervised semantic tokens from an ASR-trained encoder, then converts them to a mel spectrogram with a flow-matching model, which is a hybrid of this family and the next.

LLM backbones. A growing number of open and commercial systems attach audio tokens to a pretrained text LLM, so the model inherits language understanding and can follow instructions about emotion or style in plain text. Examples include Orpheus (built on a Llama-family model and using the SNAC codec, per its repository) and various LLM-based TTS systems from Chinese labs. The appeal is expressive control and tag-based emotion; the risk is the same hallucination profile as text LLMs plus a large inference footprint.

Voice cloning risk. Because three seconds of audio can condition a voice, the same technology that enables accessibility tools enables impersonation. Many vendors gate cloning behind consent verification, and watermarking research (for example, AudioSeal from Meta) aims to make synthetic audio detectable. Engineering teams should treat cloning as a security feature with an abuse model, not only as a quality feature.

Flow-Matching and Diffusion TTS

The third family treats speech as a continuous object to be sampled, much like image generation. Instead of predicting the next token, the model learns to transform random noise into a mel spectrogram, with the transformation guided by text.

The flow-matching idea in one page

Flow matching, introduced by Lipman and colleagues in 2022, trains a network to predict a velocity field. You define a probability path between a simple noise distribution and the data distribution. In the simplest case, the optimal-transport conditional path, a training example is a straight-line interpolation between noise x0 and data x1: x_t = (1 – t) x0 + t x1. The network is trained, with plain regression, to predict the constant velocity x1 – x0 at the point x_t, given time t and conditioning. At inference you start from noise and integrate an ordinary differential equation (ODE) along the learned field for a handful of steps.

Compared with classic diffusion, the training target is simpler, the paths are straighter, and fewer solver steps are usually needed. If the mathematics is new to you, the comparison in our flow matching versus diffusion guide walks through it with the sampling-cost trade-offs.

From Grad-TTS and Matcha-TTS to E2 and F5

Diffusion arrived in TTS early. Grad-TTS (2021) used a score-based decoder to generate mel spectrograms conditioned on text, with a Glow-TTS-style aligner, and produced high quality at the cost of many denoising steps. Matcha-TTS (ICASSP 2024) swapped in optimal-transport conditional flow matching with a compact decoder and kept monotonic alignment search, which cut the number of required ODE steps sharply and made a small, fast, non-autoregressive model.

The next step removed the alignment machinery altogether. E2 TTS (Microsoft, 2024) showed that you can pad the character sequence with filler tokens to the length of the target mel spectrogram, then train a flow-matching transformer to fill in a masked region of the mel conditioned on the text and the unmasked audio context. The model works out alignment implicitly, with no phoneme aligner, duration model or grapheme-to-phoneme step.

F5-TTS (Chen and colleagues, arXiv 2410.06885, October 2024) kept that recipe and fixed what made E2 slow and fragile to train. It uses a diffusion transformer (DiT) backbone, refines the text representation with ConvNeXt V2 blocks so text and speech can align more easily, and introduces Sway Sampling, an inference-time scheme that changes the spacing of ODE timesteps to spend more steps early in the trajectory where the coarse structure is decided. The paper trained on the roughly 100,000-hour multilingual Emilia corpus and reports a real-time factor around 0.15 on its test setup, which I treat as paper-reported and hardware-specific.

Figure 4: Flow-matching TTS inference in the E2 and F5 style. A DiT predicts the velocity field from noise, filler-padded text and a masked reference mel, an ODE solver integrates it in a few tens of steps, and a vocoder turns the mel into audio.

How inference works, concretely

Reading Figure 4 from the top: the system encodes the reference clip as a mel spectrogram and appends a region of noise whose length equals the target duration. The text, reference transcript plus the new text, is padded with filler tokens to the total mel length. The DiT takes the noisy mel, the text embedding, the masked reference and a time embedding, and outputs a velocity. An ODE solver such as Euler or midpoint integrates for typically 16 to 32 function evaluations. Classifier-free guidance, borrowed from image diffusion, is used to strengthen adherence to the text, at the cost of roughly doubling compute per step. A vocoder such as Vocos finally converts the mel to a waveform.

Two observations follow. First, the total duration must be chosen up front. F5-style systems typically estimate it from the ratio of reference duration to reference text length, or accept a user-specified duration. That is a feature (you can time speech to video), but a bad estimate makes speech sound rushed or drag. Second, the entire utterance is generated in parallel, so latency depends on the number of ODE steps and the length of the utterance, not on the number of tokens produced sequentially.

A runnable sketch

The following is a minimal, framework-light PyTorch sketch of the conditional flow-matching objective and an Euler sampler. It is illustrative, not an F5-TTS reimplementation, and the network v_theta is left abstract.

import torch

def cfm_loss(v_theta, x1, cond, sigma_min=1e-4):
    """Optimal-transport conditional flow matching loss.
    x1:   target mel batch, shape (B, T, D)
    cond: dict with text embeddings and masked reference mel
    """
    B = x1.size(0)
    x0 = torch.randn_like(x1)                    # noise sample
    t = torch.rand(B, 1, 1, device=x1.device)    # random time in [0, 1]
    xt = (1 - (1 - sigma_min) * t) * x0 + t * x1 # straight-line path
    target = x1 - (1 - sigma_min) * x0           # constant velocity
    pred = v_theta(xt, t.squeeze(), cond)
    return torch.mean((pred - target) ** 2)

@torch.no_grad()
def sample(v_theta, cond, shape, steps=16, cfg=2.0, sway=-1.0):
    x = torch.randn(shape)
    # sway sampling: warp uniform times toward early steps
    u = torch.linspace(0, 1, steps + 1)
    ts = u + sway * (torch.cos(torch.pi / 2 * u) - 1 + u)
    for i in range(steps):
        t, dt = ts[i], ts[i + 1] - ts[i]
        v_c = v_theta(x, t.expand(shape[0]), cond)
        v_u = v_theta(x, t.expand(shape[0]), None)  # unconditional
        v = v_u + cfg * (v_c - v_u)                 # classifier-free guidance
        x = x + dt * v                              # Euler step
    return x  # mel; pass to a vocoder

The sway warp follows the form given in the F5-TTS paper (a negative coefficient shifts steps toward the start of the trajectory); treat the exact constants as tunable rather than canonical. Note how little code the training objective needs: that simplicity is a large part of why this family spread quickly.

What flow matching trades away

The robustness win is real. Because every frame is generated jointly with the whole text visible, flow-matching TTS rarely skips or loops over words in the way autoregressive models can, and reported WERs on zero-shot benchmarks are competitive with or better than codec language models of similar scale. But the family has costs of its own. Total duration must be known or estimated before generation. Streaming is awkward because the whole mel is denoised together, so low-latency use requires chunked variants (CosyVoice 2 and related work address streaming with chunk-aware flow matching). Training data requirements are large, since implicit alignment needs scale to work. And licensing can be restrictive: the released F5-TTS weights carry a non-commercial license because of the Emilia training data, so check the repository before shipping.

Side-by-Side: How the Families Differ

The table below summarizes the structural differences. Entries describe typical designs, not every model in a family.

Property	VITS family	Codec language model	Flow-matching TTS
Central output	Latent z, decoded to waveform	Discrete codec tokens	Continuous mel (then vocoder)
Generation	Parallel, single pass	Autoregressive (plus optional NAR)	Parallel, iterative ODE steps
Alignment	Monotonic alignment search	Implicit in sequence	Implicit via filler padding, or MAS
Typical size	Tens of millions of parameters	Hundreds of millions to billions	Hundreds of millions
Zero-shot cloning	Weak without extensions	Strong, in-context	Strong, in-context
Streaming	Easy (chunked decoder)	Natural (token stream)	Needs chunked variants
Hallucination risk	Low	Highest (skips, loops)	Low to moderate
Expressive control	Limited	Strong, especially with LLM backbones	Moderate, via reference
Training stability	Finicky (GAN, VAE, flow)	Standard LM training	Stable regression loss
Best for	Edge, fixed voices, low cost	Expressive, conversational, cloning	Robust cloning, dubbing, offline

A latency and cost model you can reason with

Instead of quoting vendor numbers, here is the arithmetic that governs each family, which you can run with your own measurements.

For a codec language model at 75 frames per second, time to first audio is roughly the prompt prefill time plus the time to generate enough AR tokens to fill the first playback chunk. If your decode speed is S tokens per second for the first codebook, then generating one second of audio costs 75 / S seconds on the AR stage alone. Real-time streaming needs S well above 75, plus headroom for the NAR stage and codec decoding. At a hypothetical S of 300 tokens per second, the AR stage runs at about a quarter of real time. Treat that figure as illustrative.

For a flow-matching model, cost scales as the number of function evaluations times the cost of one forward pass over the whole utterance, and classifier-free guidance doubles it unless the conditional and unconditional passes are batched. With 16 steps and guidance, that is 32 transformer passes over the full sequence. Because passes are parallel over time, a GPU handles this efficiently for sentences, but cost grows with utterance length, which is why long-form audio is usually chunked at sentence boundaries.

For VITS, cost is a single forward pass of a small network. This is the family where the cost per second of audio can sit at a fraction of a CPU core, which is why it dominates offline and embedded use.

Trade-offs, Gotchas, and What Goes Wrong

Most TTS failures are not quality failures; they are failures in a specific part of the pipeline that a demo never exercises.

Text normalization is where production breaks first. Neural models learn from normalized text. Numbers, dates, currencies, units, abbreviations and mixed-language strings (“Order #4512 shipped to St. Louis on 10/30”) need a rule-based or learned normalizer before the model sees them. Models without a G2P front end guess pronunciations of rare names and acronyms from characters alone, and they guess wrong in an unpredictable way. Teams commonly find that fixing normalization improves perceived quality more than swapping models.

Hallucination is a family-specific risk. Codec language models skip, repeat or invent words, and the rate rises on long input, rare words and low-quality prompts. Mitigations: sentence-level chunking, repetition penalties, constrained sampling, and an automatic ASR check that regenerates when the transcript of the output diverges from the input. That last pattern, synthesize then verify with ASR, is cheap and effective, and it is why the ASR and TTS stacks should be designed together.

The prompt is part of the model. In zero-shot cloning, the reference clip determines the output voice, prosody, noise floor and even microphone coloration. A noisy or emotional prompt produces a noisy or emotional output. A prompt with wrong transcript text breaks the alignment for filler-padded flow-matching models. Build prompt quality checks (signal-to-noise ratio, duration, transcript match) into any cloning feature.

Prosody over long context is still weak. Most models are trained on utterances of seconds, not paragraphs. They have no explicit notion of discourse structure, so a list read aloud may stress the wrong items and a long passage may drift in pace. Chunk-level generation with consistent conditioning on the same reference helps with voice consistency but not with narrative arc.

Vocoder and codec ceilings. A model can only sound as good as its decoder allows. Codecs at low bitrate smear fine detail, and mel vocoders can produce metallic artifacts on high-pitched or singing voices. If your output has a consistent timbral glitch regardless of text, suspect the decoder, not the acoustic model.

Metric overfitting. Leaderboards that report WER and SIM encourage models that are accurate and similar but flat. A system can win on those and still lose in a listening test. Always run a small blind A/B or MOS study with your own text and target listeners; even 20 listeners rating 30 sentences will reveal differences that automatic metrics miss.

Legal and safety constraints. Voice cloning intersects with privacy, consent and impersonation law, and rules differ by jurisdiction. Training-data licenses constrain model licenses, as the F5-TTS non-commercial weights show. This article is engineering analysis, not legal advice.

Evaluation leakage. Many zero-shot benchmarks draw test speakers from LibriSpeech or LibriTTS, which also appear in training data of some systems. When a paper’s gain over a baseline is small, check for train-test overlap before you believe it. I have not audited the overlap for the specific systems named here.

Practical Recommendations

Start from your constraints, not from the newest model on a leaderboard.

If you need a small number of fixed voices on-device or at very low cost per hour, use a VITS-family or StyleTTS-derived model. Fine-tune on clean recordings, put effort into the text front end, and you will get stable, fast output. This is also the right default for accessibility readers and offline assistants where predictability beats expressiveness.

If you need zero-shot cloning with conversational expressiveness and can tolerate more serving complexity, use a codec language model, ideally with an LLM backbone for style control. Budget for hallucination handling from day one: sentence chunking, ASR verification, and a retry path. Prefer designs with streaming-friendly codecs if latency matters, and measure time to first audio, not just real-time factor.

If you need robust cloning for offline production work such as dubbing, narration or audiobook drafts, and exact duration control is valuable, use a flow-matching model. Check the license of the weights, plan for sentence-level chunking, and tune the number of ODE steps against your quality bar; the quality curve typically flattens well before the maximum.

If you need a hosted service, the architecture matters less than the contract: latency percentiles, concurrency limits, voice consent policy, and pricing per character. Our Eleven v4 breakdown shows what to ask a vendor.

A short selection checklist:

Define the latency target as time to first audio at the 95th percentile, and whether you need streaming.
List voices: fixed and curated, or arbitrary user-provided references.
Decide the hallucination tolerance. Is a skipped word an annoyance or an incident?
Confirm weight and data licenses permit your commercial use.
Build text normalization and pronunciation lexicons before model comparison.
Add an ASR round-trip check and log the word error rate per request.
Run a blind listening test on your own text with your own listeners.
Add consent checks and, where possible, watermarking for any voice-cloning feature.
Pin model, codec and vocoder versions together; swapping a decoder changes the sound.

Frequently Asked Questions

What is neural text to speech?

Neural text to speech is speech synthesis in which neural networks learn the mapping from text to audio directly from recorded speech, instead of using hand-built rules or stitched recordings. Modern systems convert text to phonemes or characters, predict an intermediate representation such as a spectrogram, latent variable or discrete audio tokens, and then decode it to a waveform with a neural vocoder or codec decoder. Quality now approaches human recordings on clean benchmarks.

How is VITS different from Tacotron 2?

Tacotron 2 is a two-stage cascade: an attention-based model predicts a mel spectrogram, and a separate vocoder renders it. VITS trains text-to-waveform end to end with a variational autoencoder, a normalizing flow and adversarial losses, and it learns alignment with monotonic alignment search rather than attention. The result is parallel inference, fewer skipped or repeated words, and no spectrogram mismatch between training and inference, at the price of a more delicate training setup.

What is a codec language model?

A codec language model is a transformer that generates speech by predicting the discrete tokens of a neural audio codec, such as EnCodec, instead of predicting waveform samples or spectrograms. Text and a short voice prompt condition the model, which then produces codec tokens autoregressively, and a codec decoder converts them to audio. VALL-E is the best-known example. The approach enables in-context voice cloning from seconds of audio but inherits language-model failures such as repetition.

Why is flow-matching TTS more robust than autoregressive TTS?

Flow-matching TTS generates the entire spectrogram in parallel while the model sees the full text, so there is no sequence of sampled tokens in which one bad choice can derail everything after it. The training objective is a simple regression to a velocity field, which is stable. Skipped and looped words are therefore rarer, though duration must be set in advance and streaming needs chunked variants. This is a tendency, not a guarantee.

Which neural TTS architecture is best for real-time use?

For the lowest compute and easiest streaming, VITS-style or small StyleTTS-derived models are usually the safest, running faster than real time on CPUs. Codec language models stream naturally token by token and can reach low time to first audio on a GPU, but need careful optimization. Flow-matching models are fast per utterance on GPUs yet need chunking for live use. Measure time to first audio on your own hardware before deciding.

Can I use these models commercially?

It depends on the specific weights. Architectures are not licensed, but checkpoints are. Some open models use permissive licenses such as Apache 2.0, as the Kokoro model card states, while others restrict commercial use because of training-data terms, as with the released F5-TTS weights. Always read the license on the exact checkpoint, check the training-data terms, and consult counsel for voice cloning, which has additional consent and personality-rights considerations.

Neural Text-to-Speech Architecture: VITS, Codec Language Models and Flow-Matching TTS (2026)