Speaker Diarization Architecture: Who Spoke When with pyannote, End-to-End Neural Diarization and Streaming Pipelines

The same open-source model scores 11.2% error on one benchmark and 44.6% on another. Nothing is broken; that spread is the honest state of speaker diarization, the task of answering “who spoke when” in a recording without knowing in advance who the speakers are or how many there are. Diarization looks like a solved preprocessing step until a sales call transcript attributes the customer’s objection to your account executive.

It matters now because transcripts have become inputs to automation. Meeting summarisers, compliance monitors, call-centre analytics and voice agents all act on “speaker A said X”. A wrong speaker label is no longer a cosmetic defect; it is a wrong fact fed to a language model. Meanwhile the architecture choices widened: modular pyannote pipelines, end-to-end neural diarization, and streaming models with speaker caches all ship as usable open weights.

You will leave with a reference architecture, a precise understanding of how diarization error rate is scored, runnable Python for offline, word-level and streaming diarization, and a failure-mode checklist.

What this covers: the modular pipeline (VAD, segmentation, embeddings, clustering), end-to-end neural diarization and attractors, streaming with speaker caches, DER scoring rules, published benchmarks, joining speakers to ASR words, cost, latency and failure modes.

Context and Background

Diarization has been an evaluated task since the NIST Rich Transcription (RT) campaigns of the 2000s, which defined the diarization error rate (DER) still used today. For most of the following decade the dominant recipe was a cascade: detect speech, cut it into short uniform segments, turn each segment into a fixed-length speaker vector, then cluster the vectors so that each cluster is one speaker. The vectors evolved from i-vectors to neural x-vectors (Snyder et al., ICASSP 2018) and then to deeper architectures such as ECAPA-TDNN (Desplanques et al., Interspeech 2020) and ResNet-based embeddings from toolkits like WeSpeaker.

The cascade had one structural blind spot: it assumed exactly one speaker per segment. Conversational speech overlaps constantly, through backchannels (“mm-hm”), interruptions and laughter. A segment-and-cluster system cannot label overlap at all, so every overlapped second becomes missed speech in the score.

Two responses emerged. The first, End-to-End Neural Diarization (EEND), from Hitachi and Johns Hopkins researchers in 2019, trained a single network to output a per-frame, per-speaker activity matrix directly, which handles overlap natively. The second, popularised by the open-source pyannote.audio toolkit, kept the cascade but replaced its weakest stage with a neural, overlap-aware segmentation model that runs on short windows, then stitched windows together with embeddings and clustering. That hybrid is what most production stacks run today.

The incumbents in 2026 are clear. pyannote.audio, now on its 4.x line (4.0.0 was published to PyPI on 29 September 2025), ships the open speaker-diarization-community-1 pipeline and a hosted precision-2 service (pyannote.audio repository). NVIDIA NeMo offers Sortformer, an end-to-end model with a streaming variant. WhisperX, a popular transcription wrapper, delegates its diarization to pyannote. Cloud speech APIs, including the models covered in our MAI-Transcribe-2 explainer, expose diarization as a flag, usually without saying which architecture sits behind it.

Diarization also sits next to automatic speech recognition (ASR) rather than inside it in most systems. If you need a refresher on how streaming recognisers produce word timestamps, our guide to streaming ASR architecture with CTC, RNN-T and Whisper covers the decoder side that diarization output must later be joined to.

Speaker Diarization Reference Architecture: Local Decisions, Global Identity

A speaker diarization system makes two different decisions. Locally, within a few seconds of audio, it decides which voices are active in each frame, including overlaps. Globally, across the whole recording, it decides which local voices are the same person. Every architecture, from pyannote to EEND to streaming Sortformer, is defined by where it makes that global identity decision.

Figure 1: The hybrid speaker diarization pipeline. Local overlap-aware segmentation runs on sliding windows, embeddings summarise each local speaker, and a global clustering step assigns persistent identities before reconstruction and export.

Read the figure left to right. Audio is resampled to 16 kHz mono and sliced into overlapping windows. A neural segmentation model labels each frame of each window with up to a few local speakers. For every local speaker in every window, an embedding model produces a vector from only the frames where that speaker is active alone. A clustering stage groups those vectors across the recording into global speakers. Finally, the local activity is re-labelled with global identities, overlapping windows are aggregated, and the result is written as RTTM (Rich Transcription Time Marked), the ten-field, one-turn-per-line format that every scoring tool reads.

Local segmentation solves overlap, not identity

The segmentation stage is where modern pipelines diverged from the 2010s recipe. pyannote’s segmentation model, described by Plaquet and Bredin at Interspeech 2023, uses a “powerset” formulation. Instead of predicting an independent probability for each speaker, it predicts one class per frame from the set of all allowed speaker combinations. With at most three speakers per 10-second chunk and at most two simultaneously, that is seven classes: non-speech, speaker 1, 2 or 3 alone, and the three pairs.

Why does this matter? Multi-label sigmoid outputs need a tuned threshold per speaker to decide activity, and that threshold interacts badly with overlap. A powerset softmax picks exactly one configuration per frame, so the detection threshold disappears as a hyperparameter. The cost is a combinatorial label space, which is why the model is restricted to a small number of local speakers. That restriction is fine locally: three or more people rarely all speak within one 10-second window in a meeting.

Notice what segmentation does not do. Its “speaker 1” in window 40 has no relationship to “speaker 1” in window 41. The labels are permutation-invariant by design, because the model was trained with a loss that ignores label order. Local segmentation is therefore a voice-activity and overlap detector that also separates voices within a window. Identity is someone else’s problem.

Embeddings carry identity across windows

The embedding model is a speaker-verification network repurposed. Trained to tell thousands of speakers apart, it maps a stretch of speech to a vector, typically 192 or 256 dimensions, where cosine similarity approximates “same voice”. x-vectors used a time-delay neural network with statistics pooling; ECAPA-TDNN added channel attention, multi-layer feature aggregation and attentive statistics pooling; WeSpeaker’s ResNet models brought image-style residual networks to the same job. pyannote’s community-1 model card cites WeSpeaker (Wang et al., 2023) as its embedding source.

The subtle engineering choice is which frames feed the embedding. If you embed a window where two people overlap, you get a vector somewhere between both voices, which then poisons clustering. pyannote handles this by masking: each local speaker’s embedding is computed only from frames where segmentation says that speaker is active and, preferably, alone. Short local speakers yield few clean frames and noisy embeddings, which is one reason backchannels are mislabelled so often.

Clustering is the global identity decision

Clustering converts thousands of local embeddings into a handful of global speakers, and it is where the speaker count is decided when you do not supply one. The three families you will meet are agglomerative hierarchical clustering (AHC), spectral clustering and VBx.

AHC repeatedly merges the closest pair of clusters until the closest pair is further apart than a threshold. It is simple, deterministic and its threshold is interpretable, but it needs the full pairwise distance matrix and is sensitive to that threshold. Spectral clustering builds an affinity graph, computes eigenvectors of its Laplacian, and estimates the speaker count from the eigengap; NeMo’s classic clustering pipeline uses a normalised maximum eigengap variant. VBx (Landini et al., Computer Speech and Language, 2022) is a Bayesian hidden Markov model over the embedding sequence, where each state is a speaker and transitions encode the prior that speakers do not switch every frame. It is usually initialised from AHC output and refines both assignments and count.

The community-1 model card cites VBx under speaker clustering, and pyannote’s own changelog credits community-1 with “marked reductions in speaker confusion” and “more reliable speaker counting” relative to the 3.1 pipeline. If you can supply the speaker count, every one of these methods gets easier: pyannote accepts num_speakers, min_speakers and max_speakers arguments, and a two-party phone call should always be diarized with num_speakers=2.

Where end-to-end neural diarization moves the decision

End-to-end neural diarization collapses the pipeline into one network. The original EEND (Fujita et al., Interspeech 2019) used a bidirectional LSTM, later self-attention (SA-EEND), to emit a frames-by-speakers activity matrix trained with a permutation-invariant loss. Because outputs are per-speaker sigmoids, overlap is natural. The problem was a fixed maximum speaker count baked into the output layer.

EEND-EDA (Horiguchi et al., arXiv 2005.09921, 2020) fixed that with encoder-decoder attractors. An LSTM encoder reads the frame embeddings, and an LSTM decoder emits one “attractor” vector per speaker until a learned existence probability falls below a threshold. Each frame’s activity for a speaker is the sigmoid of the dot product between frame embedding and attractor. The paper reports 15.29% DER on CALLHOME with an unknown number of speakers versus 19.43% for an x-vector clustering baseline, and 8.07% on the two-speaker CALLHOME subset versus 9.54% for SA-EEND.

So in EEND-EDA, the global identity decision is the attractor set. NVIDIA’s Sortformer takes another route: it trains with a “sort loss” that orders speakers by their first appearance, which removes the permutation ambiguity and makes outputs directly usable as speaker tokens alongside ASR. Its streaming variant then stores a cache of past speaker frames so identity survives across chunks, which we examine in the streaming section.

The practical lesson is that EEND-family models are excellent at local overlap and at short recordings with few speakers, and they struggle when a recording is long or contains more speakers than training data covered. Hybrid pipelines are the inverse: weaker on dense overlap at window boundaries, stronger on hour-long recordings with many speakers. That is why pyannote’s design, an EEND-style local model inside a clustering shell, has been so durable.

How Diarization Error Rate Is Actually Scored

Diarization error rate is the sum of three durations divided by total reference speech time: missed speech, false-alarm speech, and speaker confusion. The definition comes from the NIST RT-09 evaluation plan and is implemented by NIST’s md-eval.pl, which the widely used dscore tool wraps (dscore on GitHub).

Each term has a precise meaning. Missed speech is reference speech the system labelled as non-speech, or overlapped speech where the system output fewer speakers than the reference. False alarm is system speech where the reference has none, or extra speakers beyond the reference count. Confusion is time where speech was detected but attributed to the wrong speaker after an optimal one-to-one mapping between reference and system labels, computed with the Hungarian algorithm. Because errors are summed against reference time, DER can exceed 100%.

A worked example

Consider a 10-minute meeting with 600 seconds of reference speaker time, counting each overlapped second once per active speaker. Suppose your system misses 30 seconds, mostly backchannels and overlap, invents 18 seconds of speech during a door slam and keyboard noise, and attributes 42 seconds to the wrong person after a long monologue drifted its embedding. DER is (30 + 18 + 42) / 600 = 15.0%. These numbers are illustrative, but the shape is typical: the confusion term is the one that users notice, because it changes who said what.

Now rescore with a 0.25-second forgiveness collar around every reference boundary. A meeting with 400 speaker turns has 800 boundaries; at 0.5 seconds of excluded time per boundary, up to 400 seconds can be removed from scoring if collars do not overlap. In practice collars overlap and the reduction is smaller, but you can easily halve the missed and false-alarm terms. The same system output may report 15% without a collar and 9% with one.

Why published numbers rarely compare

Three scoring switches change DER more than most model improvements do:

Collar. dscore defaults to 0 ms. Older CALLHOME and AMI papers commonly used 0.25 seconds.
Overlap. dscore includes overlapped reference speech by default; the --ignore_overlaps flag drops it. Excluding overlap flatters systems that cannot handle it.
Oracle components. Some papers use oracle voice activity detection or the oracle speaker count. Both remove whole error categories.

pyannote states that its benchmarks use “fully automatic processing, no forgiveness collar, nor skipping overlapping speech”. NVIDIA’s Sortformer card includes overlap everywhere but uses a 0 s collar for DIHARD III and 0.25 s for CALLHOME. These are both honest choices; they are simply not the same choice, so a CALLHOME number from one cannot be compared with a CALLHOME number from the other.

DIHARD II also introduced the Jaccard error rate (JER), which computes a per-speaker error and averages it, so a quiet speaker who is completely missed counts as much as a dominant one. dscore notes that JER “will never exceed 100%”. When your product cares about minority speakers, such as a patient in a doctor-patient conversation, report JER alongside DER.

Published DER, with the protocol attached

The table below collects published numbers only, each with its protocol. Do not compare across rows from different sources.

System	Dataset	DER %	Protocol	Source
pyannote 3.1 (legacy)	AMI IHM	18.8	no collar, overlap scored	pyannote README, 2025-09
pyannote community-1	AMI IHM	17.0	no collar, overlap scored	pyannote README, 2025-09
pyannote precision-2	AMI IHM	12.9	no collar, overlap scored	pyannote README, 2025-09
pyannote community-1	DIHARD 3 full	20.2	no collar, overlap scored	pyannote README, 2025-09
pyannote community-1	VoxConverse v0.3	11.2	no collar, overlap scored	pyannote README, 2025-09
pyannote community-1	CALLHOME part 2	26.7	no collar, overlap scored	pyannote README, 2025-09
pyannote community-1	AVA-AVD	44.6	no collar, overlap scored	pyannote README, 2025-09
Streaming Sortformer v2, 30.4 s latency, no PP	DIHARD III full	19.68	0 s collar, overlap scored	NVIDIA model card
Streaming Sortformer v2, 1.04 s latency, no PP	DIHARD III full	19.85	0 s collar, overlap scored	NVIDIA model card
Streaming Sortformer v2, 0.32 s latency, no PP	CALLHOME part 2 full	13.57	0.25 s collar, overlap scored	NVIDIA model card
EEND-EDA	CALLHOME, unknown count	15.29	per paper	Horiguchi et al., 2020
Online overlap-aware, 5 s latency	DIHARD III	25.0	no collar, overlap scored	Coria et al., 2021
Offline VBx + overlap resegmentation	DIHARD III	19.3	no collar, overlap scored	Coria et al., 2021

Methodology caveats. pyannote numbers are self-reported by the vendor on its own evaluation harness, last updated September 2025. Sortformer’s CALLHOME numbers use a 0.25 s collar and so are not comparable with pyannote’s CALLHOME figure. Sortformer is capped at four speakers and its DIHARD III “5 or more speakers” subset scores above 40% at every latency. The online-versus-offline pair from Coria et al. is the most useful row for architects: same authors, same protocol, and a clear 5.7-point price for streaming on DIHARD III, mostly in speaker confusion (9.7 versus 4.9).

The spread within one system is the real lesson. community-1 ranges from 11.2% on VoxConverse broadcast audio to 44.6% on AVA-AVD movie audio and 46.8% on Ego4D. Domain dominates model choice. Score on your own audio before you believe any row of this table.

Walk-through: Offline, Word-Level and Streaming Diarization in Python

This section builds a working stack in four steps: an offline pyannote diarization run, a join with ASR word timestamps, a DER evaluation you can trust, and a streaming configuration. All code targets Python 3.10 or later, which pyannote.audio 4.x requires.

Step 1: offline diarization with pyannote community-1

Install the toolkit with pip install pyannote.audio, install ffmpeg, accept the model’s conditions on Hugging Face, and create an access token. The pipeline is licensed CC-BY-4.0, and the model card documents an offline mode where you clone the repository with git-lfs and pass a local directory path to Pipeline.from_pretrained.

# diarize.py - offline speaker diarization with pyannote community-1
import os
import torch
from pyannote.audio import Pipeline

pipeline = Pipeline.from_pretrained(
    "pyannote/speaker-diarization-community-1",
    token=os.environ["HF_TOKEN"],
)
if torch.cuda.is_available():
    pipeline.to(torch.device("cuda"))

# Supply the speaker count when you know it; bound it when you do not.
output = pipeline("meeting.wav", min_speakers=2, max_speakers=6)

# Regular output: overlapping turns are allowed.
for turn, speaker in output.speaker_diarization:
    print(f"{turn.start:8.2f} {turn.end:8.2f} {speaker}")

# Exclusive output: at most one speaker per instant, easier to join with ASR.
with open("meeting.rttm", "w") as f:
    output.exclusive_speaker_diarization.write_rttm(f)

Two outputs come back. speaker_diarization preserves overlap and is what you score. exclusive_speaker_diarization resolves every instant to a single speaker, a mode that pyannote says was first built for its precision-2 service to simplify aligning transcription timestamps with speaker labels. Use the exclusive view for transcripts and the regular view for analytics such as overlap rate or interruption counts.

On speed, pyannote reports community-1 processing about 31 seconds per hour of AMI audio on an NVIDIA H100 80 GB, and precision-2 about 14 seconds per hour in its self-hosted version. That is a real-time factor near 0.009 for community-1 on a top-end GPU. Expect it to be much slower on a laptop CPU; measure before you size a fleet.

Step 2: joining speakers to ASR word timestamps

Diarization produces turns; ASR produces words. The join is where most production bugs live, because the two systems disagree on boundaries by tens to hundreds of milliseconds.

Figure 3: The ASR join. Words are assigned to the exclusive speaker turn they overlap most; words that fall in gaps take the nearest turn within a tolerance; the result is re-segmented into speaker-attributed utterances.

The rule that works best in practice is maximum temporal overlap, with a nearest-turn fallback for words that land in gaps between turns. The code below is self-contained and runs without any model, so you can test it against fixtures.

# join_words.py - assign speakers to ASR words by maximum overlap
from dataclasses import dataclass
from typing import Optional

@dataclass
class Turn:
    start: float
    end: float
    speaker: str

@dataclass
class Word:
    start: float
    end: float
    text: str
    speaker: Optional[str] = None

def overlap(a0, a1, b0, b1):
    return max(0.0, min(a1, b1) - max(a0, b0))

def assign_speakers(words, turns, max_gap=0.5):
    turns = sorted(turns, key=lambda t: t.start)
    for w in words:
        best, best_ov = None, 0.0
        for t in turns:
            if t.start > w.end:
                break
            ov = overlap(w.start, w.end, t.start, t.end)
            if ov > best_ov:
                best, best_ov = t, ov
        if best is None:  # word fell in a gap: take nearest turn
            mid = 0.5 * (w.start + w.end)
            near = min(turns, key=lambda t: min(abs(mid - t.start), abs(mid - t.end)))
            dist = min(abs(mid - near.start), abs(mid - near.end))
            best = near if dist <= max_gap else None
        w.speaker = best.speaker if best else "UNKNOWN"
    return words

def to_utterances(words):
    utts = []
    for w in words:
        if utts and utts[-1]["speaker"] == w.speaker and w.start - utts[-1]["end"] < 1.0:
            utts[-1]["text"] += " " + w.text
            utts[-1]["end"] = w.end
        else:
            utts.append({"speaker": w.speaker, "start": w.start, "end": w.end, "text": w.text})
    return utts

if __name__ == "__main__":
    turns = [Turn(0.0, 3.1, "spk_0"), Turn(3.3, 6.0, "spk_1"), Turn(6.0, 8.0, "spk_0")]
    words = [Word(0.2, 0.5, "Can"), Word(0.5, 0.8, "we"), Word(0.8, 1.3, "ship?"),
             Word(3.15, 3.4, "Not"), Word(3.4, 3.9, "yet."), Word(6.1, 6.6, "Why?")]
    for u in to_utterances(assign_speakers(words, turns)):
        print(f"[{u['start']:.2f}-{u['end']:.2f}] {u['speaker']}: {u['text']}")

Running it prints three utterances, with “Not” correctly assigned to spk_1 even though it starts in the gap between turns, because it overlaps the second turn more than the first. In a real pipeline, swap the fixtures for exclusive_speaker_diarization turns and word timestamps from your recogniser.

Word timestamps quality matters as much as diarization quality. Whisper’s native timestamps are segment-level and can drift; WhisperX addresses this with wav2vec2-based forced alignment to get word-level times, and then calls pyannote community-1 for speakers. Its README is candid that diarization is “far from perfect”, that word-level diarization is “harder than first thought”, and that words containing characters outside the alignment model’s dictionary, such as “£13.60”, receive no timing at all. Your join code must tolerate words with missing timestamps; the fallback above assigns them “UNKNOWN” rather than crashing.

Step 3: score DER yourself, with explicit rules

Never ship a diarization change without a DER and JER run on your own labelled audio. The pyannote.metrics package implements the standard metric; the switches below make the protocol explicit.

# score.py - DER with explicit collar and overlap rules
from pyannote.core import Annotation, Segment
from pyannote.metrics.diarization import DiarizationErrorRate

ref = Annotation()
ref[Segment(0.0, 4.0)] = "alice"
ref[Segment(3.5, 6.0)] = "bob"      # 0.5 s of overlap
ref[Segment(6.0, 10.0)] = "alice"

hyp = Annotation()
hyp[Segment(0.0, 3.8)] = "A"
hyp[Segment(3.8, 6.2)] = "B"        # misses the overlap, small boundary slip
hyp[Segment(6.2, 10.0)] = "B"       # confuses alice with bob after 6.2 s

metric = DiarizationErrorRate(collar=0.0, skip_overlap=False)
d = metric(ref, hyp, detailed=True)
for k in ("missed detection", "false alarm", "confusion", "total"):
    print(f"{k:18s} {d[k]:6.2f} s")
print(f"DER = {d['diarization error rate']:.3f}")

Change collar=0.0 to collar=0.5 (pyannote.metrics expresses the collar as the total width around each boundary, so 0.5 corresponds to the conventional plus or minus 0.25 seconds) and watch DER fall without any change to the system. That one experiment teaches more about benchmark tables than any paper.

For intuition about what the metric does internally, here is a dependency-light frame-based DER with Hungarian speaker mapping. It approximates md-eval at a 10 ms step, ignores collars, and scores overlap.

# frame_der.py - frame-level DER with optimal speaker mapping (numpy + scipy)
import numpy as np
from scipy.optimize import linear_sum_assignment

def to_matrix(turns, labels, n_frames, step=0.01):
    m = np.zeros((n_frames, len(labels)), dtype=bool)
    for s, e, spk in turns:
        m[int(round(s / step)):int(round(e / step)), labels.index(spk)] = True
    return m

def frame_der(ref_turns, hyp_turns, duration, step=0.01):
    n = int(round(duration / step))
    rl = sorted({t[2] for t in ref_turns}); hl = sorted({t[2] for t in hyp_turns})
    R = to_matrix(ref_turns, rl, n, step); H = to_matrix(hyp_turns, hl, n, step)
    # Co-occurrence: frames where ref speaker i and hyp speaker j are both active.
    co = R.T.astype(int) @ H.astype(int)
    ri, hj = linear_sum_assignment(-co)
    n_ref = R.sum(1); n_hyp = H.sum(1)
    n_correct = np.zeros(n, dtype=int)
    for i, j in zip(ri, hj):
        n_correct += (R[:, i] & H[:, j])
    total = n_ref.sum()
    miss = np.clip(n_ref - n_hyp, 0, None).sum()
    fa = np.clip(n_hyp - n_ref, 0, None).sum()
    conf = (np.minimum(n_ref, n_hyp) - n_correct).sum()
    return {"miss": miss * step, "fa": fa * step, "conf": conf * step,
            "der": (miss + fa + conf) / total}

ref = [(0.0, 4.0, "alice"), (3.5, 6.0, "bob"), (6.0, 10.0, "alice")]
hyp = [(0.0, 3.8, "A"), (3.8, 6.2, "B"), (6.2, 10.0, "B")]
print(frame_der(ref, hyp, duration=10.0))

The formula per frame is the md-eval one: missed equals reference count minus system count when positive, false alarm is the reverse, and confusion is the smaller of the two counts minus correctly mapped speakers. It prints 0.5 s missed, 0 s false alarm and 4.0 s confusion over 10.5 s of reference speaker time, a DER of about 42.9%. In this toy case almost all the error is confusion, because the system glued alice’s final turn onto bob. That is exactly the error a summary model will turn into a false statement.

Step 4: streaming diarization with a speaker cache

Offline pipelines see the whole recording before clustering. A live captioning system, a voice agent or a call-centre assist tool cannot wait. Streaming forces the global identity decision to be made incrementally, with bounded memory and a fixed latency budget, and that is where the architectures differ most.

Figure 2: Streaming speaker diarization. Each new chunk is processed with left context from a FIFO and the speaker cache; local speakers are matched to persistent identities, labels are emitted after the right-context delay, and the cache is periodically compressed.

There are two mature open designs. The first, from Coria, Bredin, Ghannay and Rosset (2021) and packaged as the diart library, runs pyannote-style segmentation on a 5-second rolling buffer updated every 500 ms. Local speakers are matched to global centroids with a constrained Hungarian assignment that forbids two local speakers in the same buffer from merging into one global speaker. Latency is tunable from 500 ms to 5 s without retraining, because more latency simply means averaging predictions from more buffer positions. Three hyperparameters govern it: an activity threshold (tau), a centroid update threshold (rho) and a new-speaker distance threshold (delta), which the authors tune per dataset.

The second is NVIDIA’s Streaming Sortformer (Medennikov et al., Interspeech 2025, arXiv 2507.18446). It keeps an end-to-end model and adds an Arrival-Order Speaker Cache (AOSC) that stores frame-level embeddings of previously seen speakers, ordered by when each first spoke. Each new chunk is processed together with the cache and a FIFO of recent frames, so the model “remembers” voices without clustering. The published diar_streaming_sortformer_4spk-v2 checkpoint has about 117 million parameters, a 17-layer FastConformer-based NEST encoder followed by an 18-layer Transformer, an 80 ms output frame, a four-speaker maximum, and a CC-BY-4.0 license.

# stream_sortformer.py - low-latency configuration from the NVIDIA model card
from nemo.collections.asr.models import SortformerEncLabelModel

model = SortformerEncLabelModel.from_pretrained("nvidia/diar_streaming_sortformer_4spk-v2")
model.eval()

# "Low latency" preset: 1.04 s input buffer latency (chunk 6 + right context 7 frames of 80 ms)
m = model.sortformer_modules
m.chunk_len = 6
m.chunk_right_context = 7
m.fifo_len = 188
m.spkcache_update_period = 144
# speaker cache size stays at the card's default of 188 frames

segments = model.diarize(audio=["call.wav"], batch_size=1)
for seg in segments[0]:
    print(seg)

The model card’s latency arithmetic is worth understanding. Input buffer latency is chunk size plus right context, in 80 ms frames: (6 + 7) x 0.08 = 1.04 s. The ultra-low preset uses 3 + 1 frames for 0.32 s. Each preset trades latency against compute: NVIDIA reports a real-time factor of 0.093 at 1.04 s and 0.180 at 0.32 s on an RTX 6000 Ada at batch size 1, against 0.002 at its 30.4 s preset. Smaller chunks mean more forward passes per second of audio, each carrying the full cache.

What does streaming cost in accuracy? On DIHARD III, Sortformer’s published DER without post-processing moves only from 19.68% at 30.4 s latency to 20.19% at 0.32 s. On CALLHOME part 2 with a 0.25 s collar it moves from 10.50% to 13.57%, and the five-speaker subset jumps from 19.08% to 29.40%. The cache approach degrades gracefully for few speakers and sharply as the speaker count approaches the four-slot limit. The model card also notes a newer variant, Nemotron-3-Diarization, that supports eight speakers; check its own card before relying on that.

The cluster-based online approach shows the opposite profile: unlimited speaker count, but a larger accuracy penalty, 25.0% online versus 19.3% offline on DIHARD III in Coria et al., with confusion roughly doubling. Pick by speaker count first. Two to four speakers with tight latency favours an end-to-end cache model; open-ended meetings favour cluster-based streaming with a nightly offline re-diarization to correct the record.

Capacity and cost, from first principles

Offline clustering scales worse than people expect. Take a one-hour recording, 10-second segmentation windows and, as an illustrative assumption, a 1-second step: about 3,600 windows, each with up to three local speakers, so up to roughly 10,800 embeddings. A full pairwise distance matrix in float64 is 10,800 squared times 8 bytes, about 0.93 GB, or half that in condensed form. Tolerable for one file; painful for a batch of 50 concurrent four-hour depositions. Real pipelines discard empty local speakers and some subsample, so measure your actual embedding count, but the quadratic term is real.

Embedding extraction usually dominates GPU time, segmentation is cheaper, and clustering is CPU-bound. That split suggests a deployment pattern: batch segmentation and embeddings on a GPU pool, then run clustering on CPU workers so a long file does not hold a GPU hostage while AHC or VBx iterates.

For streaming, cost is per concurrent stream, not per hour of audio. At NVIDIA’s reported RTF of 0.093 for the 1.04 s preset on one RTX 6000 Ada at batch size 1, a single stream uses about 9% of the GPU’s time, which suggests on the order of ten concurrent streams per GPU before batching. Treat that as an upper-bound sketch: it ignores ASR sharing the same GPU, network jitter and the gains from batching multiple streams.

Trade-offs, Gotchas, and What Goes Wrong

Most diarization incidents are not model failures. They are mismatches between the audio, the configuration and the assumptions baked into a pipeline. The decision flow below routes the common situations to an architecture, and the failure modes after it are the ones we see repeatedly.

Figure 4: A decision flow for speaker diarization architecture. Separate channels beat any model; a known small speaker count favours end-to-end models; open-ended meetings favour hybrid clustering, with offline re-diarization to correct streaming output.

Use the channels you already have. If a call recorder captures agent and customer on separate stereo channels, you do not need diarization for those two parties at all; voice activity detection per channel gives a near-perfect answer. Running a diarization model on a down-mixed stereo file throws that information away and replaces certainty with a 10-20% error rate. The same holds for video conferencing platforms that expose per-participant audio tracks.

Speaker count is the silent killer. When count is unknown, clustering may split one speaker into two after they move away from the microphone or switch from calm to agitated, or merge two similar voices, such as same-gender colleagues on a narrowband phone line. Both show up as confusion. Supplying num_speakers or tight bounds fixes most of it. If the number is truly unknown, at least set max_speakers from the meeting invite.

Short turns and backchannels get lost. A “yes” of 300 ms yields too few clean frames for a reliable embedding. Segmentation detects it, but clustering assigns it to whoever spoke more nearby. In compliance use cases, where a single “I agree” carries legal weight, route short turns for human review rather than trusting the label.

Domain shift dominates. community-1’s own numbers range from 11.2% DER on VoxConverse to 44.6% on AVA-AVD. Far-field microphones, reverberant rooms, music beds, children’s voices and code-switching each cost accuracy. Fine-tuning segmentation on even a few hours of in-domain labelled audio is often worth more than switching models.

End-to-end models have hard ceilings. The streaming Sortformer checkpoint supports four speakers; its DIHARD III five-plus-speaker subset sits above 40% DER at every latency. Never deploy a four-slot model into a town-hall meeting.

Streaming labels change, or do not. A streaming system must either emit labels and never revise them, accepting early mistakes, or revise them and force downstream consumers to handle edits. Pick one deliberately. Many teams emit streaming labels for live captions, then re-diarize offline and overwrite the archived transcript.

Label permutation across files. Diarization labels are anonymous and per-file. “SPEAKER_00” in Monday’s stand-up is not the same person as “SPEAKER_00” on Tuesday. Linking labels to named people is speaker identification, a separate step that compares cluster centroids against enrolled voiceprints. Voiceprints are biometric data in many jurisdictions, including under the GDPR, so that step needs a lawful basis, consent handling and retention rules that diarization alone does not.

Anti-patterns to avoid. Comparing your DER with a paper without matching collar and overlap rules. Tuning clustering thresholds on the test set. Joining words to regular, overlapping turns and then wondering why one word appears twice. Ignoring words that the aligner left without timestamps.

Practical Recommendations

Start by deciding whether you need diarization at all. If you control capture, record per-speaker channels; it is cheaper and more accurate than any model. When you cannot, treat diarization as its own service with its own evaluation set, not as a checkbox on your ASR vendor.

For batch transcription of meetings, interviews and podcasts, the pyannote community-1 pipeline is a strong open default: CC-BY-4.0 weights, offline operation, overlap-aware output and an exclusive mode built for ASR alignment. If its accuracy on your audio is not enough, pyannote’s hosted precision-2 reports materially lower DER across its benchmark suite, and switching is a one-line change. For low-latency calls with two to four speakers, evaluate Streaming Sortformer at the 1.04 s preset. For open-ended live meetings, use cluster-based streaming, then re-diarize offline.

Our guide to voice AI agent architecture, latency budgets and turn-taking explains why a voice agent usually needs only “is the user speaking”, not full diarization, which saves you an entire model in the hot path.

A checklist for a production rollout:

Build a labelled set of 2-5 hours from your real audio, with overlap annotated.
Score DER and JER with dscore or pyannote.metrics, collar 0, overlap included; report the three DER components separately.
Pass num_speakers or tight min_speakers/max_speakers whenever metadata allows.
Join words to exclusive turns by maximum overlap, with a gap fallback and an “UNKNOWN” bucket.
Track confusion separately as your headline quality metric; it is what users experience.
Re-diarize streaming output offline before archiving.
Keep speaker identification, and its voiceprints, behind a separate consent and retention policy.

Frequently Asked Questions

What is speaker diarization and how is it different from speaker recognition?

Speaker diarization answers “who spoke when” in a single recording, producing anonymous labels such as speaker_0 and speaker_1 with start and end times. It does not know anyone’s name and its labels are not consistent across files. Speaker recognition, or identification, compares a voice with enrolled voiceprints to name a known person. Many products chain the two: diarize first, then compare each cluster’s average embedding with enrolled speakers. The chaining matters for privacy, because enrolment creates biometric templates that diarization alone never stores.

What is a good diarization error rate?

It depends entirely on domain and scoring protocol. Published open-model numbers with no collar and overlap scored run from roughly 11% on broadcast-style VoxConverse audio to above 40% on movie and egocentric video audio. Clean two-party telephone audio scored with a 0.25-second collar can reach single digits. Rather than chasing a universal number, set a target on your own labelled audio and pay most attention to the confusion component, because missed speech and false alarms are often invisible in a transcript, while confusion puts words in the wrong mouth.

Does pyannote diarization run offline and on CPU?

Yes. The community-1 model card documents offline use: clone the model repository with git-lfs and load it from a local directory, which works without an internet connection. You still need to accept the model conditions once to download it. The pipeline runs on CPU, but expect it to be far slower than pyannote’s reported 31 seconds per hour of audio, which was measured on an NVIDIA H100. For bulk workloads, put segmentation and embedding on a GPU and keep clustering on CPU workers.

How does end-to-end neural diarization handle overlapping speech?

End-to-end neural diarization outputs an independent activity probability for every speaker in every frame, so two or more speakers can be active at the same time by construction. The model is trained with a permutation-invariant or sorted loss, so it never needs to know which speaker is “first” in an absolute sense. EEND-EDA adds attractors so the number of speakers is not fixed in advance. The trade-off is that long recordings and many speakers strain these models, which is why hybrid pipelines use an EEND-style model only locally.

How do I combine diarization with Whisper or another ASR model?

Run ASR to obtain word-level timestamps, ideally with forced alignment, run diarization separately, then assign each word to the speaker turn it overlaps most. Use exclusive, non-overlapping turns for this join so each word gets exactly one speaker. Add a fallback for words that fall in gaps, and handle words that have no timestamps at all, such as numbers the aligner cannot map. WhisperX packages this pattern using wav2vec2 alignment and pyannote, but the join logic is short enough to own and test yourself.

Can speaker diarization run in real time?

Yes, with an accuracy and speaker-count trade-off. Cluster-based online systems such as diart operate at 500 ms to 5 s latency with no speaker limit, but published results show DER several points worse than offline. NVIDIA’s Streaming Sortformer reports DIHARD III DER near 20% even at 0.32 s latency, but supports at most four speakers in its v2 checkpoint. A common production pattern is to stream labels for live captions and then re-diarize the full recording offline for the archived record.

Speaker Diarization Architecture: Who Spoke When with pyannote, End-to-End Neural Diarization and Streaming Pipelines (2026)