Pretraining Data Curation: Dedup, Filters and Mixtures

Two teams can train the same architecture, with the same optimizer, on the same number of tokens, and end up with models that differ by ten points on MMLU. The only thing that changed is the data. That single fact has reorganised how frontier labs spend engineering effort: the model code is largely commoditised, while the pipeline that turns petabytes of crawled HTML into a few trillion clean tokens is where the real differentiation lives.

Pretraining data curation is the discipline of extracting, filtering, deduplicating, scoring, decontaminating and mixing raw text so that every training token earns its compute. It matters more each year because high-quality web text is finite, compute is not, and repeated or low-value tokens are a silent tax on every run.

This post walks the whole pipeline end to end, with the mechanisms and the arithmetic. You will leave knowing how MinHash LSH decides what counts as a duplicate (including the band math), how classifier-based filters like those behind FineWeb-Edu and DCLM work, how decontamination and data mixtures are handled, and where each stage fails.

What this covers: pipeline stages, exact and fuzzy dedup with worked probabilities, quality classifiers, decontamination, mixtures and curriculum, scale economics, open datasets, and failure modes.

Context and Background

Early large language models were trained on whatever was easy to get. GPT-3 filtered Common Crawl with a classifier against a curated reference set and fuzzy-deduplicated documents with MinHash, as described in its paper. Gopher’s MassiveText (Rae et al., 2021) published a more explicit recipe: quality heuristics on word counts and symbol ratios, repetition filters, and document-level dedup. CCNet (Wenzek et al., 2019) showed how to process Common Crawl at scale with line-level dedup, fastText language ID and a perplexity filter against a Wikipedia language model.

The next shift was the realisation that duplicates are not just wasteful but harmful. Lee et al. (2021), in “Deduplicating Training Data Makes Language Models Better”, found that common datasets contain many near-duplicates and long repeated substrings; over 1% of unprompted model output was copied verbatim from training data, and a single 61-word English sentence appeared more than 60,000 times in C4. Models trained on deduplicated data were about ten times less likely to emit memorised text, and the same paper found train-test overlap affecting more than 4% of standard validation sets.

Then the open community made the pipeline itself a public artifact. Hugging Face released FineWeb (Penedo et al., 2024), a 15-trillion-token dataset from 96 Common Crawl snapshots, along with FineWeb-Edu, a 1.3-trillion-token subset filtered for educational content, plus the datatrove library and the ablation models. AI2’s Dolma (Soldaini et al., 2024) released roughly three trillion English tokens across web, papers, code, books and encyclopedic sources, with its curation toolkit. Together AI’s RedPajama-V2 shipped over 100 trillion raw tokens with precomputed quality signals so users could filter themselves. The DataComp-LM (DCLM) project (Li et al., 2024) went furthest by turning curation into a benchmark: a standardised pool of 240 trillion tokens from Common Crawl, fixed training recipes, and 53 downstream evaluations, so that data-curation methods can be compared like model architectures.

If you are working on the synthetic side of the data problem, the companion post on synthetic data, distillation, quality filtering and model collapse covers what happens when the web runs short. The architecture side, where sparse models change the tokens-per-parameter calculus, is in the mixture-of-experts architecture guide.

The thesis of this post is simple: curation is a sequence of lossy, biased compressions, each with a tunable threshold, and the only honest way to set those thresholds is to train small models and measure. Every stage below has a principled mechanism and an empirical caveat.

The Reference Pipeline: From WARC to Training Shards

A production pretraining pipeline is a directed sequence of filters, each of which removes some fraction of documents or tokens. The ordering is not arbitrary: cheap, high-rejection stages go first, expensive model-based stages go last.

Figure 1: A reference pretraining data curation pipeline. Dashed arrows show the ablation loop that tunes the filters.

The diagram shows ten stages from raw Common Crawl WARC files to tokenised shards. The dashed feedback edge is the important one: filter thresholds are not set by intuition but by training small proxy models on each variant and comparing benchmark scores.

Extraction: the stage everyone underestimates

Common Crawl distributes raw HTTP responses in WARC files plus pre-extracted WET text files. The FineWeb authors found that extracting text directly from WARC HTML with trafilatura produced better downstream models than using the provided WET files, which retain more boilerplate such as navigation menus and cookie banners. Extraction decides what a “document” is, so errors here propagate to every later stage: a footer repeated on a million pages looks like a duplicate paragraph, and a menu fragment can pass a length filter.

The practical lesson is to evaluate extractors on downstream training, not on visual inspection. An extractor that keeps comment sections raises token yield but also raises noise; one that keeps only the main article raises quality but discards forum answers that matter for conversational ability.

Language identification and URL filtering

Language ID, usually with a fastText classifier, is a cheap filter with a threshold trade-off. A high confidence threshold gives clean monolingual text but removes code-switched pages and low-resource-language documents where the classifier is uncertain. Many English-focused pipelines accept scores above roughly 0.65 to 0.9; the right value depends on whether you want to keep multilingual leakage. Treat any specific cutoff as a tunable choice, not a standard.

URL filtering removes domains on blocklists (adult content, known spam) and is nearly free. Its limit is that blocklists are sparse and stale, so it is a first pass rather than a safety guarantee.

Heuristic quality filters

Heuristic filters encode human intuition about what text looks like: minimum and maximum word counts, mean word length, fraction of lines ending in punctuation, fraction of lines that are duplicates within the document, symbol-to-word ratio, fraction of alphabetic characters, and stop-word presence. Gopher’s rules and C4’s rules (keep lines ending in terminal punctuation, drop lines containing “javascript”, drop pages with placeholder text) are the ancestors.

FineWeb’s contribution was to derive extra filters empirically: the team computed many candidate statistics on a good dataset and a bad one, then chose thresholds where the distributions diverged, validating each with an ablation. Heuristic filters are fast and interpretable but blunt. Their characteristic failure is false rejection of unusual-but-valuable text such as tables, code snippets in prose, poetry, and non-standard punctuation.

Why order matters economically

Suppose a pipeline processes 100 units of raw text. If heuristic filters remove 50% at a cost of 1 unit per document, MinHash dedup then runs on 50 units rather than 100, and a model-based scorer costing 100 times a heuristic runs on perhaps 30 units. Reversing the order would multiply the expensive stages’ cost by two to three. These are illustrative figures, but the logic is general: put the cheapest high-rejection filter first and the most expensive filter last.

Deduplication: Exact, Fuzzy and MinHash LSH

Deduplication has three layers, and the pipeline usually uses all of them. The layers trade precision for cost: exact matching is cheap and certain, fuzzy matching is costlier and probabilistic.

Direct answer: Fuzzy deduplication finds near-duplicate documents by turning each into a set of n-gram shingles, compressing the set into a short MinHash signature whose collision rate equals Jaccard similarity, and using locality-sensitive hashing (LSH) bands to find candidate pairs without comparing every pair. Tuning the band count b and rows per band r sets the similarity threshold at which documents are treated as duplicates.

Exact and substring deduplication

Exact dedup hashes the normalised document (or each line, or each paragraph) and drops repeats. It costs one hash and one set lookup per item and has no false positives beyond hash collisions. It misses documents that differ by a timestamp or a “Posted by” line, which is why it only handles the easy cases.

Lee et al. also proposed suffix-array-based exact substring dedup: build a suffix array over the concatenated corpus and remove any substring longer than a threshold (they used 50 tokens) that occurs more than once. This catches repeated boilerplate paragraphs inside otherwise unique documents. It needs substantial memory and disk, which makes it a specialist pass rather than a default.

MinHash: the estimator

The Jaccard similarity of two sets A and B is |A∩B| / |A∪B|. Broder (1997) observed that if you apply a random hash function h to every element of both sets and take the minimum hash value of each set, the probability that the two minima are equal is exactly the Jaccard similarity. One hash function gives a one-bit estimate; k independent hash functions give a signature of k minimum values, and the fraction of matching positions estimates Jaccard with standard error roughly sqrt(s(1-s)/k).

For a pair with true Jaccard 0.8 and k = 112 hashes, that standard error is about sqrt(0.8 x 0.2 / 112) = 0.038. The estimate is good enough to rank pairs, and LSH takes care of the rest.

Documents become sets through shingling. FineWeb used word 5-grams: the sentence “the quick brown fox jumps over the dog” yields shingles such as “the quick brown fox jumps” and “quick brown fox jumps over”. Shorter shingles make unrelated documents look more similar (common phrases collide), while longer shingles make the signature too sensitive to small edits. The choice of 5 words is a reasonable middle, not a law.

LSH banding and the S-curve

Comparing all pairs is quadratic: a billion documents means about 5 x 10^17 pairs. LSH avoids this by splitting each signature of b x r values into b bands of r rows. Each band is hashed into a bucket, and two documents become a candidate pair if they land in the same bucket in at least one band.

If two documents have Jaccard similarity s, the probability that all r rows in a given band match is s^r. The probability that at least one of b bands matches is

P(candidate) = 1 – (1 – s^r)^b

This is an S-shaped curve, and the approximate similarity at which it rises steeply is (1/b)^(1/r). Choosing b and r moves that threshold and sharpens or softens the curve.

Figure 2: MinHash deduplication flow. Shingles feed 112 hash functions, the signature is cut into 14 bands of 8 rows, and shared buckets create candidate pairs that are merged by union-find.

The figure shows the full path from document to cluster. The signature has 112 values because FineWeb used 14 bands of 8 rows (14 x 8 = 112), a configuration I am taking from the FineWeb paper’s description; check the paper or the datatrove source for the exact current defaults before reproducing them.

Worked numbers for the band math

Using the formula, here are exact computed probabilities that a pair becomes a candidate, for three configurations. The first column is the FineWeb-style setting.

Jaccard s	b=14, r=8	b=9, r=13	b=20, r=5
0.50	5.3%	0.1%	47.0%
0.60	21.1%	1.2%	80.2%
0.70	56.5%	8.4%	97.5%
0.75	77.2%	19.5%	99.6%
0.80	92.4%	39.9%	99.96%
0.90	99.96%	92.9%	100%

The (14, 8) setting has a threshold near 0.72: pairs below 0.5 almost never collide, pairs above 0.8 almost always do, and the transition region is moderately wide. The (20, 5) setting has a threshold near 0.55, so it flags loosely related documents and will remove far more data. The (9, 13) setting is conservative: it only removes pairs that are nearly identical, leaving borderline near-duplicates in place.

The width of the transition is a feature you must accept. At s = 0.7, the (14, 8) configuration still misses 43.5% of true pairs. If you need high recall at a lower similarity, raise b and lower r, at the cost of more candidate pairs and therefore more false positives that you must verify or accept.

From candidates to clusters

Candidate pairs form a graph. Connected components, computed with union-find, give duplicate clusters, and the pipeline keeps one document from each. This has a subtle consequence: similarity is not transitive, so a chain A~B~C may group A and C even if their direct Jaccard is low. Large chains can swallow legitimately different documents, especially templated pages such as product listings or legal boilerplate.

Runnable example

This snippet uses the datasketch library to build signatures and an LSH index with an explicit threshold. Parameters are illustrative; production systems run this distributed.

from datasketch import MinHash, MinHashLSH

def shingles(text, n=5):
    w = text.lower().split()
    return {" ".join(w[i:i+n]) for i in range(len(w) - n + 1)}

def signature(text, num_perm=112):
    m = MinHash(num_perm=num_perm)
    for s in shingles(text):
        m.update(s.encode("utf8"))
    return m

docs = {
    "a": "the quick brown fox jumps over the lazy dog near the river bank today",
    "b": "the quick brown fox jumps over the lazy dog near the river bank tonight",
    "c": "completely different text about data curation pipelines and quality filters here",
}
# explicit band layout: 14 bands of 8 rows = 112 permutations
lsh = MinHashLSH(num_perm=112, params=(14, 8))
sigs = {k: signature(v) for k, v in docs.items()}
for k, m in sigs.items():
    for hit in lsh.query(m):
        print("candidate pair:", k, hit,
              "est. jaccard:", round(m.jaccard(sigs[hit]), 3))
    lsh.insert(k, m)

Document b differs from a by one word, which disturbs five of the fourteen shingles, so their true Jaccard is high and they collide. Document c shares nothing and does not.

Dedup scope: global versus per-snapshot

The most counterintuitive finding in the FineWeb work is that deduplicating more is not automatically better. The authors compared deduplicating each Common Crawl snapshot independently with deduplicating globally across all snapshots, and reported that global dedup did not improve downstream models as expected. Their analysis suggested that global dedup preferentially removed data that had survived in old snapshots because it was of higher quality (content that persists across crawls tends to be useful), while leaving behind more low-quality data unique to each crawl. I describe this as the paper’s reported finding; treat the mechanism as a hypothesis, not a theorem.

Meanwhile, repeated data is not strictly bad. Muennighoff et al. (2023), “Scaling Data-Constrained Language Models”, found that repeating data for up to about four epochs gave losses close to unique data, with returns diminishing sharply beyond that. The implication is that aggressive dedup that leaves you short of tokens may be worse than keeping a controlled amount of repetition of high-quality text.

Cost at scale

MinHash dedup is embarrassingly parallel for signature computation (each document independently) and then needs a shuffle by band hash. For 14 bands, each document emits 14 (band hash, document id) records, so a billion documents yield 14 billion records to group. That is large but tractable on a cluster, and it is the reason MinHash remains the default over exact pairwise methods.

Quality Filtering: From Heuristics to Learned Classifiers

Heuristics tell you whether text looks like prose. They cannot tell you whether it is informative. The second generation of filters therefore trains a model to predict “quality”, then keeps the top-scoring slice. This is where the largest recent gains in open datasets came from, and it is the most opinionated stage in the pipeline: whoever chooses the positive examples defines quality.

The classifier recipe

The recipe has four parts, and the diagram below shows how they connect.

Figure 3: Classifier-based LLM data quality filtering. A fastText model is trained on seed positives against random web negatives, scores every document, and the threshold is tuned by training proxy models.

The loop closes by training a small proxy model on the filtered data and checking benchmark scores. If scores do not improve, the seed set or threshold changes. Nothing in this loop is theoretical; it is a search over data recipes with a model as the loss function.

DCLM ran this search systematically. Its published finding is that model-based filtering is the key ingredient for a high-quality training set, and its best configuration used a fastText classifier trained with instruction-style data as positives (the reported choice was OpenHermes 2.5 together with ELI5 posts) against random web pages as negatives, keeping the top-scoring fraction of the pool, which the paper reports as roughly the top 10%. I am recalling the positive-set and fraction details from the DCLM paper and its repository; confirm them there before citing. The headline result is verified: a 7B model trained on DCLM-Baseline for 2.6 trillion tokens reached 64% 5-shot MMLU, 6.6 points above MAP-Neo (the previous open-data state of the art, trained with about 40% more compute), and comparable to Mistral-7B-v0.3 at 63% and Llama 3 8B at 66%, with the authors reporting 6.6 times less compute than Llama 3 8B.

FineWeb-Edu: LLM-annotated educational value

FineWeb-Edu uses a different idea. The team prompted Llama-3-70B-Instruct to score a sample of FineWeb documents for educational value on a small integer scale, then trained a lightweight classifier (an embedding model with a regression head) on those annotations, and applied it to the whole corpus with a threshold. As I recall, the 1.3-trillion-token release corresponds to a threshold of 3 on the 0-5 scale, while a larger variant at a lower threshold also exists; confirm the exact variants and sizes on the dataset card (unverified here). The verified headline from the paper is the effect: models trained on FineWeb-Edu perform dramatically better on knowledge and reasoning benchmarks such as MMLU and ARC.

This pattern, an expensive LLM labelling a small sample and a cheap model propagating labels to billions of documents, is the cost-structure trick behind most modern quality filtering. Annotating 500,000 documents with a 70B model is a modest GPU bill; scoring 15 trillion tokens with the same model would be prohibitive.

Why a classifier beats heuristics, and where it fails

A classifier captures correlations no human wrote down: explanatory structure, vocabulary diversity, topic coherence. In effect it learns a density ratio between your target distribution and the web.

That is also its weakness. The filter selects what resembles the positive set, so it encodes the positive set’s biases. Educational-value filters favour textbook register and Western academic topics; instruction-style positives favour Q&A phrasing. Models trained on heavily filtered data can gain benchmark points (which are themselves educational in style) while losing coverage of dialects, fiction, informal speech and niche domains. This is a form of benchmark-shaped optimisation: if you tune the filter by watching MMLU, the filter learns to prefer MMLU-like text.

There is also a ceiling effect. Aggressive filtering keeps a small fraction of the corpus, and Nemotron-CC (NVIDIA, 2024) argued that this throws away too many tokens for long training runs, proposing ensembles of classifiers and synthetic rephrasing of low-quality text to recover volume. I mention the argument as the authors’ claim; the quantitative tradeoff depends on your token budget.

Perplexity filters and their quirks

A cheaper relative is the perplexity filter, introduced for CCNet: train a small language model on Wikipedia, score each document by perplexity, and bucket into head, middle and tail. It selects text that resembles the reference model’s training data, so it carries the same bias issue and additionally penalises technical text with unusual vocabulary. It remains useful as a fast first-stage signal.

Quality signals as metadata rather than hard filters

RedPajama-V2 takes a different philosophy: compute dozens of quality signals per document and release them as metadata, deferring the thresholds to the user. This lets downstream teams run their own ablations without re-processing the raw crawl, and it makes the cost of experimentation one query rather than one pipeline run. The downside is storage and the temptation to treat the signals as validated when they are only features.

PII, toxicity and licensing

Safety-oriented filtering includes removing personally identifiable information (emails, IP addresses, phone numbers) with regex or models, and optionally filtering toxic content with classifiers. Dolma’s documentation describes tagging approaches for these that mark spans rather than deleting whole documents. Toxicity filtering has a known side effect: classifiers often over-flag text that mentions marginalised identities, which can reduce a model’s ability to discuss those groups. Licensing and opt-out signals (robots.txt, ai.txt, publisher opt-outs) are a growing compliance layer on top, and the legal landscape differs by jurisdiction. This post makes no legal claims.

Decontamination: Keeping the Benchmarks Out

If a benchmark question appears verbatim in the training set, the benchmark stops measuring generalisation. The web contains benchmark questions, solutions, and discussion threads about them, so contamination is the default state, not an exception.

The standard method is n-gram overlap: build a set of n-grams (commonly 8 to 13 tokens or words) from every evaluation set and remove or flag training documents that contain any of them. GPT-3 used a 13-gram overlap check and reported that contamination was mostly small in effect but nonzero. Llama-family reports describe similar overlap-based analyses.

Three limits matter. First, n-gram matching misses paraphrases, translations and reformatted problems; it only catches near-verbatim leaks. Second, aggressive decontamination of short benchmarks (multiple-choice questions with short stems) risks removing legitimate text that happens to share a phrase. Third, removal after the fact does not fix the benchmark if the test set is widely discussed online in ways that leak answers.

Practical teams do three things. They decontaminate against the full list of evaluation sets they will report, before the data is frozen. They keep a held-out private evaluation set that never touched the web. And they report both original and decontaminated scores, because the gap measures how much of the score was memory. Lee et al.’s finding that more than 4% of validation examples had train-test overlap in standard datasets shows why this is not a formality.

Data Mixtures and Curriculum

Once you have clean sources (web, code, books, papers, math, multilingual, encyclopedic), you must decide how often each is sampled. That mixture is a hyperparameter with large effects and no closed-form solution.

Why the mixture is hard

Domain loss curves interact. More code tends to help reasoning but hurts few-shot natural-language tasks if overdone; more math helps GSM8K-style benchmarks but shifts the tokenizer budget. The relationship between mixture weights and final quality is nonlinear, and a full-scale test costs a full training run.

Figure 4: Data mixture optimisation. Proxy models are trained across candidate mixtures, a regression predicts loss, the best mixture is searched, and training then proceeds in stages that raise the quality share over time.

The figure splits the problem into a search phase on small models and a scheduling phase on the full model. These are different decisions: the first chooses proportions, the second chooses when to apply them.

Proxy-model methods

DoReMi (Xie et al., 2023) trains a small proxy model (280M parameters) with Group Distributionally Robust Optimization to learn domain weights, then resamples the dataset with those weights to train an 8B model. On The Pile, it reported a 6.5 point average few-shot accuracy gain over default weights, reaching the baseline’s accuracy with 2.6 times fewer training steps. RegMix (Liu et al., 2024) takes another route: train many tiny models on random mixtures, fit a regression from mixture weights to validation loss, and predict the best mixture for a large run. Both depend on the assumption that the best mixture for a small model approximates the best for a large one, which holds approximately but not exactly.

Curriculum and annealing

The second lever is time. Many recent recipes train mostly on broad web data and then shift toward higher-quality, more reasoning-dense data (code, math, curated instructions) during the final portion of training, often combined with a learning rate decay. Public reports for several open models describe this “annealing” or “mid-training” phase as responsible for sizeable benchmark gains for a small fraction of total tokens. The mechanism is plausible: late in training, the learning rate is low, so the model consolidates what high-quality data teaches without being destabilised.

Be careful with attributing the gain. If you anneal on data that resembles your evaluation, you improve scores in part because you moved the training distribution toward the benchmark. That is legitimate if the benchmark reflects target use, and misleading if it does not.

Epochs versus unique tokens

The data-constrained scaling results discussed above change how you set mixture weights when high-quality sources are small. A 50-billion-token code corpus sampled four times contributes 200 billion tokens at nearly the value of unique data, so a mixture that upsamples scarce high-quality sources a few times is rational; upsampling twenty times is memorisation. The optimum depends on your total token count, which is why recipes do not transfer across scales unmodified.

Scale Economics: What Curation Costs and Saves

Curation looks like overhead, but it changes the compute equation. A 7B-parameter model trained on 2.6 trillion tokens uses roughly 6 x 7e9 x 2.6e12 = 1.1e23 floating-point operations by the standard 6ND approximation. If better data lets you reach target quality with 40% fewer tokens, you save about 4.4e22 FLOPs. That is far more than the pipeline’s CPU cost for filtering.

The pipeline’s own cost has a shape worth knowing. Text extraction and heuristic filters are CPU-bound and scale linearly. MinHash signature computation scales linearly in documents times shingles times hashes: 112 hashes over a 1,000-shingle document is about 112,000 hash operations. LSH grouping is a shuffle. Model-based scoring is GPU or CPU inference and dominates if you use an LLM scorer on every document, which is why practitioners distil the scorer into a small classifier.

A rule of thumb that survives most pipelines: the survival rate from raw crawl to final training tokens is a few percent to low tens of percent, depending on how aggressive the quality threshold is. FineWeb’s 15 trillion tokens come from 96 snapshots of a crawl whose raw text is vastly larger, and DCLM’s 240 trillion token pool shrinks to a baseline set that trains a 2.6 trillion token run. Those two figures bracket the compression you should expect, though I do not claim an exact ratio because the stated sizes refer to different stages.

The ablation budget

The hidden cost is experimentation. Each filter threshold needs a proxy model run. FineWeb’s ablations used 1.8B-parameter models trained on around 30 billion tokens, as I recall from the paper, and DCLM’s testbed spans 412M to 7B parameters. If a pipeline has ten tunable stages with five settings each, one-at-a-time ablation costs 50 runs, and interactions make that an underestimate. Budget this explicitly; teams that skip it end up with thresholds set by anecdote.

Open Datasets Compared

Dataset	Scale	What it demonstrates	Notable property
FineWeb	15T tokens, 96 snapshots	Documented ablations of dedup and filters	Open code (`datatrove`), ablation models
FineWeb-Edu	1.3T tokens (headline subset)	LLM-annotation-distilled classifier	Large knowledge-benchmark gains
Dolma	about 3T English tokens	Multi-source mix (web, papers, code, books)	Open curation toolkit
RedPajama-V2	over 100T raw tokens (as reported for the family)	Quality signals as metadata	Users pick thresholds
DCLM-Baseline	240T pool, curated subset	Curation as a controlled benchmark	fastText model filter

A table like this can mislead because the numbers measure different things (raw, filtered, or released). Use it to choose a starting point, then read each dataset card.

Trade-offs, Gotchas, and What Goes Wrong

Over-filtering narrows the model. Classifier thresholds that maximise MMLU can remove dialogue, fiction, forum speech and non-Western content. Evaluate on a wide panel, including perplexity on held-out web text you deliberately did not filter.

Dedup can remove good data. The FineWeb global-dedup result is the cautionary tale: removing duplicates across snapshots shifted the corpus toward lower-quality, crawl-unique pages. Dedup granularity (document, paragraph, line) should be an ablation, not a default.

Clusters swallow templates. Union-find over a similarity graph can chain unrelated templated pages together. Cap cluster size or inspect the largest clusters by hand, always.

Hash and threshold drift. Changing the shingle size or hash count silently changes the effective similarity threshold. Pin parameters in config and log the S-curve you expect.

Proxy models mislead. Small-model ablations do not always rank data correctly at 70B scale, particularly for capabilities that emerge late such as multi-step reasoning. Confirm key decisions with at least one larger run.

Benchmark overfitting in the loop. Tuning on a benchmark and reporting it makes the benchmark a training signal. Hold out evaluations that the curation loop never saw.

Contamination re-enters. Newly crawled snapshots contain fresh discussion of benchmarks, so decontamination is a recurring job, not a one-off.

Synthetic and AI-generated web text. The web is increasingly populated by model output, which is statistically smoother than human text. Classifier filters tend to rate it highly, which can raise the synthetic share in your “high quality” slice. Provenance tracking helps; detection is imperfect.

Reproducibility. Common Crawl snapshots, extractors and models all change. Record every version and seed, or the dataset cannot be rebuilt.

Practical Recommendations

Start from an open, documented pipeline rather than a blank page. datatrove (FineWeb) and the Dolma toolkit both give you extraction, filtering and MinHash dedup components that have been exercised at multi-trillion-token scale. Reproduce a published ablation on a small model first so you trust your evaluation harness before changing anything.

Treat every threshold as a hypothesis. Fix a proxy model size (hundreds of millions to about two billion parameters), a token budget, and an evaluation panel of a dozen or more tasks, then change one pipeline stage at a time. Prefer early-signal benchmarks that are not saturated at small scale, and watch variance: single-seed differences of a fraction of a point are noise.

For dedup, begin with the (14, 8) layout on word 5-grams, because it has a documented track record, then test a looser setting. Inspect the ten largest clusters. Decide on global versus per-snapshot dedup by ablation, not on principle.

For quality filtering, train a small classifier on a positive set that reflects your target use, keep a fraction rather than a hard cut, and mix in some unfiltered data to preserve diversity. Re-check fairness and coverage after filtering.

Checklist before freezing a corpus:

Extraction compared on downstream training, not by eye.
Exact, fuzzy and (if tested) substring dedup parameters logged with the expected S-curve.
Largest duplicate clusters manually audited.
Classifier seeds, threshold and kept fraction documented, with a coverage check.
Decontamination run against every reported benchmark, plus a private held-out set.
Mixture weights chosen by proxy-model search, with an epoch cap per source.
Provenance, license and opt-out handling documented.
Versions of crawl, extractor, language-ID model and classifier recorded.

Frequently Asked Questions

What is pretraining data curation?

Pretraining data curation is the process of converting raw crawled text into a training corpus by extracting content, filtering by language and quality, deduplicating, removing benchmark overlap, and choosing sampling weights across sources. Its purpose is to maximise the useful information per training token. In practice it often matters as much as architecture, because the same model trained on better-curated data can reach a given benchmark score with substantially fewer tokens and less compute.

How does MinHash deduplication work?

MinHash turns each document into a set of n-gram shingles, applies many hash functions, and keeps the minimum value from each to form a signature. The chance that two signatures agree at a position equals the Jaccard similarity of the documents. LSH then splits signatures into bands and hashes each band, so documents sharing any band bucket become candidates. The band count and rows per band set the similarity threshold at which documents are merged.

How do I choose the number of bands and rows in MinHash LSH?

Use the approximate threshold (1/b)^(1/r) and the probability formula 1 – (1 – s^r)^b. For a 112-hash signature, 14 bands of 8 rows gives a threshold near 0.72, which flags documents with roughly 70 to 80 percent shingle overlap. More bands and fewer rows lower the threshold and increase recall and false positives. Validate by sampling candidate pairs and measuring true Jaccard on the actual data.

Is more deduplication always better for LLM training?

No. Removing exact and near-duplicate content reduces memorisation and wasted compute, but over-aggressive or global dedup can discard high-quality content that persists across crawls, as reported in the FineWeb ablations. Research on data-constrained scaling also shows that a few repeated epochs of good data cost little. The right level is empirical: run ablations on dedup scope and granularity rather than assuming maximal dedup helps.

What is the difference between FineWeb-Edu and DCLM filtering?

Both use a learned classifier to keep high-scoring documents. FineWeb-Edu trains its classifier on educational-value ratings produced by a large LLM annotator, while DCLM’s best configuration trains a fastText classifier on instruction-style positive examples versus random web negatives. They embody different definitions of quality, so they bias the resulting corpus differently. Both reported substantial benchmark gains over unfiltered or heuristic-filtered baselines.

How do you prevent benchmark contamination in pretraining data?

The standard approach is n-gram overlap matching: build n-gram sets from every evaluation benchmark and remove or flag training documents that contain them. It catches near-verbatim leaks but misses paraphrases and translations. Strengthen it by keeping a private held-out evaluation set, rerunning decontamination for each new crawl, and reporting scores with and without overlapping examples so readers can see how much of a result reflects memorisation.