This article is a technical systems and methods explainer for engineering and research audiences. It is not medical, clinical, or diagnostic advice.

Single-Cell Foundation Models: scGPT, Geneformer and the Language of the Cell

The human body carries tens of trillions of cells, and a single scRNA-seq experiment can profile the gene expression of hundreds of thousands of them at once. Each cell becomes a long, mostly empty vector of counts across roughly twenty thousand genes. Single-cell foundation models ask a bold question: if a transformer can learn language by predicting missing words, can it learn “the language of the cell” by predicting missing gene expression across tens of millions of cells? That framing has driven a wave of models, from scGPT to Geneformer to scFoundation, that treat cellular state the way large language models treat text.

This is a methods explainer for practitioners, not clinical or medical advice. It walks through how these models are built, what they can and cannot do, and why 2026 benchmarks keep humbling them against simple baselines.

What this covers: the data and tokenization, the pretraining objectives, the major models and how they differ, downstream tasks, the honest benchmark debate against PCA, and practical guidance for teams deciding whether to adopt them.

Context and Background

Single-cell RNA sequencing (scRNA-seq) measures the messenger RNA present in individual cells. Instead of averaging expression across a bulk tissue sample, it resolves the distinct transcriptional programs of each cell type in a mixture. The output is a count matrix: rows are cells, columns are genes, and each entry counts how many transcripts of that gene were captured in that cell.

Two properties make this data hard. It is extremely sparse, because most genes are not detected in most cells at typical sequencing depths. It is also noisy, with technical variation from capture efficiency, sequencing depth, and the specific chemistry of each experimental batch.

The sparsity is not a minor inconvenience; it is the defining feature of the data. A typical cell might have measurable counts for only a few thousand of the twenty thousand genes, and many of those counts are ones and twos. This is partly biology, since a cell only actively transcribes a subset of its genome, and partly technical dropout, where a transcript was present but never captured. Distinguishing “not expressed” from “expressed but missed” is genuinely hard, and it is the source of much downstream disagreement between methods.

The pre-foundation-model workflow standardized around a well-worn pipeline. Analysts normalize counts, select the top highly variable genes (often 2,000), and reduce dimensionality with principal component analysis (PCA) to around 50 components. They then build a nearest-neighbor graph, cluster with the Leiden or Louvain algorithm, and annotate clusters using marker genes. Toolkits like Scanpy in Python and Seurat in R made this reproducible and fast.

That classical stack works remarkably well, and it sets the bar every newer method must clear. It is unsupervised, cheap to run, and interpretable, because each principal component is a linear combination of genes you can inspect. Its weakness is that each dataset is analyzed largely in isolation, with limited transfer of knowledge across experiments, tissues, and labs.

When you need to combine datasets, the classical world reaches for dedicated integration tools. Harmony aligns batches in PCA space, and scVI fits a variational autoencoder that models counts probabilistically. These are strong, well-understood methods, and they matter here because they are the baselines that foundation models must also beat, not just plain PCA. A newer, larger model is only interesting if it improves on tools that already work.

Foundation models promised to change that. Borrowing the recipe from natural language processing, the idea is to pretrain one large transformer on a huge, diverse corpus of cells, then reuse its learned representation across many downstream tasks. For readers new to the broader trend, our overview of scientific foundation models in 2026 situates single-cell work alongside protein and climate models. The canonical single-cell toolkit background lives in the open-source Scanpy documentation, which remains the reference implementation for the classical pipeline.

How Single-Cell Foundation Models Work

Single-cell foundation models are transformers pretrained self-supervised on tens of millions of single-cell transcriptomes to learn transferable, low-dimensional representations of cellular state. They convert a cell’s sparse expression vector into a sequence of tokens, mask or corrupt part of that sequence, and train the network to reconstruct it. The resulting embeddings can be reused, frozen or fine-tuned, across many downstream analyses.

The pipeline has four stages: assemble and preprocess the data, tokenize genes and their expression, pretrain a transformer with a self-supervised objective, and extract embeddings for downstream tasks. Figure 1 shows the full flow.

Figure 1: The single-cell foundation model pipeline. A cells-by-genes count matrix is quality-controlled and normalized, then tokenized into gene and expression tokens. A transformer is pretrained with a self-supervised objective such as masked expression or rank prediction, producing cell and gene embeddings that feed cell-type annotation, batch integration, gene network inference, and in-silico perturbation.

The data and preprocessing

Pretraining corpora are aggregated from public repositories such as the CELLxGENE census, the Human Cell Atlas, and thousands of individual studies. The scale is what distinguishes single-cell foundation models from earlier deep learning on single cells. scGPT was pretrained on a repository of over 33 million cells, and scFoundation on more than 50 million human single-cell transcriptomic profiles. Geneformer’s original corpus of around 30 million cells was later extended to a substantially larger set, reported near 95 million, as more public data became available.

Why now? Three things converged. Public single-cell data crossed the scale, tens of millions of cells, where self-supervised pretraining becomes viable. The transformer recipe from language modeling had matured and was easy to adapt. And GPU capacity that was once reserved for language models became accessible to computational biology groups. The result was a burst of releases between 2023 and 2026, each claiming a more general representation of cellular state.

Before tokenization, counts are quality-controlled to drop low-quality cells and normalized to account for differences in sequencing depth. Genes are typically restricted to a shared vocabulary, often the roughly 20,000 protein-coding genes, so that every cell maps onto a common feature space.

Diversity of the corpus matters as much as raw size. A model trained on hundreds of studies, dozens of tissues, and many donors sees more of the space of possible cellular states than any single lab could assemble. But diversity also imports every technical artifact of every contributing study, so the model must learn to see through batch effects rather than memorize them. This tension between coverage and confounding runs through the entire field.

Cross-species pretraining raises the vocabulary problem sharply. Human and mouse genes do not map one to one, so a model spanning species needs a way to relate genes across genomes. Universal Cell Embeddings solves this by representing each gene through the protein it encodes, using ESM2 protein language-model embeddings as the gene token. That lets a single model reason about roughly 36 million cells across eight species in one shared space.

Tokenizing genes and expression

A language model tokenizes text into sub-words. A single-cell model must tokenize a numeric expression vector instead, and there are two dominant strategies. Figure 2 contrasts them.

Figure 2: Two tokenization strategies. Value binning, used by scGPT and scFoundation, represents each expressed gene as a gene-identity token paired with a discretized expression-value token. Rank-based tokenization, used by Geneformer, orders genes by their normalized expression rank within a cell and feeds that ordered sequence to the transformer.

The first strategy is value binning. Each expressed gene contributes a gene-identity token plus a token encoding its expression level, discretized into bins. This preserves both which gene is expressed and roughly how much, at the cost of a discretization choice that can lose fine-grained magnitude.

The second strategy is rank-based encoding, introduced by Geneformer. Genes are ranked by their expression within a cell, after normalizing against a genome-wide reference so that housekeeping genes do not dominate. The model sees an ordered list of gene tokens rather than explicit values. This makes the representation robust to some technical variation and read-depth differences, because ranks are more stable than raw counts.

Both approaches must confront a hard trade-off. Sequences have a finite length, expression is extremely sparse, and the dynamic range of counts spans several orders of magnitude. Every model makes different compromises on how many genes to keep and how to compress the long tail.

The choice has downstream consequences. Rank-based encoding throws away absolute magnitude, so it cannot easily distinguish a gene expressed at moderate versus very high levels if their ranks are similar. Value binning keeps magnitude but forces a discretization, and the number and spacing of bins becomes a hyperparameter that affects everything. Neither is obviously correct, which is why tokenization has become its own research subfield, with modular frameworks now built specifically to compare tokenization schemes head to head.

It helps to notice what the natural-language analogy hides. In text, tokens have a natural order and a fixed, meaningful vocabulary. In a cell, genes have no inherent sequence order, and the “meaning” of a token depends on continuous expression that must be squeezed into discrete symbols. The transformer machinery transfers, but the assumptions that made it work for language only partly hold for transcriptomics.

Pretraining objective and embeddings

Once tokenized, the sequence enters a transformer that uses self-attention to model relationships among genes within a cell. Pretraining is self-supervised, so no manual labels are needed. The most common objective is masked prediction: randomly hide a fraction of gene tokens or expression values, then train the model to recover them from context.

scGPT is generative and uses a masked expression prediction objective in the style of a generative pretrained transformer. Geneformer is bidirectional, closer to BERT, and predicts masked genes from their surrounding rank context. scFoundation adds a read-depth recovery task on top of masked modeling, explicitly teaching the model to harmonize the read-depth differences that plague cross-dataset analysis.

Self-attention is the mechanism that makes this work. For each gene token, attention computes a weighted combination of the other tokens in the cell, letting the model condition each gene’s representation on the rest of the transcriptome. This is how the model can, in principle, learn that two genes co-vary or that one gene’s expression constrains another’s. The attention weights are also what many gene-network methods later mine, though whether they encode causal regulation or mere correlation is contested.

After pretraining, the network produces embeddings. A cell embedding is a dense vector summarizing that cell’s state, and gene embeddings capture context-dependent gene behavior. These vectors are the product you actually use downstream, and everything that follows is about how well they transfer.

A useful mental model is that pretraining compresses the corpus into a geometry. Cells with similar transcriptional programs should land near each other, so that a cluster in embedding space corresponds to a cell type or state. The hope is that this geometry is universal enough to be useful on data the model never saw, without retraining. The recurring disappointment, discussed below, is that a plain PCA also produces a useful geometry, often for far less compute.

Deeper Analysis: Models, Tasks, and Benchmarks

The field now holds a dozen major models with meaningfully different design choices. The table below compares four of the most influential, using figures reported by their authors. Treat parameter and cell counts as the values published for each release; some models have since been extended.

Model	Architecture	Pretraining cells	Tokenization	Objective
scGPT	Generative transformer (GPT-style)	~33M	Value binning (gene + binned value)	Masked expression prediction
Geneformer	Bidirectional transformer (BERT-style)	~30M, later extended (~95M)	Rank-based gene tokens	Masked gene prediction
scFoundation	Asymmetric transformer, ~100M params	~50M	Value binning, read-depth aware	Masked modeling + read-depth recovery
UCE	Transformer with protein-based gene tokens	~36M, 8 species	Gene tokens via ESM2 protein embeddings	Masked / contrastive cell objective

Each design choice reflects a hypothesis about what matters. Geneformer bet that relative rank is the robust, transferable signal, and its rank-value encoding was explicitly motivated by resistance to batch and depth artifacts. scGPT bet that keeping expression values and using a generative objective would let one model serve both representation and generation, including perturbation. scFoundation bet that read depth is a first-class confounder worth a dedicated pretraining task, so it learns to reconstruct high-depth profiles from low-depth inputs. UCE bet that grounding gene tokens in protein biology would unlock cross-species transfer.

Beyond these four, the landscape is crowded. CellPLM extended cell-level pretraining with masked expression modeling and treats cells, not just genes, as tokens, so it can model relationships between cells and not only between genes. GeneCompass injects prior biological knowledge such as regulatory relationships into pretraining, blending data-driven learning with curated priors. CellFM pushed scale to roughly 100 million human cells with improved gene-gene and cell-cell modeling. Nicheformer and related spatial models extend the idea to spatial transcriptomics, where a cell’s physical neighbors matter. A newer thread replaces attention with state-space model (SSM) backbones, such as scMamba, aiming for cheaper scaling to very long gene sequences without the quadratic cost of attention.

The proliferation is itself informative. When many groups produce many models with divergent choices and no clear winner, it usually means the problem is not yet solved and the right inductive bias is still unknown. That is a healthy sign of an active field, but it is also a warning against betting a pipeline on any single model as if it were settled.

It is instructive to compare this with structure prediction, where a single dominant architecture emerged. The protein-folding field converged because a clear objective, structure accuracy, and gold-standard labels existed, as our breakdown of AlphaFold 3’s diffusion architecture describes. Single-cell biology lacks an equivalent ground truth; there is no crystal structure of a “correct” cell embedding. Without a decisive benchmark, the field cannot yet crown a winner, and that absence of ground truth is part of why the PCA debate persists.

Downstream tasks and adaptation

The embeddings support several distinct tasks. Cell-type annotation assigns a label to each cell, often by transferring labels from a reference. Batch integration aligns cells from different experiments into a shared space so biological signal is not confounded by technical batch. Gene regulatory network inference uses attention patterns or gene embeddings to hypothesize which genes co-regulate. In-silico perturbation prediction asks what a cell’s transcriptome would look like if a gene were knocked out or over-expressed.

These tasks differ in how much they truly test the model. Annotation and integration mostly ask whether the embedding geometry is good, which a strong baseline can also deliver. Perturbation prediction asks something harder, namely whether the model has learned causal or at least counterfactual structure about gene relationships. Gene network inference sits in between, since it depends on whether attention encodes regulation or only co-expression. Reading any benchmark, it pays to ask which of these questions the task actually answers.

Adaptation happens in two modes, shown in Figure 3. Zero-shot uses the frozen pretrained model to embed cells with no additional training, which is fast and requires no labels. Fine-tuning updates the model weights, or trains a small task head on top, using labeled data for the specific task.

Figure 3: Adaptation modes. A pretrained model can be used zero-shot with frozen embeddings for label-free tasks like annotation and integration, or fine-tuned with a task head for supervised tasks like perturbation prediction and gene regulatory network inference. Both paths feed downstream biological interpretation.

The promise is that pretraining front-loads general knowledge, so a modest amount of task-specific data goes further. In practice the payoff is uneven, and it depends heavily on whether you use the model zero-shot or fine-tune it.

The honest benchmark debate

Here is where 2026 gets interesting, and where the marketing meets the data. Multiple independent benchmarks have found that single-cell foundation models do not consistently beat simple baselines, especially in zero-shot settings. Figure 4 captures the structure of this debate.

Figure 4: The evaluation debate. On the same task and metric, scFM embeddings are compared against a classical baseline of PCA on highly variable genes. Foundation models tend to help with fine-tuned tasks and label transfer, while the baseline frequently wins in zero-shot settings. Fair reporting shows both.

A widely cited perturbation-analysis benchmark, pointedly subtitled “one PCA still rules them all,” found that a simple PCA baseline matched or outperformed transcriptomics foundation models on perturbation-effect tasks. A separate zero-shot evaluation by Kedzierska and colleagues reported that Geneformer and scGPT embeddings, used zero-shot, underperformed straightforward baselines on core tasks like clustering and batch integration.

The pattern repeats in unified frameworks that benchmark many models at once. Studies comparing thirteen or more single-cell foundation models against classical baselines, across dozens of datasets, typically use a baseline of 2,000 highly variable genes reduced to 50 PCA dimensions. In zero-shot settings, that baseline is frequently competitive and sometimes superior. Some benchmarks report a more nuanced picture where scGPT can beat PCA while other models trail it, which underscores that results depend on the exact model, task, and dataset.

Newer 2026 benchmarks have refined the picture rather than overturned it. Low-supervision studies ask how models fare when only a handful of labels are available, a realistic setting for new tissues or diseases, and find that single-cell foundation models can help most exactly there, where labels are scarce and pretraining knowledge has something to add. Dynamics-focused benchmarks, testing whether zero-shot embeddings recover continuous trajectories rather than discrete clusters, again find that simple highly variable gene baselines are hard to beat. The consistent lesson is that the gap narrows or reverses in the low-label, supervised regime and widens against the models in the label-free, zero-shot regime.

It is also worth naming the incentive problem. Model papers tend to report the configurations where their model wins, and independent benchmarks tend to report where it loses. Neither is dishonest, but the reader has to triangulate. When a model’s own paper and three independent benchmarks disagree, weight the independent evidence, especially benchmarks that publish code and use leakage-safe splits.

There is also an interpretability critique. Analyses of what attention actually learns suggest it often captures gene co-expression, which correlates with the input structure, rather than uniquely identifying regulatory drivers. In other words, a model can look like it “understands” regulation while mostly rediscovering correlations already visible in the data. That does not make the models useless, but it should temper claims that they reveal novel biology on their own.

Why might a huge pretrained model lose to fifty principal components? One reason is that clustering and integration reward smooth, well-separated geometry, which linear methods produce cleanly on the highly variable genes that carry most cell-type signal. Another is that zero-shot embeddings were never optimized for your specific metric, whereas PCA is, in effect, tuned to variance in your data directly. A third is leakage in the other direction: baselines are computed on your data alone, so they cannot be contaminated by pretraining overlap. None of this means single-cell foundation models are wrong; it means the easy tasks were already easy.

The fairer summary is conditional. Fine-tuning with labeled data, and supervised tasks like label transfer, are where single-cell foundation models most often add value. Zero-shot use on integration and clustering is where they most often disappoint relative to PCA. Anyone evaluating these models should hold both facts at once, and should be suspicious of any comparison that reports only the setting favorable to the conclusion the authors wanted.

Trade-offs, Gotchas, and What Goes Wrong

Batch effects are the first trap. Foundation models are marketed as batch-robust, but a model that integrates too aggressively can erase real biological differences along with technical noise. Over-correction looks like clean integration and quietly destroys the signal you care about. This is why integration benchmarks report both a batch-mixing score and a biological-conservation score; a method that maximizes one at the expense of the other is not actually better, just differently biased.

Perturbation prediction deserves special scrutiny. The most exciting claim for these models is that they can forecast unseen genetic or chemical perturbations, effectively simulating experiments. Yet the perturbation benchmark subtitled “one PCA still rules them all” found that on held-out perturbations a linear baseline was hard to beat, suggesting that much of the apparent skill came from predicting the average cell rather than the perturbation-specific shift. Before trusting an in-silico screen, confirm the model beats the trivial baseline of predicting no change or the population mean.

Evaluation pitfalls are the second. Many published wins depend on the metric and the dataset split. A model that shines on one clustering metric can lag on another, and averaging across a handful of friendly datasets can hide poor generalization.

Data leakage is a subtle and serious risk. If cells from the same study, donor, or even the same dataset appear in both pretraining and evaluation, reported performance is optimistic. Because pretraining corpora aggregate thousands of public studies, evaluation sets can overlap with pretraining data unless splits are constructed carefully.

Compute cost is real. Pretraining these models consumes substantial GPU resources, and even fine-tuning is far heavier than running PCA. If a linear baseline matches a foundation model on your task, the extra compute buys nothing.

Distribution shift limits transfer. A model pretrained mostly on human tissues may falter on a new species, a rare tissue, a disease state, or a different sequencing chemistry. The “universal” framing oversells how far the learned representation actually generalizes. Disease tissue is a particular concern, because pretraining corpora skew toward healthy reference atlases, and the rare or aberrant states that clinical questions care about are underrepresented.

There is a scaling-law caveat too. In language modeling, more data and parameters reliably improved performance, which is part of why the field expected the same for cells. So far, the single-cell evidence for clean scaling laws is mixed; larger models and larger corpora have not delivered proportionate gains on every downstream task. That may reflect data quality and diversity limits rather than a hard ceiling, but it should discourage assuming that the next, bigger model will simply be better.

Reproducibility is a further gotcha. Preprocessing choices, gene vocabularies, and normalization differ across model implementations, so the same dataset can yield different embeddings depending on which wrapper you use. Unified deployment frameworks have emerged in 2026 precisely to standardize this, but until you control preprocessing yourself, apparent performance differences between models can be artifacts of plumbing rather than science.

Finally, biological interpretability is limited. Embeddings are dense vectors, not mechanistic explanations, and attention weights are suggestive rather than causal. Recent interpretability work, including circuit-style analysis of single-cell models, is only beginning to open the black box. Treat model outputs as hypotheses to validate experimentally, not as established regulatory findings.

Practical Recommendations

Start by defining the task precisely, because the answer to “should I use a foundation model” depends entirely on it. Zero-shot integration and clustering favor classical baselines more often than not. Supervised label transfer and fine-tuned perturbation modeling are where single-cell foundation models more reliably help.

Always benchmark against PCA, not just against another deep model. The 2,000-highly-variable-gene plus 50-dimensional PCA baseline is cheap, fast, and stubbornly strong. If you cannot beat it on your data and your metric, you have your answer.

Guard against leakage by constructing splits at the study or donor level, never by random cell shuffling. Confirm that your evaluation data is not part of the model’s pretraining corpus. Report the metric, the split strategy, and the baseline alongside every number.

Control the preprocessing yourself rather than trusting each model’s default wrapper. Use the same gene vocabulary, normalization, and quality thresholds across every method you compare, so differences reflect the models and not the plumbing. When possible, adopt one of the 2026 unified deployment frameworks that standardize these steps across models. Consistency in preprocessing often changes rankings more than the choice of architecture.

Match the model to the task. Use this checklist:

Benchmark every candidate model against a PCA-on-HVGs baseline on your own data.
Split by study or donor to prevent data leakage; verify no overlap with pretraining corpora.
Prefer fine-tuning over zero-shot when you have labels and the task is supervised.
For cross-species or spatial work, choose a model pretrained for that setting (for example, UCE for multi-species, Nicheformer for spatial).
Report multiple metrics; do not cherry-pick the one that flatters the model.
Budget compute honestly and stop if a linear baseline ties the foundation model.
Treat attention-derived networks as hypotheses, then validate experimentally.

Above all, resist the framing that single-cell foundation models are strictly better than what came before. They are a powerful, general tool that sometimes wins decisively, sometimes ties a linear baseline, and sometimes loses. The teams getting real value in 2026 are the ones treating these models as one option in a portfolio, measured honestly against cheap baselines on their own data, rather than as a default that replaces the classical pipeline wholesale.

Frequently Asked Questions

What is a single-cell foundation model?

It is a large transformer pretrained self-supervised on tens of millions of single-cell RNA sequencing profiles to learn transferable representations of cellular state. Instead of analyzing one dataset in isolation, it aims to encode general knowledge about gene expression that can be reused across tasks like cell-type annotation, batch integration, and perturbation prediction. Models such as scGPT, Geneformer, and scFoundation are the best-known examples, each differing in architecture, tokenization, and pretraining objective.

How is scGPT different from Geneformer?

scGPT is a generative, GPT-style transformer trained with masked expression prediction, using value binning to encode both gene identity and expression level. Geneformer is a bidirectional, BERT-style model that uses rank-based gene tokens, ordering genes by their normalized expression within each cell and predicting masked genes. In short, scGPT keeps explicit expression values while Geneformer relies on relative ranks, which makes it more robust to read-depth and batch variation but discards magnitude information.

Do single-cell foundation models beat PCA?

Not consistently, and that is the central 2026 debate. Independent benchmarks, including one perturbation study subtitled “one PCA still rules them all,” report that a simple PCA baseline matches or beats foundation models in many zero-shot settings. Foundation models more reliably help when fine-tuned with labels on supervised tasks such as label transfer. The honest answer is that it depends on the task, the dataset, and whether you fine-tune, so you must benchmark against PCA yourself.

What is in-silico perturbation prediction?

It is using a model to predict how a cell’s transcriptome would change if a gene were experimentally perturbed, such as knocked out or over-expressed, without running the wet-lab experiment. Foundation models attempt this by learning gene relationships during pretraining, then simulating the downstream effect of altering one gene. Results are promising in some cases but still uneven, and predicted effects should be treated as hypotheses to validate rather than conclusions.

Are there single-cell foundation models for spatial transcriptomics?

Yes. Spatial transcriptomics measures gene expression while preserving each cell’s physical location in tissue, which adds a neighborhood dimension that pure scRNA-seq lacks. Models such as Nicheformer extend the foundation-model approach to jointly represent a cell and its spatial context, and newer multi-scale spatial models continue to appear. They are earlier in maturity than dissociated single-cell models, and the same benchmarking caution against simple baselines applies.

Should my lab adopt a single-cell foundation model now?

Adopt selectively, not by default. If your task is supervised label transfer or fine-tuned perturbation modeling, a foundation model is worth trialing against a PCA baseline. If your task is unsupervised zero-shot integration or clustering, the classical Scanpy or Seurat pipeline is often competitive at a fraction of the cost. Let a head-to-head benchmark on your own data, with leakage-safe splits, decide.

Single-Cell Foundation Models: scGPT & Geneformer (2026)