Kimi K3 Explained: Moonshot’s 2.8T Open-Weight Reasoning Model (2026)

Kimi K3 is Moonshot AI’s newest flagship large language model, and it is the largest open-weight model anyone has shipped with a straight face: 2.8 trillion total parameters, spread across 896 experts, of which only 16 fire on any given token. That routing discipline is the entire story. A dense 2.8T model would be unrunnable outside a hyperscaler’s basement; a sparse one that activates roughly 1.8% of its parameter pool per token is something a well-funded lab, or a determined enthusiast with a rack of H200s, can actually serve. Kimi K3 pairs that scale with a 1,000,000-token context window, native multimodal understanding, and two attention-layer changes Moonshot says buy a 2.5x jump in scaling efficiency over its predecessor. What this covers: the architecture behind Kimi K3, how it was trained, what the third-party benchmarks actually say, what it costs to run yourself, and where it falls short.

Kimi K3 is a 2.8-trillion-parameter mixture-of-experts language model from Moonshot AI, released with fully open weights on July 26, 2026. It activates only 16 of 896 experts per token, supports a 1-million-token context window, and is positioned as a reasoning-and-agentic model competing with GPT-5.6, Claude Fable 5, and DeepSeek-V4.

Lineage and Context

Kimi K3 did not appear from nothing. It is the fourth act of a lineage that started with Kimi K2, Moonshot’s 1-trillion-parameter, 32-billion-active MoE model trained on 15.5 trillion tokens using the Muon optimizer, with a 256K context window that was considered generous at the time. K2 established the pattern Moonshot has followed since: publish real open weights, not a demo checkpoint, and compete directly with closed frontier labs on agentic and coding benchmarks rather than chasing a narrower open-source niche.

Two intermediate releases followed. K2.5, shipped January 27, 2026, tightened instruction-following and tool use without a headline architecture change. K2.6, released April 20, 2026, under a Modified MIT license, pushed further on reasoning post-training and set the stage for what Moonshot was quietly calling its “Thinking” line — a separate mode, distinct from the base Instant mode, tuned specifically for multi-step problems where the model reasons before it answers. That Instant-versus-Thinking split, introduced properly in Kimi K2 Thinking, is a design decision K3 inherits and extends rather than reinvents.

Moonshot officially retired the entire K2 series on May 25, 2026, redirecting users to Kimi K3. That is an unusually blunt signal for an open-weight lab: it means Moonshot considers K3 a strict upgrade, not a parallel option, and that anyone building on K2 checkpoints should plan a migration path. K3 went live through Moonshot’s own apps and API on July 16, 2026, roughly ten days ahead of the open-weight drop on July 26 — a day earlier than the July 27 target Moonshot had reportedly set internally, which is a small but telling detail about how the release was managed. The reported license lineage continues from K2’s Modified MIT terms, though as with any fast-moving open-weight release, engineers evaluating K3 for production use should verify the exact license text in the model repository rather than assuming continuity.

Where K3 sits in the broader landscape matters as much as its internal numbers. 2026’s open-weight MoE field is crowded: DeepSeek-V4 pushed sparse MoE efficiency hard from the China side, Mistral Large 3 represents Europe’s dense-to-sparse pivot, and closed frontier models like GPT-5.6 Sol and Claude Fable 5 still set the ceiling that every open release gets measured against. Kimi K3 is Moonshot’s answer to all three at once: bigger than DeepSeek-V4’s total parameter count, more open than the closed frontier, and — on paper — more efficient per active parameter than its own predecessor.

Architecture

Kimi K3’s headline number, 2.8 trillion total parameters, is almost meaningless on its own; what matters is that only 16 of its 896 experts activate per token, roughly 1.8% of the total pool. That sparsity ratio is what separates a model you can reason about from a number you can only marvel at — active compute per token stays in a range comparable to models a fraction of K3’s nominal size, while the full expert pool gives the router enormous specialization headroom to draw on.

Mixture-of-experts is, at its core, a way of decoupling parameter count from compute cost. Instead of every token passing through every parameter (as in a dense transformer), a lightweight router network looks at each token’s hidden state and picks a small subset of “expert” feed-forward blocks to process it. Different experts can specialize — one might get better at code syntax, another at long-form narrative structure, another at arithmetic — without any single token paying the compute cost of consulting all of them. If you haven’t worked through the mechanics of routing, load balancing, and expert capacity before, our explainer on mixture-of-experts architecture is a useful primer before going further into K3’s specific choices.

What makes K3 architecturally distinct from a “just bigger MoE” story is a pair of changes Moonshot reports drove roughly a 2.5x scaling-efficiency gain over Kimi K2: Kimi Delta Attention and Attention Residuals. Moonshot has not published the full mechanistic detail of either, so what follows is a reasoned interpretation of what these mechanisms plausibly buy, not a claim about undisclosed internals.

Kimi Delta Attention: why hybrid linear attention matters at 1M tokens

Standard self-attention scores every token against every other token, which costs compute and memory that scale quadratically with sequence length. That is fine at 8K or 32K tokens; it becomes brutal at 1,000,000. Hybrid linear-attention schemes — the general family Kimi Delta Attention reportedly belongs to — replace or supplement full quadratic attention with a recurrent or kernelized approximation that scales closer to linearly with sequence length, typically by maintaining a compressed running state rather than materializing a full attention matrix over the entire context.

The trade-off is usually recall precision: linear-attention approximations can blur fine-grained token-to-token relationships that full attention captures exactly, which is why most hybrid designs keep a subset of layers or a subset of heads running full quadratic attention and use the linear mechanism elsewhere. If Kimi Delta Attention follows that pattern — and the name and the “hybrid” framing strongly suggest it does — the practical payoff is straightforward: K3 can hold a 1-million-token context in memory and process it without the attention cost exploding, while still preserving enough exact-attention capacity to avoid the recall degradation that pure linear-attention models are known for.

Attention Residuals: rethinking cross-layer information flow

The second reported change, Attention Residuals, concerns how information propagates between transformer layers rather than within a single attention operation. In a standard transformer, each layer’s output feeds into the next primarily through the residual stream — the running sum of every layer’s contribution. As models get deeper, that residual stream becomes a bottleneck: early-layer signal can get diluted or overwritten by the time it reaches layer 80 or 100. Changes to how attention outputs get folded back into that residual path — allowing later layers more direct access to earlier attention outputs, for instance — are a known lever for improving gradient flow and representational capacity in very deep networks. Moonshot has not disclosed the specific mechanism, so treat “Attention Residuals” as a labeled but under-specified technique; the plausible payoff is better use of depth at 2.8T-parameter scale, where naive residual stacking would otherwise waste capacity.

Together, Moonshot reports these two changes as the primary driver of K3’s 2.5x scaling-efficiency improvement over K2 — meaning K3 gets more capability per unit of training and inference compute than a naively scaled-up K2 would have. That figure comes from Moonshot itself and has not been independently reproduced, so it belongs in the same “reported, not yet verified” bucket as the architecture details it explains.

K3 is natively multimodal, with visual understanding built into the base model rather than bolted on through a separate vision adapter — consistent with where most 2026 flagship releases have landed, since joint text-image training tends to produce better grounding than late-stage vision fine-tuning. Moonshot has not published exact tokenizer or vocabulary size figures for K3 at the time of writing, so those numbers are omitted here rather than guessed.

Figure 1: How a token moves through Kimi K3 — from tokenization through Kimi Delta Attention and Attention Residuals to MoE routing, where 16 of 896 experts activate.

The diagram above traces a single token’s path: it enters through the embedding layer, passes through the hybrid Kimi Delta Attention mechanism (with Attention Residuals shaping how that output threads back into the network), and finally reaches the MoE router, which selects 16 experts from the pool of 896 — plus any always-on shared experts — before the result is projected back into logits.

Training

Moonshot has disclosed less about K3’s training run than it did for K2’s 15.5-trillion-token, Muon-optimizer pretraining, so most of what follows about K3 specifically is inferred from the K2 lineage and general practice at this model scale, and should be read as reasoned estimate rather than confirmed fact where explicitly marked.

Pretraining a 2.8T-parameter MoE at K3’s scale requires a data pipeline built around three concerns: raw scale, quality filtering, and — new for K3 relative to earlier Kimi releases — genuine multimodal pairing, since native visual understanding needs image-text data mixed into pretraining rather than added afterward. Given that K2 trained on 15.5 trillion tokens and K3 is reported to carry forward the same general training philosophy at meaningfully larger active and total parameter counts, it is reasonable to assume K3’s pretraining corpus is at least comparable in scale, likely larger, though Moonshot has not published an exact token count for K3 and none should be assumed.

The Muon optimizer, which Moonshot used for K2, is worth understanding because it is a real point of technical differentiation from the Adam-family optimizers most labs still default to. Muon applies a matrix-orthogonalization step to gradient updates for two-dimensional weight matrices, which tends to produce more stable, better-conditioned updates at very large batch sizes and helps avoid some of the loss spikes that plague long pretraining runs on massive MoE architectures. Whether K3’s pretraining also used Muon is not explicitly confirmed for K3 itself, but given Moonshot’s public commitment to it for K2 and the continuity of the K-series training stack, it is a reasonable assumption — flagged here as such rather than stated as fact.

Post-training is where K3’s reasoning identity gets built, and this is the stage with the clearest evidence, since the shipped model behavior (Instant and Thinking modes) tells you directly what the post-training pipeline was optimized for. After base pretraining, the model goes through supervised fine-tuning on curated instruction and reasoning-trace data, followed by what the industry broadly calls RLVR — reinforcement learning with verifiable rewards, where the model is trained against tasks (math, code, structured reasoning) that have a checkable correct answer, rather than relying purely on human preference judgments. RLVR has become the dominant post-training technique for reasoning models across the industry in 2026 because it scales without needing proportionally more human raters, and it produces the kind of step-by-step, self-correcting behavior visible in K3’s Thinking mode.

Layered on top of RLVR, preference-based alignment — RLHF, DPO, or a hybrid of the two — shapes tone, refusal behavior, and instruction adherence for cases where there is no single verifiable answer. The result is a model that ships with two distinct operating modes: Instant, for latency-sensitive queries where a direct answer is good enough, and Thinking, which spends additional inference-time compute working through intermediate reasoning steps before committing to a final answer. This mirrors the same test-time-compute philosophy that made reasoning models like OpenAI’s o3 a turning point for the field — trading inference latency and cost for accuracy on hard problems, on a per-query basis rather than baking a single fixed capability level into every response.

Figure 2: Kimi K3’s training pipeline, from curated pretraining data through supervised fine-tuning and reasoning-focused post-training into the shipped Instant and Thinking modes.

The diagram traces the reported pipeline: a large curated multimodal corpus feeds pretraining, producing a base checkpoint that then undergoes supervised fine-tuning, RLVR-style reasoning-trace training, and preference alignment, before splitting into the Instant and Thinking inference modes that ship in the final release.

Capabilities and Benchmarks

On the third-party benchmarks that carry the most weight in 2026, Kimi K3 lands just behind the two closed frontier leaders rather than at the top: Artificial Analysis’s Intelligence Index v4.1 scores K3 at 57.1, placing it roughly third among tracked model families, behind GPT-5.6 Sol Max at 58.9 and Claude Fable 5 at 59.9. A 1.8-to-2.8-point gap against the top two frontier models is real but narrow — closer to rounding-error territory than a generational gap — and worth treating with the standard caveat that composite leaderboard indices move week to week as labs update models and as contamination concerns get raised and investigated.

Two other reported numbers stand out. On LMArena’s Frontend Code leaderboard, K3 ranks #1 with a score of 1679, ahead of every other tracked model on that specific coding-oriented, human-preference benchmark — a genuinely strong result for an open-weight model against closed competitors. And on a 1-million-token long-context evaluation, K3 scored 90.4 with no explicit context management, meaning the model was tested against its full advertised context window without any retrieval-augmentation or chunking scaffolding to help it. That is the kind of number that validates the architectural bet on Kimi Delta Attention: a model that degrades badly at long context would post a much lower score here, since nothing masks weak long-range recall in a raw full-context test.

Benchmark	Kimi K3	Comparison
Artificial Analysis Intelligence Index v4.1	57.1	Claude Fable 5: 59.9, GPT-5.6 Sol Max: 58.9
LMArena Frontend Code	1679 (rank #1)	Highest tracked score at time of writing
1M-token context eval (no context management)	90.4	Reported as a strong result at full advertised context length

A methodology note worth taking seriously: leaderboard composite indices weight different task categories differently, and small methodology changes between index versions (v4.1 versus a future v4.2, for instance) can shift a model’s rank without any change to the model itself. LMArena scores are derived from aggregated human preference votes, which correlate with real usefulness but are also subject to prompt-selection bias and rater demographics. And a single long-context evaluation score, however impressive, does not tell you how the model behaves on every task type at 1M tokens — needle-in-haystack retrieval, multi-hop reasoning across distant sections, and generation quality at extreme context lengths can diverge even within the same model. None of these numbers should be read as a definitive final verdict; treat the three above as a snapshot from named third-party evaluators, current as of this writing, not a permanent ranking.

Figure 3: Kimi K3’s reported third-party benchmark standing — Artificial Analysis Intelligence Index, LMArena Frontend Code rank, and the 1M-token long-context evaluation.

The diagram groups K3’s three headline reported scores by source: its Intelligence Index position relative to the two frontier closed models ahead of it, its #1 rank on LMArena’s coding-specific leaderboard, and its long-context evaluation score tested at the full 1M-token window.

Access and Deployment

Kimi K3 is available two ways: through Moonshot’s managed API and apps, live since July 16, 2026, or as fully open weights, published July 26, 2026, for anyone with the hardware to self-host. The managed path is the pragmatic default for most teams — usage-based pricing, no infrastructure to provision, and Moonshot handles serving the 2.8T-parameter model’s expert-parallel inference at scale, which is nontrivial engineering that most teams should not want to own.

Self-hosting is where K3’s sparsity ratio becomes the deciding factor in whether this is even feasible. The headline 2.8 trillion parameters describes the full checkpoint that must be loaded into accessible memory across a cluster — sparsity reduces active compute per token, not the storage and memory footprint of the model itself, since any token could in principle route to any expert. At full precision, a parameter count in that range implies a multi-terabyte checkpoint; even aggressive quantization to FP8 or INT4 leaves a footprint that requires a multi-GPU, likely multi-node cluster with high-bandwidth interconnects, since expert-parallel serving needs fast communication between GPUs holding different experts. This is not a model that runs on a single high-end workstation, regardless of quantization — the honest floor is a serious multi-GPU deployment, and teams evaluating self-hosting should budget accordingly rather than assuming quantization alone closes the gap.

Quantization choices matter more for K3 than for most models because of its scale. FP8 roughly halves memory footprint relative to FP16/BF16 with typically minor quality loss on well-calibrated checkpoints; INT4 goes further but risks larger accuracy degradation, particularly on the reasoning-heavy Thinking mode where small per-step errors can compound across a long chain of intermediate reasoning tokens. Teams self-hosting K3 for reasoning-heavy workloads should validate quantized accuracy against their specific task distribution rather than assuming published aggregate benchmark scores transfer unchanged to a quantized deployment.

Cost, in practice, breaks into two very different shapes. API access converts to a straightforward per-token cost with no capital outlay, which is the right model for variable or unpredictable load, or for teams that don’t want to carry GPU capacity risk. Self-hosting converts to amortized hardware cost — the multi-GPU cluster, its power and cooling, and the ops team to run it — against which per-token marginal cost can look very cheap at high, sustained utilization, but which carries real fixed cost and idle-capacity risk at low or spiky utilization. As a rule of thumb across the industry, self-hosting a model at K3’s scale only beats API pricing once utilization is high and sustained enough to amortize both the hardware and the specialized MLOps expertise expert-parallel serving requires; below that utilization threshold, the API is usually cheaper once engineering time is priced in.

Figure 4: Two paths to running Kimi K3 — the managed API’s pay-per-token simplicity versus the open-weights self-host path through quantization and expert-parallel multi-GPU serving.

The diagram lays out both routes from the same starting point: the managed API branch resolves quickly into simple, usage-billed integration, while the self-host branch requires downloading the full checkpoint, choosing a quantization scheme, provisioning a multi-GPU cluster capable of expert-parallel serving, and only then reaching a per-token inference cost that has to be weighed against sustained hardware amortization.

Limitations, Safety, and Failure Modes

No model at this scale ships without real trade-offs, and K3’s specific combination of extreme sparsity, a hybrid attention mechanism, and a brand-new architecture generation means some failure modes are still being mapped by the broader community rather than fully characterized by Moonshot itself.

Long-context degradation is the first place to look skeptically, despite the strong reported 90.4 score on the 1M-token evaluation. A single aggregate score does not guarantee uniform performance across the full context window — models frequently show a “lost in the middle” effect, where information placed in the middle of a very long context is retrieved less reliably than information at the start or end, even when overall scores look strong. Because Kimi Delta Attention is a hybrid mechanism balancing linear-attention efficiency against full-attention recall, it is reasonable to expect some recall degradation at the far end of the 1M-token window on tasks that were not well represented in the specific evaluation that produced the 90.4 figure — this is an inference from how hybrid attention mechanisms generally behave, not a documented K3-specific finding, and worth testing directly against your own long-document use case before relying on it.

Hallucination behavior in reasoning models generally follows a specific pattern worth knowing before deploying K3 in a Thinking-mode-heavy workflow: multi-step reasoning chains can produce confident, internally consistent-sounding errors partway through a chain of thought, and because each step conditions on the previous one, an early mistake can compound rather than self-correct. RLVR-style training reduces this by rewarding verifiably correct final answers, but it optimizes for getting the final answer right on the training distribution’s task types — math, code, structured logic — more reliably than it guarantees faithful, error-free intermediate reasoning on open-ended or out-of-distribution problems. Treat K3’s visible reasoning traces as useful for debugging and calibration, not as a guarantee of correctness at every step.

As a very new, very large open-weight release, K3 has not yet been through the multi-month red-teaming cycle that closed frontier models accumulate before and after release. Open-weight distribution means jailbreak techniques discovered against K3 propagate through the community fast — both good, in that patches and community-hardened deployment configs also propagate fast, and bad, in that anyone deploying K3 in a customer-facing product should assume publicly known jailbreak patterns for MoE reasoning models will eventually be tried against it, and should not rely solely on the model’s built-in refusal training as a safety boundary.

Finally, the MoE architecture itself introduces a subtler failure class: expert load imbalance. If certain experts are disproportionately selected by the router for common query types, those experts effectively become bottlenecks, and their quality dominates the model’s behavior on high-frequency tasks in ways that are harder to audit than a dense model’s uniform parameter usage. Moonshot has not published detailed load-balancing diagnostics for K3’s 896-expert pool, so this remains a known risk category for MoE models generally rather than a documented K3-specific issue — but it is the kind of failure mode worth testing for on your own most common query patterns rather than assuming away.

How Kimi K3 Compares

Use case	Kimi K3	DeepSeek-V4	Mistral Large 3	GPT-5.6 Sol
Long-document analysis (>500K tokens)	Strong — native 1M context, 90.4 on full-window eval	Context window typically shorter; check current specs	Context window typically shorter; check current specs	Strong, closed-model context handling, cost premium
Self-hosted deployment on owned hardware	Possible but demanding — multi-GPU, expert-parallel serving required	Generally more accessible at smaller active-parameter footprint	Dense-leaning design, different hardware profile	Not available — closed model, API only
Coding / agentic workflows	Very strong — #1 reported on LMArena Frontend Code	Strong open-weight coding performance, worth direct comparison	Competitive, different training emphasis	Frontier-tier, typically at higher cost
Cost-sensitive, high-volume API use	Competitive via Moonshot API; can be cheaper self-hosted at high sustained utilization	Competitive open-weight pricing	Competitive open-weight pricing	Premium closed-model pricing

This is a directional comparison, not a precision leaderboard — exact current pricing and benchmark numbers for every peer model move often enough that readers evaluating a specific decision should re-check each vendor’s current published numbers rather than treating this table as a permanent snapshot. The consistent pattern across the row is that K3’s specific advantages — the 1M-token window and top-ranked coding leaderboard result — are real and reported by independent third parties, while its self-hosting story is genuinely more demanding than smaller open-weight peers simply because of its total parameter footprint.

Frequently Asked Questions

What is Kimi K3?

Kimi K3 is Moonshot AI’s flagship open-weight large language model, released with full weights on July 26, 2026. It is a 2.8-trillion-parameter mixture-of-experts model that activates only 16 of 896 experts per token, supports a 1-million-token context window, and includes native multimodal visual understanding. It succeeds the Kimi K2 series, which Moonshot officially retired in May 2026.

How many parameters does Kimi K3 have?

Kimi K3 has 2.8 trillion total parameters distributed across 896 experts. Because it uses a mixture-of-experts architecture, only 16 experts activate per token — about 1.8% of the total pool — which keeps active compute per token far below what the 2.8T headline figure suggests, while still requiring the full checkpoint in memory to serve.

What is Kimi Delta Attention?

Kimi Delta Attention is Moonshot’s reported hybrid linear-attention mechanism in Kimi K3, one of two architectural changes (alongside Attention Residuals) the company credits with a roughly 2.5x scaling-efficiency gain over Kimi K2. Hybrid linear attention generally combines sub-quadratic-cost approximations with selective full attention, which plausibly explains how K3 handles a 1-million-token context without quadratic compute blowup, though Moonshot has not published full mechanistic details.

Is Kimi K3 free to use?

Kimi K3’s weights are open and published on Moonshot’s model repository, so self-hosting carries no licensing fee beyond your own infrastructure cost, under a license reportedly continuing the Modified MIT lineage from Kimi K2 — verify the exact terms in the repository before commercial use. Moonshot’s managed API and apps are usage-billed rather than free, which is the more practical access path for most teams.

How does Kimi K3 compare to GPT-5.6 and Claude Fable 5?

On Artificial Analysis’s Intelligence Index v4.1, Kimi K3 scores 57.1, versus 58.9 for GPT-5.6 Sol Max and 59.9 for Claude Fable 5 — a narrow gap placing K3 third among tracked frontier-class models. K3 reportedly leads on LMArena’s Frontend Code leaderboard at 1679, ahead of both closed competitors on that specific coding benchmark, while GPT-5.6 and Claude Fable 5 remain ahead on the broader composite index.

What hardware do you need to self-host Kimi K3?

Kimi K3’s 2.8-trillion-parameter checkpoint requires a multi-GPU, likely multi-node cluster with high-bandwidth interconnects for expert-parallel serving, even after quantization to FP8 or INT4. Sparsity reduces active compute per token but not the memory footprint of the full checkpoint, since any expert can be routed to. This is not a single-workstation deployment; teams should budget for serious multi-GPU infrastructure or default to the managed API.

Further Reading

• Mixture-of-Experts (MoE) LLM Architecture, Explained
• DeepSeek-V4: Architecture and Benchmarks Deep-Dive
• Mistral Large 3 Explained: Architecture and Benchmarks
• OpenAI o3: Reasoning and Test-Time Compute Scaling
• Moonshot AI on GitHub — model repository and release notes
• Artificial Analysis — independent model benchmarking and the Intelligence Index
• LMArena — human-preference leaderboards including Frontend Code

By Riju — about