Kimi K3 Explained: Architecture, Benchmarks, and Deployment (2026)

On 16 July 2026, Moonshot AI shipped Kimi K3, and the headline number is hard to ignore: 2.8 trillion parameters, making it the largest open-source model released to date. But the number that should actually change your planning is the one underneath it — only 16 of 896 experts fire per token, so the model you serve is sparse, not dense. Understanding the Kimi K3 architecture is the difference between treating this as an unreachable 2.8T monolith and treating it as a routable, quantizable system you can reason about, price, and possibly self-host. K3 also debuted at #1 on the Arena Frontend-Code leaderboard, which is the sort of result that forces a re-benchmark rather than a footnote.

This is a durable reference page for one specific model. It is not a survey of “the best LLMs.” It walks the mechanism from router to expert to attention block, sets the reported benchmarks against real caveats, and gives you concrete access and deployment paths.

What this covers: the lineage from Kimi K2, the sparse MoE and Kimi Delta Attention internals, the training pipeline, the published benchmark suite with contamination caveats, pricing and self-hosting options, honest failure modes, and a decision matrix against peer models.

Context and Background

Open-weight large language models spent 2024 and 2025 closing the gap to closed frontier systems one release at a time. DeepSeek popularized fine-grained Mixture-of-Experts (MoE) at scale; Qwen pushed multilingual and coding depth; Moonshot AI’s own Kimi line built a reputation for long context and agentic tool use. By mid-2026 the open camp was no longer “cheaper but a generation behind.” It was contending for the top of specific leaderboards.

Kimi K3 lands squarely in that shift. Moonshot positions it as the scale-up successor to Kimi K2, and the third-party coverage has been unusually direct: VentureBeat framed it as the largest open-source model ever, “rivaling top U.S. systems,” and independent agentic benchmarks placed it inside the closed frontier’s own bracket rather than a tier below. That framing matters because it changes the buying question. The question is no longer “is open good enough?” It is “for which task families does an open 2.8T MoE beat a closed API on cost-adjusted quality?”

Two things make K3 a genuine architecture story rather than a parameter-count stunt. First, the sparsity ratio is aggressive: 16 active experts out of 896 is a very fine-grained routing scheme, which keeps the per-token compute far below the total parameter count would suggest. Second, Moonshot did not just scale a known recipe — it shipped two internal architectural changes, Kimi Delta Attention and Attention Residuals, and claims measurable scaling gains from each. When a lab publishes named mechanisms alongside a release, the release is worth reading closely.

If you are new to how these systems are assembled, our companion deep-dive on the Kimi K2 architecture and benchmarks is the right prerequisite, since K3 inherits much of K2’s serving philosophy and agentic posture. For the broader open-MoE context, the VentureBeat launch report is a good primary-adjacent source.

Lineage and Context: From Kimi K2 to K3

Kimi K3 is the next entry in a family, and reading it as a point release undersells the jump. Kimi K2 was a roughly 1T-class MoE — large for its moment, tuned hard for coding and tool-calling, and widely deployed through Moonshot’s API and OpenRouter. K3 is a near-3x scale-up in total parameters, but the more interesting delta is not the size. It is what Moonshot changed to make the size pay off.

The kimi k3 vs kimi k2 comparison breaks down along three axes. On raw scale, K3’s 2.8T total dwarfs K2’s ~1T, and the expert count climbs to 896 with a top-16 routing scheme. On attention, K2 used a more conventional attention stack, while K3 introduces Kimi Delta Attention, a hybrid linear-attention mechanism designed specifically so that decoding does not collapse in cost as context grows toward a million tokens. On post-training, K3 ships with an always-on reasoning mode — Moonshot calls it “thinking mode” — rather than treating reasoning as an optional, separately-priced variant.

The practical upshot is that K3 is not “K2 but bigger.” It is a redesign aimed at a specific failure mode of large models: quadratic attention cost and memory pressure at long context. K2 could handle long inputs, but the economics degraded. K3’s headline claim is that Kimi Delta Attention delivers up to 6.3x faster decoding in million-token contexts, which is the kind of number that only matters once you actually run agents that read entire repositories or multi-hundred-page documents in a single call.

There is a lineage lesson worth stating plainly. Each Kimi generation has leaned harder into agentic and coding workloads, and K3 continues that. If your workload is short-prompt chat, most of what is special about K3 is dormant. If your workload is long-horizon, tool-heavy, repository-scale work, K3 is built for exactly your pain. Readers evaluating the broader open-MoE field should also weigh the DeepSeek V4 architecture deep-dive, since DeepSeek’s fine-grained MoE lineage is the most direct architectural cousin to what Moonshot is doing here.

The Kimi K3 Architecture: Sparse MoE and Kimi Delta Attention

Kimi K3 is a sparse Mixture-of-Experts transformer with 2.8 trillion total parameters, of which only 16 of 896 experts activate per token, paired with a hybrid linear attention mechanism called Kimi Delta Attention and a residual-connection replacement called Attention Residuals. That single sentence is the model. The rest of this section unpacks why each choice is there.

Figure 1: The Kimi K3 architecture. Input tokens pass through Kimi Delta Attention and Attention Residuals, then a router selects the top 16 of 896 experts, whose outputs are weighted and combined into the layer output.

Figure 1 traces one transformer block. Tokens enter the attention stage, where Kimi Delta Attention and Attention Residuals do their work, then reach a router that scores all 896 experts and dispatches the token to the 16 highest-scoring ones. Each selected expert is a feed-forward sub-network; their outputs are combined with router weights and passed on. The critical property is that 880 experts stay idle for any given token, so the compute per token is a small fraction of the 2.8T total.

Why 16 of 896 is the number that matters

The ratio of active to total parameters is the single most important operational fact about any MoE model. Total parameters set your memory footprint — you must hold all 896 experts in memory to serve the model. Active parameters set your compute cost per token — you only run 16. A fine-grained scheme like 16/896 gives the optimizer a large, specialized “vocabulary” of experts to route among, which tends to improve quality per unit of active compute, while keeping inference FLOPs modest.

This is the trick that makes a 2.8T model tractable. If K3 were dense, every token would touch all 2.8T parameters and inference would be economically absurd for most users. Because it is sparse, the per-token math is closer to a mid-size dense model, and the “cost” of the extra scale is memory and routing overhead rather than raw multiply-accumulates. That is why the honest description is “large to hold, moderate to run.”

Kimi Delta Attention: linear attention that survives a million tokens

Standard softmax attention is quadratic in sequence length: double the context and you roughly quadruple attention cost. At a 1M-token context window, that curve is brutal. Kimi Delta Attention (KDA) is Moonshot’s answer — a hybrid linear attention mechanism. Linear-attention variants replace the full pairwise softmax with a formulation whose cost grows closer to linearly with sequence length, trading some expressive precision for dramatically better scaling.

“Hybrid” is the operative word. Pure linear attention historically loses quality on tasks that need sharp, content-based lookups. A hybrid interleaves linear-attention layers with a smaller number of full-attention layers, so the model keeps precise recall where it needs it while paying linear-ish cost most of the time. Moonshot’s reported 6.3x decoding speedup at million-token contexts is consistent with that design: the win shows up exactly where the quadratic term would otherwise dominate. Treat the specific multiplier as a vendor-reported figure until third parties reproduce it.

Attention Residuals: a drop-in replacement for residual connections

The second named change is Attention Residuals, which Moonshot describes as a drop-in replacement for the standard residual connection that yields consistent scaling gains. Residual connections — the “add the input back to the output” skip paths that made deep networks trainable — are one of the most load-bearing tricks in the transformer. Replacing them is not a cosmetic tweak; it touches gradient flow through every layer. The claim of “consistent scaling gains” means the benefit grows with model size, which is precisely why it is worth doing at the 2.8T scale rather than a small model where the gain would be marginal.

Context window, modalities, and what stays unconfirmed

K3 ships with a 1,048,576-token (1M) context window and native visual understanding, so it is multimodal at the input rather than a text-only model bolted to a separate vision encoder after the fact. That native-vision claim is reported by Moonshot and echoed across launch coverage; treat the exact vision-tower details as unconfirmed until the technical report and weights land. The tokenizer and vocabulary specifics were not fully disclosed at launch — this is a case where the brief’s rule applies: name a number precisely or not at all, and here the precise tokenizer figures are simply not yet public.

Training: Pretraining, Post-Training, and the Always-On Reasoner

Moonshot has disclosed the shape of the training pipeline more than its exact quantities, so this section names the stages precisely and labels the magnitudes as reported or estimated rather than asserting numbers that were never published.

Figure 2: The reported Kimi K3 training pipeline. A large web, code, and vision corpus feeds next-token pretraining, followed by a long-context midtraining stage, supervised fine-tuning, reinforcement learning with verifiable rewards, and an always-on thinking mode.

Figure 2 shows the stages in order. The base is standard large-scale next-token pretraining over a mixed corpus of web text, code, and visual data — the multimodal capability implies vision data enters at or near pretraining rather than only at fine-tuning. A dedicated long-context stage then extends the effective window toward 1M tokens; models are almost never pretrained at full target length from scratch because it is wasteful, so a midtraining phase that grows the context is the standard and most likely approach here.

Post-training: SFT then reinforcement learning

After pretraining, K3 goes through supervised fine-tuning (SFT) on instruction and dialogue data, then reinforcement learning. The most credible read, given the benchmark profile and the industry direction in 2026, is reinforcement learning with verifiable rewards (RLVR) for the reasoning, coding, and agentic tasks — that is, rewarding the model when a unit test passes, a math answer checks out, or a tool call succeeds, rather than only from a learned preference model. This is the recipe that has driven the largest gains on agentic and coding benchmarks across the field, and K3’s Terminal-Bench and FrontierSWE numbers are consistent with heavy RLVR investment. Label this as an informed inference: Moonshot has not published the full reward design.

The always-on thinking mode

One deliberate product choice stands out. K3 runs an always-on reasoning mode — “thinking mode” — rather than exposing reasoning as a separate, more expensive model or an optional flag. Practically, that means the model spends inference tokens on internal deliberation before answering, which improves hard-reasoning accuracy but raises token consumption and latency per request. It is a defensible default for a model aimed at agentic and analytical work, but it has a cost you will see on your bill and in your p95 latency. If you are building latency-sensitive chat, budget for it or check whether a lighter configuration is exposed.

What compute was used

The training compute and total token budget were not disclosed in a way that supports a precise claim at launch. A 2.8T-total MoE trained to frontier quality implies a very large cluster and a multi-trillion-token corpus, but any specific FLOP or token count you see attributed to K3 right now should be treated as estimate, not fact, until the technical report is out. The full weights are scheduled for Hugging Face around 27 July 2026, and the accompanying documentation is the moment to replace these inferences with disclosed numbers.

Capabilities and Benchmarks

Kimi K3’s reported benchmark suite is strong across reasoning, coding, agentic, and browsing tasks, and the pattern is coherent: it is tuned for long-horizon, tool-using work rather than only short-prompt trivia. Here are the official and reported scores at launch.

Figure 3: Kimi K3 benchmarks positioned against Kimi K2 and closed frontier peers. K3’s reasoning and coding scores put an open-weight model into the frontier tier while retaining the open-weights and pricing advantage.

Benchmark	Task family	Kimi K3 (reported)
GPQA Diamond	Graduate-level science reasoning	93.5%
Terminal-Bench 2.1	Agentic terminal / tool use	88.3%
FrontierSWE	Software engineering	81.2%
DeepSWE	Software engineering	67.5%
Program Bench	Program synthesis	77.8%
BrowseComp	Web browsing / research	91.2%
MCP Atlas	Tool-use via MCP	84.2%
SWE Marathon	Long-horizon software tasks	42.0

Two independent agentic evaluations round out the picture. On Artificial Analysis’s GDPval-AA v2, which measures real-world tasks across 44 occupations and 9 industries, K3 reportedly scored 1,687 — third overall, behind Claude Fable 5 Max (1,815) and GPT-5.6 Sol Max (1,747.8), and ahead of Claude Opus 4.8 (1,600). On AA-Briefcase, a private long-horizon knowledge-work benchmark, K3 reportedly placed second at 1,527, beating GPT-5.6 Sol Max and trailing only Fable 5 Max. These are third-party numbers reported around launch; verify against the benchmark authors’ own boards before quoting them in a decision document.

Reading the pattern, not just the numbers

The scores tell a consistent story. GPQA Diamond at 93.5% is near the top of what any model reports on graduate-level science reasoning. Terminal-Bench 2.1 at 88.3% and MCP Atlas at 84.2% signal that the agentic and tool-use training paid off. BrowseComp at 91.2% is a strong research-and-retrieval result, which pairs naturally with the 1M context. The software-engineering trio — FrontierSWE 81.2%, DeepSWE 67.5%, Program Bench 77.8% — is high but not saturated, and SWE Marathon at 42.0 is the honest reminder that long-horizon, multi-step engineering is still hard for every model, K3 included.

Contamination and methodology caveats

Benchmark numbers deserve skepticism, and three caveats apply specifically here. First, contamination: any benchmark old enough to appear in training data can be inflated, and vendor-run evaluations are self-reported. Second, harness sensitivity: agentic scores like Terminal-Bench and SWE-family results depend heavily on the scaffolding, retry budget, and tool wiring around the model, so a number is only comparable within the same harness. Third, the thinking-mode confound: an always-on reasoner will score higher on hard benchmarks partly because it spends more inference compute, so a head-to-head against a non-reasoning peer is not apples-to-apples. None of this makes the scores fake. It means you should reproduce the two or three benchmarks that map to your workload rather than trusting the aggregate.

Access and Deployment

Kimi K3 is available through Moonshot’s own API and via OpenRouter today, with full open weights scheduled for Hugging Face around 27 July 2026, so your deployment options span hosted API, aggregator, and eventual self-host. The path you pick is mostly a function of scale, data-residency needs, and how much GPU you are willing to operate.

Figure 4: Kimi K3 access paths. Hosted API and OpenRouter are token-metered and available now; open weights on Hugging Face enable self-hosting on multi-node GPU with INT4 or FP8 quantization for those who need control or residency.

Pricing on the hosted paths

At launch, kimi k3 pricing on OpenRouter is reported at roughly $3 per million input tokens and $15 per million output tokens, with the Moonshot API offering a cache-hit rate near $0.30 per million tokens for reused prompt prefixes. Confirm live rates on the provider pages before you build a budget — inference pricing moves fast and promotional rates are common in a model’s first weeks. The framing that holds across coverage is that K3 is priced substantially below closed frontier peers of comparable quality, which is the entire commercial argument for an open-weight frontier model: similar output quality at a fraction of the token cost, plus an exit to self-hosting that closed APIs cannot offer.

Prompt caching is worth engineering around. The gap between the $3 standard input rate and the ~$0.30 cache-hit rate is a 10x saving on the reused portion of a prompt, which for agentic workloads that resend a large system prompt or repository context on every step is not a rounding error — it can dominate your bill. Structure prompts so the stable prefix is identical across calls and let the cache do its job.

Self-hosting the open weights

Once the weights land, self-hosting is real but not casual. A 2.8T-total MoE must fit all 896 experts in memory even though only 16 run per token, so the memory bill is set by total parameters, not active ones. In practice that means a multi-node GPU deployment with expert-parallel and tensor-parallel sharding across a cluster, served by a stack such as vLLM or SGLang once K3 support lands. Quantization is the lever that makes it affordable: FP8 roughly halves memory versus 16-bit, and INT4 (via schemes like AWQ or GPTQ) roughly quarters it, at some quality cost. Expect the community to publish quantized checkpoints within days of the weight release — that is the pattern for every major open model — and expect INT4 variants to be the ones most people actually run.

License is the open question. Moonshot has not published the exact license text at launch; the K2 family shipped under a Modified MIT license, so an open, permissive license in that class is the reasonable expectation for K3 — but treat the specific license name as “expected open, to be confirmed at weight release” rather than a fact. If your use is commercial, read the actual license the day it appears; a “Modified MIT” can carry attribution or usage clauses that a plain MIT does not.

A pragmatic decision rule

Use the hosted API or OpenRouter for prototyping, spiky traffic, and anything below the volume where a dedicated cluster amortizes. Move to self-hosted quantized weights when you have sustained high volume, hard data-residency requirements, or a need to fine-tune. The 1M context and agentic strengths are available on every path; only the cost curve and the control differ.

Trade-offs, Gotchas, and What Goes Wrong

No model is free of sharp edges, and K3’s are specific enough to plan around. The honest limitations matter more than the launch benchmarks because they are what you will actually hit in production.

The first gotcha is the memory-versus-compute asymmetry. Teams read “only 16 experts active” and assume K3 is cheap to host. It is cheap to run per token and expensive to hold in memory — the full 896-expert weight set must be resident. If you are sizing a cluster from active parameters, you will under-provision by a large margin and OOM on load.

The second is context degradation. A 1M-token window is the maximum, not a promise of uniform quality across it. Hybrid linear attention buys you cost savings, but linear-attention layers can lose precise recall on needle-in-a-haystack retrieval deep in a very long context. Test recall at your actual context lengths; do not assume that because 1M fits, 1M works equally well everywhere in the window.

The third is the thinking-mode tax. Always-on reasoning raises both latency and token spend. For analytical and agentic work that is a fair trade; for high-QPS, latency-sensitive chat it is a liability. Measure p95 latency and output-token counts on your traffic before committing, because the benchmark-winning configuration is not necessarily the one you want in a chat box.

The fourth is maturity risk. K3 is days old at the time of writing. Weights are not fully posted yet, the technical report with disclosed training details is pending, serving-stack support for the exact architecture is still stabilizing, and independent reproductions of the headline numbers are thin. That is normal for a launch, but it means anything you read today — including parts of this page labeled “reported” — should be re-verified against the primary documentation when it lands around 27 July.

The fifth is the standard open-weight safety posture. Open weights mean anyone can remove guardrails via fine-tuning, and native vision widens the input surface for prompt-injection and jailbreak attempts embedded in images. If you deploy K3 in an agent with tool access, treat its outputs as untrusted and gate real-world actions behind validation, exactly as you would for any capable model.

Practical Recommendations

Treat Kimi K3 as a long-context, agentic workhorse first and a general chat model second — that framing will steer most of your decisions correctly. If your workload is repository-scale coding, multi-document research, or tool-using agents, K3 is squarely in its element and worth benchmarking against your incumbent. If your workload is short, latency-sensitive chat, the always-on reasoner and its token cost may make a smaller model the better fit.

Start on the hosted API or OpenRouter to validate quality on your own tasks before committing engineering to self-hosting. Reproduce the two or three benchmarks that actually map to your use case rather than trusting the aggregate scoreboard. Engineer prompt caching from day one to capture the roughly 10x saving on reused prefixes. And wait for the 27 July weight release before finalizing any self-host architecture, since the license, exact config, and serving-stack support will all be clearer then.

A short pre-adoption checklist:

• Confirm live pricing on Moonshot and OpenRouter before you budget.
• Load-test memory sizing from total parameters, not active experts.
• Run a needle-in-a-haystack recall test at your real context length.
• Measure p95 latency and output-token counts with thinking mode on.
• Read the actual license the day the weights ship if your use is commercial.
• Gate any tool-using agent’s real-world actions behind validation.

Frequently Asked Questions

What is Kimi K3 and who built it?

Kimi K3 is a 2.8-trillion-parameter open-weight Mixture-of-Experts language model released by Moonshot AI on 16 July 2026. It activates 16 of 896 experts per token, uses a hybrid linear attention mechanism called Kimi Delta Attention, offers a 1-million-token context window, and includes native vision. At launch it was the largest open-source model to date and debuted at #1 on the Arena Frontend-Code leaderboard, positioning open weights against closed frontier systems.

How is Kimi K3 different from Kimi K2?

K3 is a near-3x scale-up of K2, moving from a roughly 1T-class MoE to 2.8T total parameters with 896 experts. Beyond size, K3 adds Kimi Delta Attention for cheaper long-context decoding, Attention Residuals as a residual-connection replacement, and an always-on thinking mode instead of a separate reasoning variant. In short, K3 is a redesign targeting long-context and agentic economics, not just a larger K2.

What are Kimi K3’s benchmark scores?

Reported scores include GPQA Diamond 93.5%, Terminal-Bench 2.1 88.3%, FrontierSWE 81.2%, DeepSWE 67.5%, Program Bench 77.8%, BrowseComp 91.2%, MCP Atlas 84.2%, and SWE Marathon 42.0. Independent agentic evaluations reportedly placed it third on GDPval-AA v2 (1,687) and second on AA-Briefcase (1,527). These are vendor and third-party figures reported at launch; reproduce the ones relevant to your workload before relying on them.

How much does Kimi K3 cost to use?

On OpenRouter, Kimi K3 is reported at roughly $3 per million input tokens and $15 per million output tokens, with a Moonshot API cache-hit rate near $0.30 per million tokens for reused prompt prefixes. Confirm current rates on the provider pages, since early pricing changes often. The consistent message across coverage is that K3 undercuts closed frontier peers of similar quality, which is the core commercial case for an open-weight frontier model.

Can I self-host Kimi K3, and what hardware do I need?

Yes, once full weights reach Hugging Face around 27 July 2026. Because it is a 2.8T-total MoE, memory is set by total parameters — all 896 experts must be resident — so expect a multi-node GPU deployment with expert- and tensor-parallel sharding on a stack like vLLM or SGLang. Quantization is essential: FP8 roughly halves memory and INT4 roughly quarters it. Community-quantized INT4 checkpoints are the version most teams will actually run.

What license will Kimi K3 use?

Moonshot had not published the exact license text at launch. The predecessor K2 family shipped under a Modified MIT license, so a permissive, open license in that class is the reasonable expectation — but treat the specific license as unconfirmed until the weights and documentation appear around 27 July 2026. If your deployment is commercial, read the actual license terms on release day rather than assuming plain MIT.

Further Reading

• Kimi K2 explained: architecture and benchmarks — the predecessor in this series and the best primer on Moonshot’s serving and agentic philosophy.
• DeepSeek V4 architecture and benchmarks deep-dive — the closest architectural cousin, and a useful contrast for fine-grained open MoE design.
• Qwen 3.6 explained: architecture and benchmarks — another leading open-weight family to weigh against K3 for coding and multilingual work.
• VentureBeat: China’s Moonshot AI releases Kimi K3 — launch reporting with the scale and positioning claims.
• MarkTechPost: Moonshot AI releases Kimi K3 with Kimi Delta Attention and 1M context — a technically detailed overview of the named architectural changes.

By Riju — about