WebGPU LLM Inference: WebLLM, WGSL Kernels, VRAM Limits and When It Beats WASM (2026)

A plant-floor kiosk, a technician’s tablet and an operations dashboard all share one problem: they sit next to the data, yet any language model they use lives in a cloud region that may be unreachable, slow or forbidden by policy. For years the browser could not help, because JavaScript and WebAssembly (WASM) on the CPU decode a billion-parameter model at a pace nobody enjoys. WebGPU LLM inference changes the arithmetic: the same tab can now dispatch compute shaders to the machine’s graphics processor, keep the weights in GPU memory and stream tokens without a network round trip.

It matters now because the pieces have matured together. WebLLM compiles models into WebGPU kernels, Transformers.js and ONNX Runtime Web expose a WebGPU backend, and 4-bit weights with 16-bit activations make small models fit inside browser memory limits. The catch is that the browser imposes buffer ceilings, a sandbox and uneven platform support that a native runtime never meets.

You will leave with a mental model of the pipeline from WGSL to tokens, worked VRAM and bandwidth arithmetic, a runnable JavaScript snippet, and a decision rule for choosing WebGPU or WASM.

What this covers: the WebGPU compute model, WGSL kernels, how WebLLM, Transformers.js and ONNX Runtime Web differ, quantization and memory limits, caching, a WASM comparison, browser support, failure modes and an IoT deployment checklist.

Context and Background

Running a language model is dominated by two operations: large matrix-vector or matrix-matrix products, and attention over a growing key-value (KV) cache. Both are massively parallel and both are limited, during token-by-token decoding, by how fast weights can be streamed out of memory rather than by arithmetic. That is why a GPU, with an order of magnitude more memory bandwidth than a laptop CPU, is the natural target, and why the browser needed a way to reach it.

WebGL, the earlier graphics API, could be bent into doing general computation by encoding data in textures, but it had no compute shaders, no shared memory between threads in a workgroup and no storage buffers. WebGPU, the W3C successor, was designed from the start around explicit pipelines, bind groups, storage buffers and compute passes, modelled on the modern native APIs (Vulkan, Metal and Direct3D 12). Its shading language is WGSL, the WebGPU Shading Language. For ML this is the decisive difference: a matrix multiply can finally be written as a real compute kernel with workgroup memory instead of a disguised full-screen triangle.

On the CPU side, WASM with SIMD and threads is the incumbent. It runs everywhere, needs no GPU, and tools such as llama.cpp compiled to WASM and ONNX Runtime Web’s WASM execution provider make it a reasonable baseline. Its weaknesses are the 32-bit address space (a single WASM memory is limited to 4 GiB unless the newer 64-bit memory proposal is available), thread support that depends on cross-origin isolation, and raw throughput that is bounded by the CPU’s memory bandwidth and core count.

The research anchor for the GPU path is the WebLLM paper, which describes a high-performance in-browser inference engine built on Apache TVM and MLC-LLM, and which reports retaining up to about 80% of native performance on the same device (WebLLM paper, arXiv 2412.15803). We will unpack exactly what that number does and does not claim later. For the neighbouring native story, see our measurements on edge LLM benchmarks across Jetson Orin, Llama, Phi and Gemma, and for the CPU-side alternative see WebAssembly versus containers for edge functions.

A final piece of context: the browser is a deployment target with unusual properties. There is no installer, the model is fetched over HTTPS and cached by the origin, the user’s own GPU does the work, and the page can run fully offline once cached. For IoT dashboards and kiosks that combination is attractive, but it also means you inherit the user’s hardware lottery and the browser vendor’s limits.

Reference Architecture for WebGPU LLM Inference

Direct answer: WebGPU LLM inference compiles a model into compute-shader kernels (written in WGSL), uploads quantized weights into GPU storage buffers once, then runs one prefill pass over the prompt and repeated single-token decode passes that read a growing KV cache. A web worker hosts the engine so the UI thread stays responsive, and the Cache API or OPFS stores weight shards.

Figure 1: The in-browser stack. The page talks to a worker engine, which drives WebGPU compute passes over weights and KV cache held in GPU buffers, fed by a persistent shard cache.

The diagram separates four responsibilities. The page owns user interaction and sends OpenAI-style chat requests. The worker owns the engine, tokenizer and scheduler, so a long prefill never freezes scrolling. The GPU owns weights and the KV cache in storage buffers, plus the compiled pipelines. Persistent storage owns the downloaded weight shards so the second visit does not repeat a multi-hundred-megabyte download. Each boundary is a place where something can fail, which is why the later sections treat them one at a time.

The WebGPU compute model in one page

A WebGPU program starts by requesting an adapter (a handle on a physical GPU) and then a device (a logical connection with chosen features and limits). Work is expressed as a compute pipeline built from a WGSL shader module. Data lives in GPUBuffer objects, bound to the shader through bind groups. A command encoder records compute passes, each of which dispatches a grid of workgroups; the queue then submits the encoded commands. Nothing runs until submission, and reading results back to JavaScript requires mapping a staging buffer asynchronously.

Three consequences shape LLM design. First, dispatch and submission have fixed overhead, so a model with hundreds of small kernels per token needs those kernels fused or batched into few passes. Second, readback is slow and asynchronous, so the engine reads back only the sampled token (or a small logits slice), never intermediate activations. Third, the API validates everything, so shape and limit mistakes surface as validation errors rather than silent corruption.

The default limits matter enormously. According to MDN, the documented defaults include a maximum buffer size of 268,435,456 bytes (256 MiB), a maximum storage buffer binding size of 134,217,728 bytes (128 MiB), 8 storage buffers per shader stage, 16,384 bytes of workgroup storage, 256 invocations per workgroup and 65,535 workgroups per dimension (MDN, GPUSupportedLimits). Those are the floor every compliant device must meet, not what a given GPU reports; an adapter usually offers more, and an application must request a higher limit explicitly when creating the device. An engine that ignores this will try to bind a 600 MiB embedding table in a 128 MiB window and fail.

WGSL kernels: what the model really needs

An LLM forward pass reduces to a short list of kernel families. Dequantize-and-matmul handles the linear layers (Q, K, V and output projections, plus the feed-forward gate, up and down projections). Attention computes scores against the KV cache and applies softmax. RMSNorm (or LayerNorm) normalizes activations. Rotary position embedding (RoPE) rotates query and key vectors. Sampling picks the next token from logits. The matmul family dominates runtime and memory traffic.

During decode there is a single new token, so each linear layer is a matrix-vector product. The kernel is bandwidth bound: every weight is read once and used for only a couple of operations. The efficient design assigns each workgroup a slice of output rows, has threads cooperatively load packed 4-bit weights, unpack and scale them in registers, multiply by an activation vector staged in workgroup memory, and finish with a tree reduction. During prefill, the prompt supplies many tokens, so the same layers become matrix-matrix products, which are compute bound and benefit from tiling into workgroup memory.

Here is a deliberately small WGSL kernel that illustrates the decode-time pattern for an f16 matrix-vector product with a workgroup reduction. It is a teaching sketch, not production code, and it was not run against a real device in this article.

// Illustrative only: y = W * x for an f16 weight matrix, one row per workgroup.
enable f16;

struct Dims { rows: u32, cols: u32 };

@group(0) @binding(0) var<storage, read> w: array<f16>;
@group(0) @binding(1) var<storage, read> x: array<f16>;
@group(0) @binding(2) var<storage, read_write> y: array<f16>;
@group(0) @binding(3) var<uniform> dims: Dims;

var<workgroup> partial: array<f32, 128>;

@compute @workgroup_size(128)
fn main(@builtin(workgroup_id) wg: vec3<u32>,
        @builtin(local_invocation_id) lid: vec3<u32>) {
  let row = wg.x;
  var acc: f32 = 0.0;
  var c = lid.x;
  loop {
    if (c >= dims.cols) { break; }
    acc += f32(w[row * dims.cols + c]) * f32(x[c]);
    c += 128u;
  }
  partial[lid.x] = acc;
  workgroupBarrier();
  var stride = 64u;
  loop {
    if (stride == 0u) { break; }
    if (lid.x < stride) { partial[lid.x] += partial[lid.x + stride]; }
    workgroupBarrier();
    stride = stride >> 1u;
  }
  if (lid.x == 0u) { y[row] = f16(partial[0]); }
}

Note the enable f16; directive: half-precision arithmetic in WGSL is an optional feature that the device must expose as shader-f16. Accumulation in f32 even with f16 storage is a standard guard against precision loss across thousands of terms. Real engines go further: they use packed integer storage for 4-bit weights, read 32 bits at a time, and (where supported) use subgroup operations to replace some of the shared-memory reduction. The WebLLM paper itself lists newer WebGPU features such as shuffling as untapped optimization headroom.

Why a compiler, not hand-written shaders

Hand-writing every kernel for every model and every GPU is untenable. WebLLM therefore does not ship a library of fixed shaders. The paper explains that MLC-LLM and Apache TVM compile the model computation into WebGPU kernels ahead of time, because WebGPU lacks accelerated kernel libraries comparable to those native CUDA developers get for attention and matrix products. WASM, compiled with Emscripten, covers CPU-side duties: a grammar engine for structured generation, paged KV-cache sequence management and tensor manipulation for launching kernels. The result is a model library (a compiled artifact containing WGSL and WASM) plus a set of weight shards, both fetched from a URL.

That ahead-of-time split has a practical effect for you. Supporting a new model architecture means someone has to compile it into the MLC format, so the catalogue of prebuilt WebLLM models is narrower than the full Hugging Face universe. Transformers.js and ONNX Runtime Web take the opposite approach: they interpret an ONNX graph and map each operator onto a prebuilt WebGPU kernel, which covers many more architectures at the cost of less aggressive fusion.

Three Runtimes, One GPU: WebLLM, Transformers.js and ONNX Runtime Web

Most teams will meet WebGPU LLM inference through one of three libraries. They share the same hardware path but make different bets about compilation, model coverage and API shape, so the choice is an architectural decision rather than a preference.

Figure 2: Two ways to reach WebGPU. WebLLM compiles a model ahead of time into fused kernels; Transformers.js sits on ONNX Runtime Web, which maps graph operators to prebuilt WebGPU kernels.

The figure shows the two lineages. On the left, TVM-compiled artifacts are loaded directly by the WebLLM engine. On the right, Transformers.js hands an ONNX graph to ONNX Runtime Web, whose WebGPU execution provider dispatches operator kernels. Both end at the same browser WebGPU implementation and the same driver.

WebLLM and MLC-LLM

WebLLM exposes an OpenAI-compatible API in the browser: you create an engine for a model identifier, then call chat.completions.create with messages, optionally streaming. Per the paper, a frontend engine in the page communicates by message passing with a backend engine running in a web worker, so decoding does not block the UI. It also supports structured generation constrained by JSON Schema or a context-free grammar through the XGrammar library, which matters for dashboards that need machine-readable output rather than prose.

The measured claim deserves precise reading. The paper’s Table 1 compares WebLLM version 0.2.75 against native MLC-LLM on an Apple MacBook Pro with an M3 Max chip, running WebLLM in Chrome Canary 133. For Llama-3.1-8B it reports 41.1 tokens per second for WebLLM against 57.7 native, a retained fraction of 71.2%. For Phi-3.5-mini (3.8B) it reports 71.1 against 89.3, or 79.6%. The headline “up to 80%” is therefore the rounded best case from two models on one machine, compared against one native runtime. It is encouraging evidence that the browser penalty is bounded, not a universal guarantee. Treat these as vendor-adjacent, single-device figures, and measure your own target hardware.

Transformers.js with the WebGPU backend

Transformers.js v3 added WebGPU support by collaborating with ONNX Runtime Web. Enabling it is a one-option change: you pass device: "webgpu" when creating a pipeline. The Hugging Face announcement lists a dtype option that replaced the older boolean quantized flag, with choices including fp32, fp16, q8, int8, uint8, q4, bnb4 and q4f16, noting that availability depends on the model (Transformers.js v3 announcement). The same post describes 120 supported architectures and more than 1,200 pre-converted models, and advertises WebGPU as “up to 100x faster than WASM”. That multiplier is a best-case marketing figure with no methodology on the page; it likely reflects small encoder models where the CPU path is slow. Do not expect two orders of magnitude for a decoder-only chat model.

The practical strength of this route is breadth: embeddings, speech recognition, vision and small language models all share one API, which is valuable when a single dashboard needs transcription, retrieval and generation together.

ONNX Runtime Web’s WebGPU execution provider

If you want control, use ONNX Runtime Web directly. Per its documentation you import from onnxruntime-web/webgpu (or load ort.webgpu.min.js from a script tag) and request the provider explicitly with executionProviders: ['webgpu'] in the session options. The documentation notes availability by default in recent Chrome and Edge on Windows, macOS, Android and ChromeOS, behind a flag in Firefox and in Safari Technology Preview at the time of the page, and points to a live status tracker (ONNX Runtime WebGPU EP docs). It also warns that for very lightweight models the default WASM provider can be preferable because of its smaller binary size, that graph capture requires static shapes with every kernel on WebGPU, and that GPU-resident tensors must be disposed manually or memory leaks. These caveats are exactly the ones LLM code hits: dynamic sequence lengths defeat graph capture unless the engine pads or buckets shapes.

Choosing among them

A rule of thumb: pick WebLLM when you want the highest decode speed on a supported chat model and an OpenAI-shaped API with grammar-constrained output; pick Transformers.js when you need many task types or a model that has an ONNX export but no MLC build; pick raw ONNX Runtime Web when you must manage buffers, IO binding and sessions yourself, for instance to keep activations on the GPU between models.

Quantization, VRAM Arithmetic and Where the Limits Bite

Memory, not compute, decides whether a model runs at all in a tab. Three budgets compete for the same GPU memory: weights, KV cache and scratch activations. Two browser-specific ceilings (buffer size and the browser’s own allocation policy) sit on top of the physical VRAM.

Figure 3: The memory budget. Weights are fixed by parameter count and bit width; the KV cache grows linearly with context; every single buffer must also fit under the device’s maxBufferSize.

What q4f16 means

The WebLLM model catalogue uses names such as q4f16_1: 4-bit quantized weights (q4) with 16-bit floating-point activations and scales (f16). Group quantization stores a small scale per group of weights. As a worked, illustrative example, suppose a group size of 32 with one 16-bit scale per group (the exact group size and zero-point handling vary by format, so this is arithmetic, not a spec): each weight costs 4 bits plus 16/32 = 0.5 bits of scale, or 4.5 bits, i.e. 0.5625 bytes.

Apply that to an 8-billion-parameter model: 8e9 x 0.5625 bytes is about 4.5 GB for weights. A 1.2-billion-parameter model comes to roughly 0.7 GB. Those figures are why 1B to 3B models are the comfortable range for a general consumer machine and 7B to 8B models need a discrete or unified-memory GPU with several gigabytes free. Embedding and output tables are often kept at higher precision, so real files run somewhat larger.

The KV cache, with real architecture numbers

The KV cache stores keys and values for every layer and every past token. Its size per token is 2 (K and V) x layers x KV heads x head dimension x bytes per element. For Llama-3.1-8B, which uses 32 layers, 8 key-value heads (grouped-query attention) and a head dimension of 128, in f16 that is 2 x 32 x 8 x 128 x 2 = 131,072 bytes, or 128 KiB per token. A 4,096-token context therefore needs 512 MiB; a 32,768-token context needs 4 GiB, as large as the weights. For a 1B-class model with 16 layers, 8 KV heads and head dimension 64, the figure is 32 KiB per token, so 4,096 tokens cost 128 MiB. Grouped-query attention is the reason these numbers are tolerable; with full multi-head attention the cache would be several times larger.

WebLLM manages the cache in pages, which the paper describes as paged KV cache sequence management in the WASM layer, so the context window is a configurable cap rather than an automatic maximum. Choosing a smaller window is the cheapest way to avoid out-of-memory failures.

Why single-buffer limits matter

Even when total VRAM is ample, WebGPU binds buffers individually. The default maximum storage binding is 128 MiB and the default maximum buffer is 256 MiB. A 128,000-entry vocabulary embedding with a 4,096 hidden size at 4 bits is about 128000 x 4096 x 0.5 = 262 MB, which already exceeds the default buffer size, and at f16 it would be about 1.05 GB. Engines solve this in two ways: they request raised limits from the adapter when creating the device, and they shard tensors into several buffers that the shader indexes across. Both have costs. Requesting limits fails on weaker adapters, and sharding adds indexing logic to every kernel.

The important rule for developers is to read adapter.limits and adapter.features at startup, choose a model that fits the reported values, and fail with a clear message rather than a cryptic device-lost error.

Bandwidth arithmetic and a decode-speed ceiling

Decode speed has a simple upper bound: tokens per second cannot exceed memory bandwidth divided by bytes read per token, because each token reads every weight once. As an illustrative calculation (not a measurement), a GPU with 400 GB/s of effective bandwidth decoding a 4.5 GB model has a ceiling near 89 tokens per second; a 0.7 GB model could in principle approach 570. Compare the WebLLM paper’s 41.1 tokens per second for the 8B model: well under the ceiling, as expected once KV reads, kernel launch overhead and imperfect bandwidth utilisation are counted. The value of the exercise is relative: halving the bit width nearly doubles the ceiling, and shrinking the model beats any shader micro-optimisation.

Loading, Caching and Running a Model: The Developer Walk-through

The lifecycle of a browser model has a first-visit cost and a repeat-visit cost, and they are very different. Understanding the sequence prevents the two most common complaints: “it takes forever the first time” and “it re-downloads every time”.

Figure 4: Model lifecycle. The first visit pays download and shader compilation; later visits read shards from persistent storage, upload to the GPU and start streaming tokens.

Cache API versus the Origin Private File System

Two browser storage mechanisms hold weight shards. The Cache API stores request and response pairs and is what WebLLM-style engines use by default to avoid repeating network fetches. The Origin Private File System (OPFS) offers a private, high-performance file store with synchronous access handles inside workers; some runtimes prefer it for large binary blobs. Both are subject to the browser’s storage quota for the origin, which is a fraction of free disk space and is not guaranteed. Persistent storage can be requested with navigator.storage.persist(), and a kiosk should do so, otherwise the browser may evict a 1 to 4 GB model under storage pressure and trigger a silent re-download.

Sharding helps in three ways. Downloads can be resumed per shard; each shard stays below per-buffer and per-response limits; and parallel fetches saturate the link. The trade-off is more files to verify. For regulated deployments, pin shard hashes and serve from your own origin with immutable cache headers, not from a public hub at runtime.

A runnable WebLLM snippet

The following is a minimal, plausible example based on WebLLM’s documented OpenAI-style interface. It has not been executed for this article, and model identifiers change between releases, so check the current catalogue (webllm.prebuiltAppConfig.model_list) before copying.

// Untested sketch. Requires: npm i @mlc-ai/web-llm, and a WebGPU-capable browser.
import * as webllm from "@mlc-ai/web-llm";

async function main() {
  if (!("gpu" in navigator)) throw new Error("WebGPU not available");
  const adapter = await navigator.gpu.requestAdapter();
  if (!adapter) throw new Error("No GPU adapter");
  console.log("maxBufferSize", adapter.limits.maxBufferSize,
              "shader-f16", adapter.features.has("shader-f16"));

  if (navigator.storage?.persist) await navigator.storage.persist();

  // Pick a small model id from the prebuilt list (name is illustrative).
  const modelId = "Llama-3.2-1B-Instruct-q4f16_1-MLC";
  const engine = await webllm.CreateMLCEngine(modelId, {
    initProgressCallback: (p) => console.log(p.text),
  });

  const stream = await engine.chat.completions.create({
    messages: [
      { role: "system", content: "You summarise machine alarms in one sentence." },
      { role: "user", content: "Pump P-104 vibration 9.2 mm/s, bearing temp 91 C." },
    ],
    stream: true,
    temperature: 0.2,
    max_tokens: 64,
  });

  let out = "";
  for await (const chunk of stream) out += chunk.choices[0]?.delta?.content ?? "";
  console.log(out);
}
main().catch(console.error);

For production, run the engine in a worker through WebLLM’s worker-based engine so the main thread never blocks, and bind a status indicator to the progress callback; first-load progress is the single biggest user-experience factor.

The same idea with Transformers.js

For a task such as embedding sensor-maintenance notes for local retrieval, Transformers.js needs only a few lines. Again this is an untested sketch following the documented option names.

import { pipeline } from "@huggingface/transformers";

const embed = await pipeline(
  "feature-extraction",
  "mixedbread-ai/mxbai-embed-xsmall-v1",
  { device: "webgpu", dtype: "fp16" } // dtype availability depends on the model
);
const vecs = await embed(["bearing wear", "seal leak"], { pooling: "mean", normalize: true });
console.log(vecs.dims);

If fp16 is not published for a model, fall back to q8 or fp32, and always wrap creation in a try/catch that retries with device: "wasm".

Shader compilation and first-token latency

The first run on a new device pays for compiling pipelines: the browser translates WGSL to the platform’s native shading language and the driver compiles that. This can take seconds for a full model. Browsers and drivers may cache compiled pipelines, but you cannot count on it. Two mitigations apply: warm up with a short dummy generation behind a loading screen, and avoid creating pipelines lazily in the middle of an interactive session.

Latency has two components worth separating. Time to first token is dominated by prefill, which is compute bound and scales with prompt length. Inter-token latency is dominated by decode, which is bandwidth bound. A prompt of 1,500 tokens with a 1B model feels fast to prefill; the same prompt on an 8B model on an integrated GPU can stall for seconds. For dashboards, keep system prompts short and push static context into retrieval rather than into every request.

When WebGPU Beats WASM, and When It Does Not

The comparison is not simply “GPU fast, CPU slow”. Each backend has a regime where it wins, and the boundary depends on model size, batch shape, hardware and deployment constraints.

Why the GPU wins on large decoders

Decode is bandwidth bound. A typical laptop CPU has a fraction of the memory bandwidth of its own integrated or discrete GPU, and WASM adds restrictions: SIMD is limited to 128-bit vectors, threads need cross-origin isolation (the page must be served with specific COOP and COEP headers), and the engine must live inside a 4 GiB 32-bit address space unless 64-bit memory is available. Prefill is compute bound and shows the largest gap, because thousands of prompt tokens become dense matrix products that map cleanly to thousands of GPU lanes. Anything beyond roughly 1 to 3 billion parameters is where a WASM path becomes uncomfortable on typical hardware, and an 8B model is effectively a GPU-only proposition in a tab.

Why WASM still wins in some places

There are real cases where the CPU path is the better engineering choice:

Tiny models and encoders. For a small classifier or an embedding model of a few tens of megabytes, the GPU path pays device initialisation, pipeline compilation and buffer-copy overhead that can exceed the compute savings. ONNX Runtime’s own documentation hints at this by recommending the WASM provider for very lightweight models.
Unsupported or blocklisted GPUs. Browsers maintain GPU blocklists for known-buggy drivers; a fleet of old industrial PCs or thin clients may expose no adapter at all.
Determinism and auditability. CPU floating-point results are easier to reproduce across machines. GPU results can differ by driver, a point that matters if you log outputs for compliance.
Contention. If the same GPU drives a trend chart, a 3D digital-twin view and the LLM, they compete. A CPU-bound model can leave the graphics pipeline smooth.
Memory pressure at the edge. A kiosk with an integrated GPU shares system RAM with the OS; a WASM model confined to a fixed heap can be more predictable than a GPU allocation the browser may reclaim.

A decision matrix

The table gives a qualitative guide. The tiers are judgements from the mechanisms above, not benchmark results.

Scenario	WebGPU path	WASM SIMD path	Better choice
1B to 3B chat model, modern laptop or tablet	Interactive streaming likely	Usable but slow prefill	WebGPU
7B to 8B chat model	Needs several GB free GPU memory	Impractical in a tab	WebGPU, or server
Small embedding or classifier (tens of MB)	Init overhead may dominate	Fast enough, smaller binary	WASM
Old industrial PC, no usable adapter	Unavailable	Works	WASM
Reproducible outputs for audit logs	Driver-dependent	More reproducible	WASM, or log inputs
Dashboard with heavy 3D rendering	GPU contention	Leaves GPU free	WASM, or smaller model
Structured JSON output at low latency	Grammar decoding on WASM CPU side plus GPU kernels	Slower	WebGPU (WebLLM)

A defensible production pattern is capability detection with graceful degradation: try WebGPU, check limits, and fall back to a smaller WASM model or a server endpoint when the check fails. The fallback tier should be tested as often as the happy path, since it is the path real fleets hit.

The 64-bit memory question

WASM’s 4 GiB ceiling has been the main obstacle for CPU inference of larger models. The WebAssembly memory64 proposal lifts it. From my recollection it has shipped in Chromium and Firefox in the past year or so, but I did not verify exact versions or Safari status for this article, so check the current WebAssembly feature roadmap before depending on it. Even with 64-bit addressing, a CPU decoding a multi-gigabyte model remains limited by memory bandwidth, so memory64 widens what fits and does little for tokens per second. It also does not give WebGPU’s buffer limits any relief; those are separate.

Browser Support, Privacy and Where This Fits in IoT

Support matrix: what is safe to assume

WebGPU support is the first deployment gate. The facts I could verify this run: ONNX Runtime’s documentation says WebGPU is on by default in recent Chrome and Edge on Windows, macOS, Android and ChromeOS, behind a flag in Firefox and in Safari Technology Preview as of its text. The Transformers.js v3 post reported global support of about 70% as of October 2024 according to caniuse, and named the flags dom.webgpu.enabled for Firefox, a WebGPU feature flag for Safari and enable-unsafe-webgpu for older Chromium builds. Those statements are dated and likely stale in places.

From memory and unverified for this run: Chrome shipped WebGPU on desktop platforms in version 113 in 2023 and extended it to Android later; Firefox began shipping on Windows in 2025; Safari added WebGPU in its version 26 releases. I did not confirm these version numbers, so treat them as leads and consult the live status tracker and caniuse before you publish a support promise. Linux desktop support in particular has historically lagged behind Windows, macOS and ChromeOS, which matters for engineering workstations.

The honest engineering position is that you cannot assume WebGPU on an arbitrary visitor. On a managed fleet you can: you control the browser version, flags and GPU driver, and you can test them. That is why this technology fits kiosks and internal dashboards far better than public marketing pages.

Privacy, offline use and data gravity

The strongest argument for browser inference is not speed but where data goes. Prompts, sensor excerpts and operator notes never leave the device, which simplifies data-residency questions and removes per-token cloud cost. After the model is cached, the page can work offline, subject to the origin keeping its storage. A service worker can cache the application shell so a kiosk boots even with the network down.

Two caveats stop this from being a blanket privacy win. The weights come from a network origin, so supply-chain integrity (pin and verify hashes, use Subresource Integrity for scripts, serve over HTTPS) is your responsibility. And local inference does not make logs private: if the page also sends telemetry or conversation history to a server, the benefit evaporates.

Fit for IoT dashboards and kiosks

Concrete uses where a 1B to 3B model in the tab is useful: summarising an alarm burst into a sentence for the operator; translating a maintenance note on the shop floor; turning a natural-language question into a structured query over local data using grammar-constrained JSON output; embedding and searching local manuals via an embedding model with local retrieval; and offline assistance for a field technician whose tablet loses connectivity in a plant basement.

Uses where it is a poor fit: deep multi-document reasoning, anything needing current knowledge, and safety-critical decisions. A small quantized model hallucinates, and an alarm summary must never be the sole input to a control action. Pair the model with deterministic rules and keep humans in the loop. For the related question of when logic should run at the edge in a different sandbox, our piece on the WebAssembly component model and wasmCloud edge architecture covers the CPU-side runtime design.

Trade-offs, Gotchas, and What Goes Wrong

Browser LLM deployments fail in a small number of repeatable ways. Knowing them in advance is worth more than another optimisation.

Device lost and out-of-memory errors. WebGPU can lose the device when the GPU is reset, a driver crashes or the browser reclaims resources. A model that fits comfortably on a test laptop can crash on a machine with a browser running many tabs. Always listen for the device’s lost promise, surface a retry button and design the engine so it can reinitialise without a page reload.

Limits mismatch. The most common cryptic failure is a buffer that exceeds the adapter’s maxBufferSize or maxStorageBufferBindingSize. Check limits before choosing a model, not after a failed download of two gigabytes. Treat the reported limits as hardware-and-browser specific; MDN itself notes that browsers may report tiered values rather than exact GPU limits.

Missing shader-f16. Some adapters do not expose half-precision shader arithmetic. A model built for q4f16 will not run there, and you need an f32-activation variant, which uses more memory and runs slower. Detect the feature, and ship both variants if your fleet is mixed.

Background tabs and throttling. Browsers throttle timers and may deprioritise GPU work for hidden tabs. A kiosk that screensavers or switches tabs can see generation stall. Keep the inference page visible or use appropriate keep-alive handling, and test it.

Eviction and cold starts. Without persistent storage, a weight cache of a few gigabytes may be cleared. Users then face a long re-download. Request persistence, show cache status and provide a way to pre-warm devices during provisioning.

Thermal and power behaviour. Sustained decoding drives the GPU hard. On battery-powered tablets this drains the battery and may throttle clocks; throughput measured in a five-second benchmark is not the throughput after ten minutes. Cap generation length and avoid running continuous background inference.

Numerical differences. Quantization and f16 accumulation introduce small errors. A model’s quality at 4 bits is usually acceptable for summarisation but can degrade on arithmetic, code and long-context recall. Evaluate on your own task before shipping, and do not generalise from a public leaderboard.

Security surface. A page that loads a model from a third-party origin and executes compiled shaders is running code you did not write. Pin versions, host assets yourself and apply a strict Content Security Policy. Also remember that model output rendered as HTML is untrusted input; escape it.

Benchmark traps. Tokens per second for decode and prefill are different numbers; first-run numbers include compilation; and results depend on Chrome version, driver and power state. When you read the WebLLM paper’s figures, remember they come from one machine and two models. Report your own with the exact browser build, model identifier and context length.

Practical Recommendations

Start from the deployment, not the library. If you control the browser fleet and the hardware, WebGPU LLM inference is a realistic way to give dashboards and kiosks private, offline-capable language features. If you do not, treat it as a progressive enhancement over a server or WASM fallback.

Choose a model that fits comfortably: for a general fleet, a 1B to 3B model at 4-bit weights, with a context window capped to what the task needs. Compute the KV cache cost using the architecture’s layer count, KV heads and head dimension, and leave headroom for the browser and the rest of the page. Prefer WebLLM when you want the fastest chat-style decoding and structured output; prefer Transformers.js for a family of small tasks sharing an API; use ONNX Runtime Web directly when you must control tensors and sessions.

Design the loading experience deliberately. The first visit is a multi-hundred-megabyte to multi-gigabyte download, followed by shader compilation. Show real progress, request persistent storage, and pre-provision kiosks so operators never see the cold start.

Checklist before shipping:

Detect navigator.gpu, adapter limits and shader-f16; pick the model variant accordingly.
Run the engine in a worker; keep the UI thread free.
Cap context and generation length; budget weights plus KV cache plus headroom.
Call navigator.storage.persist(); verify shard hashes; self-host assets with a strict CSP.
Handle device loss and out-of-memory with a retry path and a smaller fallback model.
Test on the oldest and weakest machine in the fleet, not the developer laptop.
Measure time to first token and sustained tokens per second after warm-up, with the browser build recorded.
Keep a human or deterministic rule between model output and any control action.
Re-check the browser support status and library versions at each release; this area moves quickly.

Frequently Asked Questions

What is WebGPU LLM inference?

It is running a large language model inside a web page by executing the model’s matrix and attention operations as WebGPU compute shaders on the user’s GPU. Weights are downloaded once, cached by the browser and held in GPU buffers; the page streams generated tokens locally with no server round trip. Libraries such as WebLLM, Transformers.js and ONNX Runtime Web provide the engine, tokenizer and kernels so applications use a high-level API.

How fast is WebLLM compared with native inference?

The WebLLM paper reports retaining up to about 80% of native MLC-LLM performance on the same device. Its table shows 41.1 versus 57.7 tokens per second for Llama-3.1-8B (71.2%) and 71.1 versus 89.3 for Phi-3.5-mini (79.6%), measured on an Apple M3 Max in Chrome Canary. That is one machine and two models, so treat it as indicative and benchmark your own hardware and model before committing.

How much memory does a browser LLM need?

Add weights and KV cache, plus headroom. At roughly 4.5 bits per weight, an 8B model needs about 4.5 GB for weights and a 1B-class model about 0.7 GB (illustrative arithmetic). The KV cache for Llama-3.1-8B is 128 KiB per token in f16, so 4,096 tokens add 512 MiB. Individual buffers must also fit the adapter’s maxBufferSize, which defaults to 256 MiB unless higher limits are requested.

Is WebGPU faster than WASM for LLMs?

For decoder models beyond about one billion parameters, usually yes, because GPU memory bandwidth and parallelism favour both prefill and decode, and WASM is bound by CPU bandwidth and a 4 GiB address space unless memory64 is available. For tiny models and encoders the GPU’s initialisation overhead can erase the gain, and ONNX Runtime documentation suggests WASM for very lightweight models. Claims such as “up to 100x” are best-case figures without published methodology.

Which browsers support WebGPU for LLM inference?

Recent Chrome and Edge support it by default on Windows, macOS, Android and ChromeOS according to ONNX Runtime’s documentation, while Firefox and Safari were behind flags or preview builds at the time of that page. Support has been expanding, so check the live status tracker and caniuse for current versions. Also require shader-f16 for q4f16 models and verify adapter limits at runtime.

Can browser LLMs work offline and keep data private?

Yes, once the weights are cached in the Cache API or OPFS and the app shell is served by a service worker. Prompts and outputs stay on the device unless your own code sends them elsewhere. Request persistent storage to prevent eviction, pin and verify model hashes, and remember that local inference protects data only if telemetry and logging are equally local.