ExecuTorch 1.5 On-Device LLM Serving (2026): Batched Scheduling, Cancellation and Off-Graph KV Cache

For three years the on-device large language model question was a conversion question: can I export this model, can I quantize it small enough, will the delegate take my attention kernel. ExecuTorch 1.5 on-device LLM serving quietly retires that framing. The 16 September 2026 release notes list multi-method export, batched request scheduling, bounded cancellation and off-graph KV-cache layouts in a single bullet — four capabilities that only matter once a phone or a Jetson-class board is holding more than one live request at a time. That is a server problem, and it has now arrived on a device with no operator, no autoscaler and a battery.

The consequence is architectural rather than numerical. An on-device assistant in late 2026 needs a request lifecycle, not just a quantized artifact. And the KV-cache residency decision — where generation state lives and who owns it — is the one that decides whether you can cancel a half-finished answer without leaking memory until the app is killed.

What this covers: what actually shipped in 1.5, how the export artifact, the scheduler and the cache pool fit together, how a request is admitted, prefilled, streamed and cancelled, which failure modes are new, and what to decide before you adopt it.

Context and Background

ExecuTorch is PyTorch’s on-device runtime: a model is captured with torch.export, lowered through to_edge_transform_and_lower() with a backend partitioner, and serialized to a .pte program that a small C++ runtime executes. It reached 1.0 general availability in 2025 and has shipped on a fast minor cadence since — v1.4.0 on 7 August 2026, v1.4.1 a week later, and v1.5.0 on 16 September 2026.

Through that period the competitive field was defined by conversion coverage and kernel quality. llama.cpp offered a self-contained GGUF pipeline. MLX gave Apple silicon a native array framework. ONNX Runtime and LiteRT covered the heterogeneous-accelerator middle ground. Our own ONNX vs TFLite vs ExecuTorch vs Core ML comparison framed the choice exactly that way, because that was the choice teams were actually making: which runtime can ingest my architecture and survive the delegate.

Two things changed underneath that framing. First, on-device models stopped being one model. A useful assistant runs a speech front end, a text decoder, an embedding pass for retrieval and possibly a reranker — and 1.5’s expanded workflow list (Qwen3.5 MoE, Muse Glimmer, Supertonic, Voxtral) reflects that mix directly. Second, the interaction pattern stopped being one-shot. Background summarisation, a foreground chat turn and a wake-word-triggered voice request can overlap on a single device, and the user is entitled to abandon any of them mid-sentence.

Server stacks solved this years ago with continuous batching, paged attention and preemption. Those techniques assume a datacentre memory hierarchy and a machine that does nothing else. Porting the ideas to a phone is reasonable; porting the assumptions is not. That gap is what 1.5 starts to close, and what the rest of this post is about.

The Serving Layer Is the New Hard Part: A Reference Architecture

ExecuTorch 1.5 turns an exported model into a servable unit by adding three things above the runtime: an artifact that can hold several methods, a batch-aware scheduler that admits and sequences concurrent requests, and a KV-cache pool that lives outside the exported graph so the runtime can allocate, clone and reclaim it independently of any single generation.

Figure 1: How an exported ExecuTorch artifact becomes a servable unit — multi-method PTE, capacity-aware memory plan, batch control, scheduler, session slots and an off-graph KV cache pool.

Reading the diagram top to bottom: a PyTorch model is exported through export_llm with a multi-method configuration, producing a .pte program and, for large weight sets, a companion .ptd file. The module executor loads it with per-backend load options and a capacity-aware memory plan computed against a target memory map. Batch control publishes a max_seqs() limit. The scheduler admits requests up to that limit, binds each to a session slot, and drives the executor a batch step at a time. Generation updates stream back to the application callback, while a separate cancel-and-shutdown path can retire a session between steps.

From one method to many: what multi-method export changes

The release notes are precise about this: 1.5 “added multi-method model support to the export pipeline and per-backend load options.” An ExecuTorch program has always been able to carry more than one method — the custom-LLM export guide shows constant methods like get_max_seq_len, get_max_context_len, use_kv_cache, enable_dynamic_shape, get_bos_id and get_eos_ids serialized alongside forward so the runner can query them at load time. What 1.5 does is make the export pipeline itself multi-method aware, so distinct compute graphs — a prefill graph shaped for long sequences and a decode graph shaped for one token, say — can be produced and lowered together rather than glued from separate exports.

That matters more than it sounds. Prefill and decode have genuinely different shapes. Prefill wants a dynamic sequence dimension and large matrix multiplies; decode wants a fixed single-token shape and is bandwidth-bound on weight reads. Exporting them separately means two .pte files, two module loads, two sets of delegate blobs and — worst of all — two copies of the weights unless you arrange external weight sharing by hand. One artifact with two methods lets the memory planner see both at once.

Per-backend load options are the second half. ExecuTorch has always required a separate .pte per backend target, because lowering is backend-specific. Load options let one artifact express how it should be brought up on a given delegate rather than forcing a separate build for each configuration knob. The 1.5 CUDA notes make the intent visible: the delegate gained per-thread streams, a warm memory pool and cross-method weight sharing, plus clearer errors when a method’s weights are missing. Cross-method weight sharing is the load-bearing phrase — it is what makes a two-method artifact cost roughly one model’s worth of memory instead of two.

The scheduler is now yours to reason about

Batched request scheduling means the runtime will interleave work for several live generations. The public tree makes the shape of it legible: the off-graph batching stack lives under extension/llm/batching/, with runner.h/runner.cpp driving generations, module_executor.h/.cpp constructing the executor and sizing capacity, executor.h defining the backend contract, and metrics.h/.cpp accounting for the result. BatchControl::max_seqs() is the hard ceiling on concurrent sessions.

The important design point is that a scheduler is a policy, and a policy has to be chosen. On a server the default policy is throughput-maximising: admit everything, batch aggressively, let queueing theory sort it out. On a device that default is usually wrong, because the requests are not fungible. A foreground chat turn where the user is watching characters appear has a latency budget measured in the tens of milliseconds between tokens; a background note-summarisation job has a budget measured in minutes. Batching them together gives them the same per-step latency — which means the interactive request pays for the batch job.

This is my analysis rather than a documented guarantee, but it follows directly from how batched decode works: every session in a batch advances one token per step, so the step time is set by the batch, not by the request. Mixing a long-context background job into a batch with a short-context foreground turn raises the foreground turn’s inter-token latency for the whole duration of the background job. On a server you hide that with more replicas. On a phone there is no second replica.

The practical implication is that the admission decision — not the kernel — is where your product’s responsiveness is decided. If your app has a notion of foreground and background work, that notion has to reach the scheduler.

Capacity-aware memory planning is the budget that binds it

The third runtime change in 1.5 is the one that makes the other two survivable: “capacity-aware memory planning over target memory maps and support for shared allocations with offsets.” ExecuTorch has always planned activation memory ahead of time — that is a core part of how it runs without a heap allocator on microcontrollers. Capacity-aware planning against a target memory map extends that to say: here is the memory this device actually has, and here is how the arenas must fit inside it.

Shared allocations with offsets are the mechanism that lets multiple sessions and multiple methods carve out of one arena rather than each demanding its own. Combined with max_seqs(), this gives a bounded worst case: the maximum number of concurrent sessions, multiplied by the per-session cache footprint implied by get_max_context_len, has to fit the plan. If it does not, the correct behaviour is to refuse the session at admission, not to fail mid-generation.

That is the real reason serving is harder on-device than on a server. A server can overcommit and page. A phone under memory pressure gets its app killed by the operating system, and the user experiences that as data loss, not as a slow response. Bounded, pre-planned capacity is not an optimisation here; it is the correctness property.

The artifact contract got a version number

The quietest item in the 1.5 runtime list is the one most likely to bite a shipping team: “schema-version checks for PTE and PTD files and clearer failures for invalid device-planned copies and missing delegate data.”

A .pte is the program; a .ptd carries weights separately, which is how large models avoid embedding hundreds of megabytes in the program file. Those two artifacts have to agree. Once you are shipping models out-of-band — and most serious on-device LLM products do, because nobody wants a 3 GB app-store binary — the program and the weights stop being built and shipped together. The app updates on the store’s cadence; the model updates on yours.

That decoupling creates a matrix of version pairs you did not deliberately design. An app carrying a 1.4-era runtime can receive a .ptd produced by a 1.5 exporter. A user who skipped three app updates can pair an old program with a new weight file. Before explicit schema checks, the failure mode for a mismatch was whatever happened when the runtime read fields it did not understand: a confusing error at best, and silently wrong tensors at worst.

Explicit version checks convert that into a clean, early refusal, which is the behaviour you want — but only if your delivery layer is prepared to handle the refusal. Concretely: your model-download service needs to know which runtime version each client is running and serve a compatible artifact, or your client needs to fall back to a bundled model when the check fails. Neither is hard; both are easy to forget until a field report arrives.

The related “missing delegate data” error is the same story one level down. A .pte lowered for a delegate carries blobs that only that delegate can interpret. Loading it on a build where the delegate was compiled out previously produced obscure failures; a clear error makes the CI assertion writable. Assert it in CI, not on a device in a tester’s hand.

Walking the Request Lifecycle: Admission, Prefill, Decode, Cancellation

The documented C++ runner API is the right place to ground this, because it shows what the application actually holds. IRunner exposes is_loaded(), load(), generate(), generate_from_pos() and stop(). GenerationConfig carries echo, max_new_tokens, warming, seq_len, temperature, num_bos and num_eos, plus a resolve_max_new_tokens(max_context_len, num_tokens_occupied) helper that reconciles the user’s request against the model’s context limit and the positions already consumed. Inside, TextLLMRunner orchestrates, TextPrefiller fills the cache from the prompt, TextTokenGenerator runs the autoregressive loop, and TextDecoderRunner drives the module forward pass.

Figure 2: A single request through admission, slot reservation, prefill, token streaming and cancellation — with slot release as an explicit, acknowledged step rather than a side effect.

The sequence runs: the app submits a prompt; the runner asks the scheduler to admit it; the scheduler reserves a session slot from the KV pool and receives a handle; prefill runs as one or more batch steps; first logits return; tokens stream to the app callback. Then the app cancels. The runner sets a cancelled flag, the scheduler stops the session at the next batch boundary, releases the slot back to the pool, and the teardown is acknowledged. Note that release is drawn as its own message. That is the point of the diagram.

Off-graph KV-cache layouts: flat, ring and cell

In the classic ExecuTorch LLM flow, KV cache tensors are model-owned buffers. The custom-LLM guide is explicit: register key and value caches as module buffers so they are part of the exported program state, update them by tensor position rather than Python-side counters, keep shapes predictable, and return logits only. That design is excellent for a single sequential generation. It is the wrong shape for concurrency, because the cache is now part of the graph’s state and there is exactly one of it.

Off-graph KV cache inverts that ownership. The cache becomes a runtime-managed allocation that the executor binds to a session, rather than a buffer baked into the program. The 1.5 notes describe the Core ML and MLX work concretely: “MLX off-graph KV-cache flat, ring, and cell layouts with shared pools and cross-thread execution fixes.” Three layouts, one pool abstraction.

Figure 3: Flat, ring and cell layouts for an off-graph KV cache pool, and the operational property each one buys or costs.

Reading it across: a flat layout gives each session a contiguous region — simple to reason about and trivial to reclaim, but prone to fragmentation when prompt lengths vary widely across concurrent sessions. A ring layout wraps at capacity, which bounds the footprint absolutely but means the oldest positions leave the window; the PR history for the batching stack explicitly tests “physical ring wrap” behaviour. A cell layout hands out blocks from a shared pool, which is what makes cloning and prefix reuse possible at all, at the cost of needing retention rules to decide whether a previously computed prefix is still valid.

The release notes say these layouts exist. They do not say any of them is faster, and I am not going to claim otherwise — the notes contain no performance numbers at all. What the layouts buy is control: the ability to bound, share and reclaim generation state independently of the model graph.

Bounded cancellation and what “bounded” has to mean

IRunner::stop() has existed for some time and is documented as immediately stopping the generation loop, typically called from another thread. “Bounded cancellation” in the 1.5 notes is a stronger claim, and the distinction is worth being precise about.

An unbounded stop sets a flag and hopes. The generation loop notices at some point, exits, and whatever memory the session held is reclaimed whenever the owning object is destroyed — which, in a batched runtime, may be much later, or never if a handle is held somewhere. A bounded cancellation guarantees a worst-case interval between the cancel call and the point at which the session’s resources are back in the pool.

The batching stack’s own tests describe exactly this territory: extension/llm/batching/test/runner_test.cpp covers “reuse, replay, accounting, cancellation, and shutdown,” and the runner code checks an atomic cancelled flag on the generation state inside the step loop. Reviewers on the prefix-cache work walked the clone failure and shutdown paths specifically looking for leaks, noting that a non-published clone erases its registered record and calls close_session.

The reason this is the pivotal feature — and this is the thesis of the post — is that on a device, a leaked session slot is not a slow leak. It is a fixed fraction of a hard budget. If max_seqs() is small, as it must be on a phone, then leaking two slots can mean the app can no longer admit a foreground request at all, and the user sees an assistant that has simply stopped working. Cancellation correctness and capacity correctness are the same property viewed from two sides.

Prefix reuse, cloning and the metric that lies

The most instructive recent work in this area is the prefix-cache addition to the off-graph batching stack, which introduces Session::clone_async(upto) for independently writable committed prefixes, a caller-owned PrefixCache doing longest-prefix matching with LRU eviction, and a capture_prompt(...).wrap(...) / collect() flow that requests the clone before the first user callback and inserts it after generation completes. Completed opening prefills publish immutable backend snapshots; a fresh session clones the longest usable token prefix, computes only the suffix, and replays the final requested token to obtain fresh logits. Caching is off by default.

Two details in that description deserve emphasis because they generalise beyond ExecuTorch.

First, retention rules decide validity, not token equality. A matching token prefix is necessary but not sufficient — the backend’s retention rules determine whether the stored state is still usable, including for models that mix full-attention and sliding-window layers. A sliding-window layer that has already evicted positions cannot serve a prefix that depends on them, even though the tokens match. Any prefix cache that checks only token identity will, on such a model, return state that is silently wrong.

Second, accounting is easy to get wrong in a flattering direction. A reviewer on that change caught precisely this: recording a cache hit against the full prompt-token count makes a prefill-throughput metric use all n_prompt_tokens as its numerator while the timing span starts at the first executed batch and therefore excludes prefix restoration. A large cache hit then reports cached work as prefill throughput, and the rate can be made arbitrarily high. If you build your own instrumentation on top of this stack, decide up front whether your denominator includes restoration time — otherwise your dashboard will reward you for doing less work while the user waits the same amount of time.

This is a good general warning for on-device serving benchmarks in 2026. The interesting numbers are per-request latency under concurrency and worst-case memory at max_seqs(), not aggregate token throughput. Throughput is the metric a batching system can always improve by making individual users wait longer.

Matching workload shape to layout and policy

The three cache layouts and the admission policy are not independent choices — a workload shape usually implies both. The table below is my synthesis rather than a recommendation from the project, but the reasoning behind each row is stated so you can disagree with it specifically.

Workload	Shape	Layout that fits	Admission policy
Single foreground chat	One long-lived session, user watching	Flat	Concurrency 1, reserve the whole budget
Chat plus background summarisation	Two classes, very different latency budgets	Cell	Priority classes, never co-batch across classes
Voice assistant with barge-in	Short sessions, frequent cancellation	Cell with strict teardown	Admit eagerly, cancel aggressively
Long-running document agent	One session, context exceeds the window	Ring	Concurrency 1, accept window eviction
Repeated system-prompt calls	Shared prefix across many short requests	Cell with prefix reuse	Admit in batches, cache the shared prefix

The logic: a flat layout is the right default whenever there is exactly one session, because its reclaim path is trivial and fragmentation cannot occur with a single tenant. A ring becomes attractive the moment the workload’s natural context exceeds what you can afford to keep resident, because it converts an unbounded requirement into a fixed one — at the explicit cost of forgetting the oldest positions, which you must design the product around rather than discover.

Cell layouts earn their complexity in exactly two situations: several concurrent sessions of unpredictable length, and shared prefixes worth cloning. If neither is true, the block-management machinery and the retention rules that come with it are cost without benefit.

Barge-in deserves its own row because it is the case that makes cancellation correctness visible fastest. A user who interrupts a spoken answer expects the device to stop and start listening within a perceptual instant, and they may do it several times a minute. A session teardown path that leaks even occasionally will exhaust a small max_seqs() ceiling within a single conversation.

Trade-offs, Gotchas, and What Goes Wrong

The first and least glamorous gotcha is that the documentation lags the code. As of this writing the published stable ExecuTorch documentation set still renders as the 1.3 documentation — the serving features described here are in the release notes and the source tree, not yet in the narrative docs. If you are evaluating ExecuTorch 1.5 on-device LLM serving, read the release notes and the headers under extension/llm/, and treat any tutorial you find as describing the single-session world.

Figure 4: Four failure modes that only appear once a device serves concurrent requests, the first thing to audit for each, and the regression test they all converge on.

The chart maps symptom to suspect. Memory growing after a cancel points at a slot that was never returned to the pool — audit the teardown path end to end, including the case where the client drops its handle without calling stop. Answers drifting across repeated identical prompts points at a prefix cache hit that should not have been served — audit retention rules, especially on sliding-window models. Latency spiking under load points at head-of-line blocking from a long request sharing a batch with short ones — audit admission policy before touching kernels. A program that fails to load at all points at the new schema-version checks for .pte and .ptd files, which are a 1.5 addition and will reject mismatched pairs that older runtimes accepted silently.

Beyond the chart, three honest limits.

Delegate coverage is uneven and always will be. The 1.5 C++ SDK ships linkable libraries for the CUDA, Core ML, MLX, OpenVINO and Qualcomm delegates. The off-graph KV-cache layout work is called out for MLX. Arm’s Ethos-U85 gained KV-cache export and int8 KV cache in the same release. None of that means the full serving stack behaves identically across every backend, and you should assume it does not until you have tested your target.

Batching is not free on an unbatched accelerator. Some NPUs compile a fixed batch geometry at conversion time. If your target only accepts batch size one, the scheduler’s value collapses to time-slicing between sessions, which still buys you fairness and cancellation but not arithmetic efficiency. Check this before you design around it; the quantization and dispatch decisions in our INT4 vs INT8 vs FP8 edge NPU guide interact with it directly.

Thermals and power are the uncounted scheduler input. A device that sustains concurrent decode will throttle. Nothing in the runtime knows that. Any admission policy that ignores thermal state will, under sustained load, produce a product that gets slower the more the user relies on it — which is the worst possible failure shape for a feature you are trying to build a habit around.

One more scoping note. Much of this applies only to devices that can plausibly hold multiple sessions. The same 1.5 release expanded the Cortex-M and Ethos-U story considerably — Ethos-U65 support, Ethos-U85 KV-cache export, opt-in Cortex-M explicit-layout lowering with convolution and pooling kernels, int8 KV cache, and FP16, BF16 and MXFP8 coverage — and on that class of hardware the serving discussion mostly evaporates. A microcontroller running a small model has one caller, no operating-system memory pressure to speak of, and a memory plan that was fixed at build time by necessity rather than policy. The interesting work there is the opposite of scheduling: it is squeezing a KV cache into int8 so it fits at all. Do not import a phone-shaped serving architecture onto a Cortex-M target; the constraints do not rhyme.

Practical Recommendations

Treat the upgrade as a serving-architecture change, not a version bump. The conversion work you already did still applies; what is new is everything above the .pte.

Start by writing down your request taxonomy before you touch the scheduler. Most on-device assistants have two or three classes — interactive, opportunistic, background — with radically different latency budgets. That taxonomy is the input to admission policy, and without it any scheduler configuration is guesswork.

Then establish your memory ceiling empirically. Take max_seqs(), multiply by the per-session cache footprint implied by your get_max_context_len, add the weight and activation arenas, and compare against the memory your app is realistically allowed on the oldest device you support. If the number does not fit, reduce context length or concurrency — those are the only two levers that move it, and reducing context is usually the less visible of the two.

Leave prefix caching off until you have a correctness harness. It is off by default for good reason: a cache that serves a stale prefix produces a wrong answer that looks completely plausible, which is far worse than a slow one.

A checklist to run before shipping:

• Confirm your target delegate actually supports off-graph KV cache, rather than assuming parity across backends.
• Re-export with multi-method configuration and verify cross-method weight sharing reduced, not duplicated, resident memory.
• Add a leak test that cancels a generation mid-decode and asserts the slot count returns to baseline.
• Add a second leak test that destroys the client handle without calling stop.
• Test admission at exactly max_seqs() and at max_seqs() plus one; the second must refuse cleanly.
• Measure inter-token latency for a foreground request while a background request is in flight, not in isolation.
• Verify that a .pte / .ptd pair from your build system passes the 1.5 schema-version checks in CI, not on device.
• Instrument prefill timing with a denominator that includes prefix restoration.
• Exercise sliding-window models specifically if you enable prefix reuse.
• Re-run the whole suite after any delegate or SDK bump, since load options and retention behaviour are backend-owned.

Frequently Asked Questions

What is off-graph KV cache in ExecuTorch 1.5?

Off-graph KV cache means the key/value state for generation is allocated and owned by the runtime rather than baked into the exported program as model buffers. In the classic ExecuTorch flow the cache is a module buffer that is part of the .pte state, which works well for one sequential generation. Off-graph placement lets the runtime bind cache regions to individual sessions, share a pool across them, clone committed prefixes, and reclaim state when a session ends. The 1.5 notes describe flat, ring and cell layouts for MLX with shared pools.

Does ExecuTorch 1.5 make on-device LLMs faster?

The release notes do not say so, and they contain no performance figures, so no honest answer can claim a speedup. What 1.5 adds is capability and control: the ability to run multiple methods from one artifact, serve concurrent requests, cancel them with bounded resource release, and plan memory against a real device budget. Whether any of that improves latency on your hardware depends entirely on your model, delegate and workload, and you should measure it rather than infer it from the changelog.

What does multi-method export actually give me?

It lets one exported artifact carry several compute graphs — commonly a prefill graph with a dynamic sequence dimension and a decode graph with a fixed single-token shape — produced and lowered together rather than as separate exports. The practical benefits are one module load instead of several, a memory planner that can see all the methods at once, and cross-method weight sharing so you are not paying for duplicate weights. ExecuTorch programs already carried constant methods for metadata; 1.5 extends multi-method awareness into the export pipeline itself.

How is bounded cancellation different from calling stop?

IRunner::stop() interrupts the generation loop, typically from another thread. Bounded cancellation is the stronger property that the session’s resources return to the pool within a known worst case, rather than whenever the owning object happens to be destroyed. On a device with a small max_seqs() ceiling, that difference is the difference between an app that recovers from an abandoned request and one that gradually loses its ability to accept new ones. The batching test suite covers cancellation, closure and shutdown explicitly.

Should I turn on prefix caching for repeated prompts?

Not before you can test it. The prefix cache is disabled by default, and the correctness condition is subtler than matching tokens: the backend’s retention rules determine whether stored state is still usable, and sliding-window attention layers can evict positions a matching prefix depends on. A cache that serves an invalid prefix produces a confidently wrong answer. Also check your metrics — counting cached tokens in a prefill-throughput numerator while excluding restoration time from the denominator inflates the number without helping the user.

Is ExecuTorch 1.5 ready for production on my SoC?

That is not a question the release notes answer, and nothing here should be read as claiming general availability on any particular chip. Delegate coverage varies: the 1.5 C++ SDK ships linkable libraries for CUDA, Core ML, MLX, OpenVINO and Qualcomm; the off-graph cache layout work is called out for MLX; Arm gained Ethos-U85 KV-cache export and int8 KV cache. Validate your own target — concurrency behaviour, memory ceiling and cancellation teardown — before committing.

Further Reading

• ONNX vs TFLite vs ExecuTorch vs Core ML in 2026 — the conversion-layer comparison this post sits on top of.
• On-device LLM runtimes: llama.cpp vs MLC vs ONNX Runtime — where ExecuTorch fits among the alternatives.
• INT4 vs INT8 vs FP8 quantization for edge NPUs — the precision decisions that set your per-session memory footprint.
• Jetson Thor vs Jetson Orin AGX — hardware context for edge boards that can realistically serve concurrent LLM requests.
• ExecuTorch v1.5.0 release notes — the primary source for everything asserted above.
• Running LLMs with C++ — ExecuTorch runner API — the documented IRunner and GenerationConfig surface.

By Riju — about