MLPerf Edge Agentic Inference: How TensorRT Edge-LLM Beat llama.cpp 6.4x

A 27-billion-parameter model ran the same 1,007-turn agent workload twice on the same Jetson AGX Thor developer kit. One run took 2 hours 37 minutes. The other took 24 minutes 36 seconds. Same silicon, same model family, same recorded conversations — a 6.4x gap produced entirely by the serving stack. That comparison is the headline of the new MLPerf Edge Agentic Inference benchmark, introduced in MLPerf Inference v6.1 on 16 September 2026, and it is more interesting than the number suggests. The slow run is MLCommons’ deliberately plain llama.cpp reference. The fast run is NVIDIA’s tuned TensorRT Edge-LLM submission. Neither is a claim about hardware. Both are claims about how much headroom sits between a working edge deployment and an optimised one.

What this covers: how the benchmark is actually constructed, why agentic workloads invert the usual prefill-versus-decode arithmetic, a decomposition of the 6.4x into four independent optimisations, a runnable reproduction path for both stacks, and an honest account of what this result does not establish.

Context and Background

Edge inference benchmarking has, until now, measured the wrong shape of work. The established MLPerf edge categories score single-shot requests: one prompt in, one response out, throughput and latency reported per query. That model described chatbots and classification pipelines well enough. It describes an agent badly.

An agent does not answer a prompt. It selects a tool, reads the tool’s output, folds that output back into the same conversation, and reasons again. Each iteration re-sends nearly everything that came before. A coding agent twenty turns into a debugging session is carrying a transcript of every file it read and every command it ran. The input to turn twenty is not a new prompt — it is turn nineteen’s prompt plus a few hundred tokens. Measuring that with a single-shot benchmark produces numbers that are technically correct and operationally useless.

This matters more at the edge than in a data centre, because the constraints bind differently. A vehicle, a robot, or an industrial gateway has a fixed power envelope, a fixed memory budget, and no ability to shard a long context across eight accelerators. The question an edge architect actually needs answered is not “how many tokens per second” but “can this device sustain a twenty-turn tool-calling loop inside its thermal budget while still emitting syntactically valid function calls.”

MLCommons’ answer is MLPerf Edge Agentic Inference, one of two new tests in the v6.1 round — the other being an end-to-end retrieval-augmented generation benchmark scored separately for corpus ingestion and query answering. According to MLCommons, reported via the round’s press release, v6.1 drew 30 submitting organisations, 120 systems and 486 datacenter and edge results, with six first-time submitters. The same release notes that speculative decoding became supported in the interactive scenario for two existing benchmarks plus GPT-OSS — a detail that matters here, because speculative decoding is one of the four levers behind the 6.4x.

NVIDIA published its submission analysis the same day the results went live. If you want the broader hardware context for the device under test, our comparison of Jetson Thor and Jetson Orin AGX covers the generational change; the TensorRT-LLM versus llama.cpp analysis for Jetson covers the two runtimes in question outside a benchmark setting. The primary source for everything attributed to NVIDIA below is its TensorRT Edge-LLM MLPerf submission post.

Inside the MLPerf Edge Agentic Inference benchmark

MLPerf Edge Agentic Inference measures an OpenAI-compatible model endpoint in two phases driven by one shared configuration file. The performance phase replays 20 recorded software-engineering agent conversations totalling 1,007 generated turns, scored by an inline checker. The accuracy phase runs a sampled Berkeley Function Calling Leaderboard v4 gate of roughly 995 single-turn prompts. Both phases must run against the same server under the same settings.

That last constraint is the design decision worth dwelling on, and it is the reason this benchmark is harder to game than it looks.

Figure 1: The two phases of the MLPerf Edge Agentic Inference benchmark and the gates each must clear.

The diagram traces both phases from the single online_edge_full_run.yaml configuration. The left branch is the performance workload: a recorded trajectory file drives 20 conversations and 1,007 generated turns at a peak input length of roughly 23.5K tokens, with an inline Intersection-over-Union checker scoring the emitted tool calls against the recorded ones. The right branch is the accuracy gate: a per-category sample of BFCL v4 single-turn prompts producing an overall accuracy score, which must reach at least 0.97 times the reference. A run is only valid if zero turns are missing.

The performance phase replays real trajectories, not synthetic prompts

The performance dataset is a recorded set of agentic-coding trajectories. The model receives a user request, generates a tool call, observes the tool result, and continues the same conversation. Input length grows across turns and reaches approximately 23.5K tokens, which makes long-context handling a first-class part of the measurement rather than an afterthought.

The dataset is constructed so that no conversation overflows a 32K served context. That is not an arbitrary choice. It means a submitter cannot quietly win by truncating history, and it means every turn completes — the validity criterion is literally zero dropped turns. A stack that silently evicts context to keep up produces an invalid run, not a fast one.

Scoring inside the performance phase uses an inline checker rather than a post-hoc accuracy pass. MLCommons’ configuration sets the evaluation method on the performance dataset itself, so the recorded trajectory serves simultaneously as the workload and as its own ground truth. The reference run scores an inline IoU of 0.6335. The purpose is not to rank models on that number — it is to catch a submission that runs fast because it stopped doing the task correctly.

The accuracy phase is a sampled gate, not a leaderboard run

The gated accuracy metric is BFCL v4, single-turn only, with reasoning disabled. Three categories are in scope: non_live, live, and hallucination. Per-category sampling is tuned to draw roughly 995 samples — 62% of non_live for about 712 samples, 10% of live for about 171, and 10% of hallucination for about 112 — with any subset of 25 or fewer taken in full so small subsets are not reduced to statistical noise.

The hallucination category is the one most people skip past, and it is arguably the most important for an agent. It scores whether the model declines to call a tool when no tool is appropriate. An agent that fabricates a function call is worse than an agent that does nothing, because the fabricated call enters the loop as a real action. The reference scores 97.16% there, against 82.59% on non_live abstract-syntax-tree checking and 84.12% on live.

The pass criterion is a 3% one-sided band anchored on the validated reference: a submission passes at 0.97 times the reference score, with no upper bound. Because decoding is deterministic at temperature 0 with a fixed seed, MLCommons treats accuracy as hardware-independent, so the same thresholds apply on any device. The two gated figures are an overall single-turn score of 86.23% (pass at 83.64%) and a non_live-normalised score of 87.96% (pass at 85.32%).

Multi-turn BFCL subsets remain available as an optional exploratory run but sit outside the gate. MLCommons is explicit about why: the small sampled multi-turn subsets are dominated by per-entry granularity noise. For reference, a single full run of the 200-entry multi_turn_base scored 140/200, or 70.00%.

The reference submission is deliberately plain — and that is the point

The MLCommons reference is Qwen3.6-27B quantised to Q4_K_M GGUF, served by a natively built llama.cpp llama-server at a pinned commit, compiled with CUDA on a Jetson AGX Thor running JetPack 7 and CUDA 13. It runs single-slot with -np 1, a 32768-token context, flash attention on, all layers offloaded to GPU, reasoning off, seed 42.

Nothing about that configuration is incompetent. It is a sensible, reproducible, widely deployed way to serve a quantised model on an edge device — which is exactly why MLCommons chose it as the baseline. But it is also, by construction, a stack with no prefix caching, no speculative decoding, and a weight-only 4-bit quantisation scheme that leaves activations at higher precision. Every one of the four optimisations discussed below is absent from it. Keep that in view whenever the 6.4x comes up.

One more detail from the reference configuration deserves a mention, because it is a free finding for anyone deploying a tool-calling agent. MLCommons measured the same 1,007-turn workload with server-side reasoning enabled and disabled. Reasoning off: IoU 0.6335 in 2 hours 37 minutes. Reasoning on: IoU 0.6374 in 4 hours 13 minutes. That is roughly 60% more wall-clock time for an accuracy difference inside run-to-run noise. On a function-calling workload, the reasoning tokens are largely wasted work.

Why an agent turn breaks the usual prefill-versus-decode model

The conceptual core of MLPerf Edge Agentic Inference is that the standard mental model of LLM serving — prefill is compute-bound, decode is memory-bound, and the ratio between them determines your bottleneck — stops describing the workload once conversations repeat their own history.

Figure 2: Where an agent turn’s prompt tokens actually go when prefix caching is working.

The diagram follows a single turn. The request arrives carrying the full prior history plus a new tool result. A prefix match against hot KV pages splits it into a cache-hit portion — roughly 96% of prompt tokens on this workload — and a cache-miss suffix. Only the suffix is prefilled. The hit portion is restored from cached attention pages plus, for this model, recurrent state. Decode then proceeds bounded by memory bandwidth, the turn emits a tool call, the result is appended, and the loop closes.

13.6 million prompt tokens, half a million actually prefilled

NVIDIA reports the concrete figures for its run: across all turns of the workload, approximately 96% of prompt tokens are served from hot cache, and the runtime prefills only about 0.5M of the total 13.6M prompt tokens.

Work through what that implies. Without reuse, a stack must prefill 13.6M tokens. With reuse, it prefills 0.5M — a 27-fold reduction in prefill work. If prefill were the dominant cost of this workload, eliminating 96% of it would produce a far larger speedup than 6.4x. It does not, which tells you prefill was never the dominant cost. Decode is.

This is the inversion. In a single-shot benchmark with a long prompt and a short answer, prefill dominates and the tokens-per-second headline is close to meaningless. In an agent loop with prefix caching, prefill collapses to a rounding error and the entire run becomes a very long decode with occasional short prefills stapled on. The 52.33 tokens per second NVIDIA reports is therefore a more meaningful headline for this workload than it would be for a single-shot one — the opposite of the usual advice.

It also reframes what hardware matters. A decode-dominated workload at batch size one is bounded by DRAM bandwidth, not by peak floating-point throughput. That is why the first optimisation in the stack is a quantisation change rather than a kernel change.

What TTFT and TPOT actually mean when the prefix is hot

Time to first token and time per output token are reported as medians in NVIDIA’s summary: 247.12 ms and 14.68 ms respectively. Both need careful reading in an agentic context.

TTFT here is not “time to process a 23.5K-token prompt.” It is mostly the time to prefill a few hundred new suffix tokens plus the cache lookup and state restoration. A median TTFT of 247 ms on a context that peaks near 23.5K tokens is only achievable because of the cache hit rate. Quote that number without the caching context and it reads as a claim about prefill throughput that the stack is not making.

TPOT of 14.68 ms corresponds to roughly 68 tokens per second of raw decode stepping, against a reported end-to-end output throughput of 52.33 tokens per second. The gap between those two figures is where the per-turn overheads live: cache lookup, state restoration, draft-model execution, and the verification passes that speculative decoding requires. That gap is a real cost and it is worth knowing it exists before you assume TPOT and throughput are two ways of saying the same thing.

The MLCommons harness records a richer picture than the four-number summary NVIDIA publishes. Its performance results file carries throughput, TTFT, TPOT, per-turn latency and input/output sequence-length distributions, and the upstream MLPerf submission checker verifies the primary-metric queries per second, p99 latency, and p99 TTFT and TPOT alongside the accuracy score. If you are evaluating a stack rather than reading a press summary, the percentile tail is where the useful information sits — an agent loop is only as usable as its worst turn.

The inline checker exists because speed is easy to fake

It is worth being explicit about why the performance phase scores correctness at all. Every optimisation in the list below is, in principle, capable of trading accuracy for speed. Aggressive 4-bit activation quantisation can degrade structured output. Prefix caching can serve stale state if the reuse logic mishandles a model’s recurrent components. Speculative decoding changes the sampling distribution if verification is implemented loosely.

The inline IoU checker catches the coarse failures during the performance phase, and the BFCL gate catches the subtle ones afterwards. Running both from one configuration file closes the obvious loophole — measuring performance with aggressive settings and accuracy with conservative ones. This is a benchmark designed by people who have watched that trick before.

Decomposing the 6.4x into four stacked optimisations

NVIDIA’s submission ran Qwen3.6-27B in SingleStream mode on one Jetson AGX Thor Developer Kit with 128 GB of unified memory at MAXN power mode, reporting 52.33 tokens per second, median TTFT of 247.12 ms, median TPOT of 14.68 ms, and BFCL overall accuracy of 87.94%, completing the workload in 24 minutes 36 seconds against the reference’s 2 hours 37 minutes.

Four changes account for that gap. They are not alternatives — they stack, and they act at different points in the inference path. Decomposing them is the most durable thing to take from this round of MLPerf Edge Agentic Inference, because three of the four are portable to hardware NVIDIA did not build.

Figure 3: Where each of the four optimisations acts inside a single agent turn.

The diagram places each optimisation at its point of action. Prefix cache lookup happens before any compute. NVFP4 weights and activations plus an FP8 KV cache change the arithmetic and the memory footprint of every step. Tree-based multi-token prediction wraps the decode loop, with a draft model proposing candidates and the target model verifying a 16-node tree in a single forward pass. The resulting latency profile is the observable output of the other three.

NVFP4 for weights and activations, FP8 for the KV cache

Low-batch decoding on an edge platform is bounded by DRAM bandwidth. Every decode step must stream the model’s weights from memory; at batch size one there is no other work to amortise that traffic against. Shrinking the bytes moved per step therefore translates directly into decode rate.

The submitted model uses NVFP4 — a 4-bit floating-point format supported by the Blackwell GPU in Jetson AGX Thor — for weights and activations, including the language-model head. The KV cache is FP8. That “W4A4” designation is the substantive difference from the reference’s Q4_K_M, which is a weight-only scheme. Quantising activations as well as weights shrinks the working set of the kernels themselves, not just the parameter store.

Including the LM head is a detail worth noticing. For a 27B model with a large vocabulary, the output projection is a meaningful fraction of both parameters and per-step compute, and it is commonly left at higher precision because it sits closest to the sampling distribution. Quantising it to NVFP4 and still clearing the accuracy gate is a non-trivial result on its own.

The second-order effect matters for an agent specifically. A smaller model representation leaves more of the 128 GB unified memory free for long context, speculative-decoding state, and whatever application is actually consuming the agent’s output. On an edge device the model does not run alone; it runs alongside perception, control, or logging workloads competing for the same memory. Our comparison of INT4, INT8 and FP8 on edge NPUs covers the accuracy-versus-footprint trade space across formats in more detail.

A deployment team does not need to repeat the calibration. NVIDIA published a calibrated NVFP4 checkpoint at centml/Qwen3.6-27B-NVFP4-W4A4-mlpinf, and post-training quantisation can be done once on any development system before deployment rather than on the target device.

KV cache and recurrent-state reuse on a hybrid architecture

The second optimisation is the one most directly aimed at the agentic shape of the workload. Each new request contains most of the preceding conversation plus a new model response or tool result. Without reuse, the shared history is prefilled again on every turn, and the cost rises as the conversation grows — quadratically in the number of turns if each turn adds a fixed amount of text.

TensorRT Edge-LLM identifies reusable prompt prefixes and restores their cached attention KV pages, then prefills only the new suffix. That much is standard prefix caching, available in most serious serving runtimes.

What is not standard is the handling of Qwen3.6’s hybrid architecture. A hybrid model interleaves full attention layers with layers carrying recurrent state, and recurrent state is not a KV cache — it is a running summary that depends on the entire prefix in sequence order. You cannot restore it by looking up pages. NVIDIA describes the runtime as restoring the recurrent state and partial KV-page state required to continue execution correctly, alongside the attention pages.

This is the part that is genuinely hard, and it is where a naive implementation silently produces wrong answers rather than slow ones. If your serving stack advertises prefix caching and your model is hybrid, verify that it handles recurrent state rather than assuming it does. The failure mode is degraded output quality that looks like a model problem.

The two optimisations are complementary in a specific way: cache reuse reduces cost before generation begins, while multi-token prediction reduces the number of target-model steps during generation. They do not overlap, which is why they stack cleanly.

Tree-based multi-token prediction

Standard autoregressive decoding produces one token per model invocation. Multi-token prediction uses a small draft model to propose several future tokens, which the target model then verifies together in a single forward pass. Recent models frequently ship official MTP weights trained alongside the main model.

Linear MTP keeps one predicted continuation. Tree-based MTP organises high-probability candidates into a tree, verifies all of them in one target-model forward pass, and accepts whichever path matches. If several candidate tokens are accepted, generation advances by several positions at once.

Figure 4: Why a verification tree survives a mismatch that truncates a linear draft chain.

The diagram contrasts the two. A linear chain of three draft steps is verified in one pass, but the first mismatched token ends the accepted run — everything after it is discarded. A tree of 8 draft steps with the top 2 candidates at each depth, verified as a 16-node tree in the same single pass, lets a sibling branch survive a mismatch that would have truncated the linear chain.

The MLPerf server configuration uses exactly those parameters: 8 draft steps, top-2 candidates at each drafting depth, a 16-node verification tree. NVIDIA reports that compared with linear MTP at 3 draft steps, the tree variant could achieve an additional ~40% decoding performance gain on this workload.

Why function calling in particular? Because tool-calling output is structurally predictable in a way that prose is not. Tool names come from a fixed schema. JSON syntax is deterministic — after a key, a colon; after a value, a comma or a brace. Argument structures repeat across calls. A draft model gets the scaffolding right almost every time.

Where it gets things wrong is inside argument values, and that is precisely the case a tree handles better than a chain. The model knows it is about to emit a file path; it is unsure which one. A linear draft commits to one guess and loses the remainder of the chain when that guess misses. A tree keeps two candidates alive at that depth, and the branch that matches carries the rest of the structure with it. The ~40% figure is the aggregate consequence of that pattern repeating thousands of times across 1,007 turns.

The resulting latency profile, and why an agent loop cares

The fourth item is not an optimisation so much as the shape the other three produce, and it deserves its own treatment because agentic and conversational workloads weight the two latency components differently.

In a chat interface, TTFT is what the user perceives; tokens stream after that and a human reads slower than the model generates. In an agent loop the user perceives neither directly. What they perceive is total wall-clock time to complete the task, and that is the sum over every turn of TTFT plus output length times TPOT.

With 1,007 turns, a 100 ms difference in median TTFT is worth about 100 seconds of end-to-end time on its own. A 1 ms difference in TPOT, multiplied by however many tokens each turn emits, can easily be worth more. Neither component dominates in the way a single-shot benchmark would suggest — which is the practical reason the benchmark reports both rather than a single headline latency.

There is also a structural constraint the numbers hint at. Because tool calls must be syntactically complete before the tool can execute, partial output is worthless in an agent loop. You cannot start running a function call that is 80% emitted. The whole turn is a barrier. This is why per-turn latency distributions, rather than averages, determine whether a device is usable for interactive agentic work.

Reproducing the run

Both stacks are reproducible, and running the reference yourself is the single most useful thing you can do with MLPerf Edge Agentic Inference. The gap is more instructive when you have watched the slow side of it.

Running the MLCommons reference

The reference implementation lives in mlcommons/endpoints, at examples/11_Edge_Agentic_Example/. The mlcommons/inference repository carries a README pointer under language/edge-agentic but not the runnable code.

Prerequisites are Python 3.12 or later and roughly 24 GB of memory — GPU, VRAM, or unified. The Q4_K_M GGUF is about 16.8 GB on disk and the remainder is KV cache at a 32768-token context. MLCommons states plainly that 16 GB is not enough.

Pull the model:

pip install -U "huggingface_hub[cli]"
hf download unsloth/Qwen3.6-27B-GGUF Qwen3.6-27B-Q4_K_M.gguf --local-dir ./models

Build llama.cpp at the validated commit. On Jetson AGX Thor the CUDA architecture is sm_110; the toolkit ships with JetPack at /usr/local/cuda:

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git checkout cfff1fc
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=110
cmake --build build --config Release -j --target llama-server

On a DGX Spark (GB10) set -DCMAKE_CUDA_ARCHITECTURES=121 instead. Other devices need their own architecture value.

Serve it with the reference flags — these are not suggestions, they are what the published numbers were produced under:

./build/bin/llama-server \
  --model /path/to/Qwen3.6-27B-Q4_K_M.gguf \
  --host 0.0.0.0 --port 8080 \
  --ctx-size 32768 \
  -np 1 \
  --reasoning off \
  --flash-attn on \
  --n-gpu-layers 99 \
  --seed 42

Install the harness and run the accuracy gate on its own — roughly three hours on an edge device:

git clone https://github.com/mlcommons/endpoints.git
cd endpoints
pip install -e ".[bfcl]"
inference-endpoint --help

cd examples/11_Edge_Agentic_Example/
inference-endpoint benchmark from-config \
  --config online_edge_full_run.yaml \
  --accuracy-only

Drop --accuracy-only to run performance and accuracy back to back — the mandated combined run, roughly 5.5 hours. Launch that one from the repository root, because the combined config loads the performance dataset by a repo-root-relative path while the accuracy-only run self-downloads its dataset:

inference-endpoint benchmark from-config \
  --config examples/11_Edge_Agentic_Example/online_edge_full_run.yaml

Set model_params.name to whatever your server reports before running either. Both scores land in results/edge_agentic_full_run/: the gated BFCL figure under accuracy/accuracy_results.json as the bfcl_v4::function_calling entry, the inline performance checker in scores.json, and the full latency picture in performance/result_summary.json. A run is valid when turns.missing is zero.

Note the arithmetic: 2 hours 37 minutes for the reference performance phase plus roughly 3 hours for the accuracy gate lands close to the 5.5-hour combined figure MLCommons quotes. The two published numbers are consistent, which is a small but genuine confidence signal.

Running the NVIDIA stack

The submission implementation is on the TensorRT Edge-LLM release/0.9.1-mlpinf branch, which includes export settings, engine build commands, server configuration and client configuration.

git clone --branch release/0.9.1-mlpinf \
  https://github.com/NVIDIA/TensorRT-Edge-LLM.git
cd TensorRT-Edge-LLM
git submodule update --init --recursive

huggingface-cli download \
  centml/Qwen3.6-27B-NVFP4-W4A4-mlpinf \
  --local-dir "$WORK/Qwen3.6-27B-NVFP4-W4A4-mlpinf"

Export with the tree-MTP interface, then build the base and draft engines per mlperf/README.md:

$VENV/bin/python -m tensorrt_edgellm.scripts.export \
  "$WORK/Qwen3.6-27B-NVFP4-W4A4-mlpinf" \
  "$WORK/onnx" \
  --mtp-tree-base --skip-visual

The --mtp-tree-base flag is what makes the tree-structured draft interface available; without it you get linear MTP at best. Launch the OpenAI-compatible server and point the same MLCommons harness at it:

export REPO="$PWD"
export VENV=/path/to/venv-edgellm-export
export WORK=/path/to/mlperf-artifacts
bash mlperf/serve_edgellm.sh

NVIDIA’s documented harness install uses the dev extra alongside bfcl — pip install -e ".[dev,bfcl]" — where the MLCommons example uses ".[bfcl]" alone. Either resolves the scoring dependency; the dev extra pulls additional tooling. The run itself uses temperature 0, seed 42, reasoning disabled, concurrency 1, matching the reference configuration exactly. That matching is what makes the comparison legitimate.

If you intend to submit rather than just measure, scripts/publish_submission.py reformats a run directory for the upstream MLPerf submission checker, which reads performance/result_summary.json, accuracy/accuracy_results.json and config.yaml directly.

Trade-offs, gotchas, and what this benchmark does not tell you

Take the 6.4x at exactly the weight it deserves: it is a tuned stack against a reference stack, on one vendor’s silicon, submitted by that vendor. Every element of that sentence constrains the conclusion.

It is not a silicon comparison. Both runs used a Jetson AGX Thor developer kit. Nothing in this result compares Thor to a Qualcomm, AMD, Apple, or Rockchip part, and nothing in it establishes that Thor is the fastest edge device for agentic inference. What it establishes is that on this device, the gap between a default deployment and an optimised one is 6.4x. That is a statement about software maturity.

The reference was chosen to be plain, not to be competitive. MLCommons needed a reproducible baseline that anyone could stand up, and llama.cpp with a Q4_K_M GGUF is the obvious choice. It is not a tuned llama.cpp — no prefix caching for this workload shape, no speculative decoding, weight-only quantisation. A llama.cpp deployment configured with prompt caching enabled would close part of the gap. So would vLLM, SGLang, or MLC on the same hardware. None of those comparisons exist in this round, and their absence is the single biggest gap in what we can conclude.

There is a disclosed calibration overlap. Submitters who quantise the reference model themselves calibrate with bfcl_calib.jsonl, a 364-record sample of BFCL v4 single-turn prompts. MLCommons discloses that roughly 30% of those prompts also appear in the ~995-sample accuracy gate. The set is used only to estimate quantiser scales — weight, activation and KV-cache — and is not scored. Calibration is not training, and estimating a scale factor is a far weaker form of exposure than gradient updates. But a 30% overlap between calibration and evaluation is a real methodological wrinkle, MLCommons was right to disclose it, and it belongs in any honest reading of a quantised submission’s accuracy score.

The accuracy comparison has an ambiguity. NVIDIA reports “BFCL overall accuracy 87.94%.” The reference’s two gated metrics are an overall single-turn score of 86.23% and a non_live-normalised score of 87.96%. Which of those NVIDIA’s figure is directly comparable to is not stated in its post. If it is the overall metric, the submission beat the reference by 1.7 points while running 6.4x faster, which would be a notable result in itself. If it is the normalised metric, the submission is 0.02 points below reference — comfortably inside the gate, and a fine outcome, but a different story. Do not assume either reading without checking the submission detail.

Single-stream, single-slot, MAXN. The submission ran SingleStream at concurrency 1 on maximum power mode. That is the right scenario for an on-device agent serving one user, and it is the scenario the benchmark defines. It tells you nothing about batched serving on an edge box handling several agents, and nothing about behaviour in a constrained power mode — which is the mode a battery-powered or passively cooled deployment will actually use. Sustained thermal behaviour over a 24-minute run at MAXN is not reported.

Gated accuracy is single-turn; the workload is multi-turn. This is the sharpest structural criticism available against MLPerf Edge Agentic Inference as currently defined. The performance phase measures a multi-turn agent loop, but the accuracy gate is single-turn only, because MLCommons found the sampled multi-turn subsets too noisy to gate on. Multi-turn error compounds in ways single-turn scoring cannot detect — a model that handles each isolated call correctly can still drift, lose track of state, or repeat a failed action across twenty turns. The inline IoU checker partially covers this, but it scores against recorded trajectories rather than evaluating task success. MLCommons is transparent about the limitation; it remains the thing a v6.2 or v7.0 round most needs to fix.

What it implies for non-NVIDIA edge silicon

The four optimisations decompose into one that is hardware-specific and three that are not.

NVFP4 requires a Blackwell-class GPU — the format is supported in hardware on Thor. Other silicon has analogous but not identical options: INT4 weight-only with INT8 activations, various vendor block-float formats, or FP8 where supported. A W4A4 scheme that holds accuracy is not universally available, and this is the piece a non-NVIDIA platform genuinely cannot copy.

Prefix caching with correct hybrid-state restoration is a runtime engineering problem, not a silicon feature. Any stack can implement it. Tree-based multi-token prediction likewise needs draft weights and a verification implementation, not special instructions — the target model verifies a tree in one forward pass using ordinary attention masking. And the finding that reasoning tokens cost ~60% more wall-clock for no accuracy gain on function calling applies to every platform.

The honest read for an architect on non-NVIDIA hardware is therefore encouraging rather than discouraging: most of the 6.4x is portable. If your edge stack is running a quantised model through a runtime with no prefix caching and no speculative decoding, a large multiple is sitting on the table regardless of who made your accelerator. Our survey of on-device LLM runtimes is a reasonable starting point for working out which of these features your current runtime actually supports.

Practical Recommendations

If you are running or planning an on-device agent, the useful move is to treat MLPerf Edge Agentic Inference as a diagnostic rather than a scoreboard. Run the reference on your own hardware first. It costs about 5.5 hours of wall-clock for the combined phase and gives you a defensible baseline on your silicon, under your thermal conditions, with your memory budget — which no published submission can give you.

Then audit your stack against the four levers in order of expected return. Prefix caching first: it is the largest single win on an agentic workload and it is available in most modern runtimes. Verify it actually works for your model’s architecture rather than trusting the feature flag, especially for hybrid models with recurrent state. Speculative decoding second, since function-calling output is unusually predictable and benefits more than prose generation does. Activation quantisation third, because it is the most invasive and the most likely to cost accuracy. Reasoning mode last — turn it off for tool-calling workloads and measure, because on this workload it bought nothing and cost 60% more time.

A short checklist before you commit to a device:

• Reproduce the MLCommons reference on your target hardware and record the inline IoU and wall-clock figures as your baseline.
• Confirm zero missing turns; a fast invalid run is not a result.
• Check whether your runtime restores recurrent state as well as KV pages if your model is hybrid.
• Measure at the power mode you will actually deploy at, not at maximum.
• Read the p99 per-turn latency, not the average — an agent loop is gated by its worst turn.
• Keep reasoning disabled for function calling unless your own measurement contradicts MLCommons’.
• Treat any vendor multiple as tuned-versus-reference until you see the reference tuned too.

Frequently Asked Questions

What is the MLPerf Edge Agentic Inference benchmark?

It is a benchmark introduced in MLPerf Inference v6.1 on 16 September 2026 that measures agentic AI workloads on edge devices. It runs two phases against one OpenAI-compatible endpoint: a performance phase replaying 20 recorded software-engineering agent conversations totalling 1,007 turns with contexts reaching about 23.5K tokens, and an accuracy phase gating on roughly 995 sampled Berkeley Function Calling Leaderboard v4 single-turn prompts. Both phases share a single configuration so performance and accuracy cannot be measured under different settings.

Is the 6.4x speedup a comparison between Jetson Thor and other hardware?

No. Both the TensorRT Edge-LLM submission and the llama.cpp reference ran on the same Jetson AGX Thor developer kit with the same 27B model. The 6.4x measures the difference between a heavily optimised serving stack and a deliberately plain reference implementation on identical silicon. It says nothing about how Thor compares with edge accelerators from other vendors, and no such comparison exists in this round.

What model does MLPerf Edge Agentic Inference use?

Qwen3.6-27B. The MLCommons reference serves it as a Q4_K_M GGUF of roughly 16.8 GB via llama.cpp, requiring about 24 GB of memory once the 32768-token KV cache is accounted for. NVIDIA’s submission used the same model quantised to NVFP4 for weights and activations with an FP8 KV cache, from a published calibrated checkpoint. Accuracy is treated as hardware-independent because decoding is deterministic at temperature 0 with a fixed seed.

Why does KV cache reuse matter so much for agents?

Because each agent turn resends nearly the entire prior conversation plus a small amount of new text. Without reuse, the shared history is prefilled on every turn and the cost grows as the conversation lengthens. With reuse, about 96% of prompt tokens were served from hot cache in NVIDIA’s run — only about 0.5M of 13.6M total prompt tokens were actually prefilled. That collapses prefill to a minor cost and makes the workload decode-dominated.

What is tree-based multi-token prediction?

It is a speculative decoding variant that organises several high-probability draft continuations into a tree and verifies all of them in one target-model forward pass, accepting whichever path matches. NVIDIA’s MLPerf configuration used 8 draft steps, the top 2 candidates at each depth, and a 16-node verification tree, reporting roughly 40% more decoding gain than linear MTP with 3 draft steps. It suits function calling because tool names and JSON structure are predictable while argument values are not.

Should I enable reasoning mode for a tool-calling agent?

MLCommons’ own measurement argues against it on this workload. On Jetson AGX Thor with the Q4_K_M reference, the 1,007-turn run scored an inline IoU of 0.6335 in 2 hours 37 minutes with reasoning off, versus 0.6374 in 4 hours 13 minutes with it on — an accuracy difference inside run-to-run noise for roughly 60% more wall-clock time. The reference configuration therefore runs with reasoning disabled. Measure on your own workload before generalising.

Further Reading

• TensorRT-LLM vs llama.cpp on Jetson — the two runtimes compared outside a benchmark harness.
• Jetson Thor vs Jetson Orin AGX — the hardware generation the submission ran on.
• INT4 vs INT8 vs FP8 edge NPU quantization — the accuracy and footprint trade-offs behind the NVFP4 choice.
• On-device LLM runtimes: llama.cpp vs MLC vs ONNX — which runtimes actually implement prefix caching and speculative decoding.
• NVIDIA: TensorRT Edge-LLM completes the MLPerf Edge Agentic benchmark 6.4x faster on Jetson AGX Thor — the submission analysis and reproduction commands.
• MLCommons Edge Agentic reference implementation — the benchmark definition, accuracy targets and pointers to the runnable example in mlcommons/endpoints.

By Riju — about