TensorRT-LLM vs llama.cpp on Jetson: Throughput, VRAM & Setup (2026)

Every team that puts an LLM on a Jetson board eventually hits the same fork in the road: tensorrt-llm vs llama.cpp jetson is the question that decides whether your next two weeks go into compiling CUDA graphs or pulling a GGUF file and hitting run. Both paths get tokens out of an Orin’s Ampere GPU, but they get there through opposite philosophies — one compiles a hardware-locked engine for maximum throughput, the other interprets a portable weight format for maximum flexibility. Picking wrong costs you either a fragile deployment pipeline or silicon you’re leaving on the table.

What this covers: build and setup complexity for each engine, real-world quantization support, measured tokens/sec and time-to-first-token on Orin hardware, how Jetson’s unified memory architecture changes the VRAM calculus versus a discrete GPU, model-conversion friction, and a decision framework for production deployment versus rapid prototyping versus multi-model flexibility.

Context and Background

Jetson boards occupy an unusual middle ground in the inference landscape. They run a full CUDA stack — the same instruction set family as datacenter A100s and H100s, just at a fraction of the SM count and power envelope — but they ship with unified LPDDR memory shared between CPU and GPU rather than dedicated VRAM. That single architectural fact reshapes almost every assumption engineers bring from server-side LLM serving. A framework’s discrete-GPU playbook (pin the model to a VRAM budget, leave system RAM alone) doesn’t map cleanly onto a device where a 64GB AGX Orin’s entire memory pool is fair game for either compute unit, contested at the OS level.

Into that environment land two very different inference engines. TensorRT-LLM is NVIDIA’s own compiler-based stack: it takes a Hugging Face checkpoint, runs it through quantization and graph-optimization passes, and emits a TensorRT engine — a binary blob compiled for one exact GPU architecture, one exact TensorRT version, one exact precision configuration. llama.cpp, by contrast, is a portable C/C++ inference runtime built around the GGUF format; it interprets quantized weights at load time with a CUDA backend for GPU offload, and the same GGUF file runs — sometimes with a rebuild, sometimes not — on an x86 workstation, a Raspberry Pi, or a Jetson.

There’s a second variable that most comparisons gloss over: Jetson’s power mode. Both engines’ throughput numbers shift meaningfully depending on whether the board is running in a low-power mode (15W) or the high-performance MAXN/MAXN SUPER mode, because clock rates on both the CPU cluster and the GPU’s streaming multiprocessors scale with the selected nvpmodel profile. A benchmark run at 15W and one at MAXN on the same board, same model, same engine can differ by well over 30% in tokens/sec — which is one more reason single-number “X tok/s on Jetson” claims floating around forums should be read with the power mode attached, or discounted if it isn’t stated.

This piece assumes you already have a JetPack-flashed Orin (or are evaluating Thor) and are deciding which runtime to standardize on. For the surrounding ecosystem — MLC-LLM, ONNX Runtime, and other on-device options — see our companion breakdown of on-device LLM runtimes: llama.cpp vs MLC vs ONNX. For background on how NVIDIA frames TensorRT-LLM’s build-then-run design outside the edge context, NVIDIA’s own TensorRT-LLM build workflow documentation is the canonical source and is worth reading before you commit engineering time to either path.

Setup, Build Complexity, and the Jetson Support Gap

Short answer: llama.cpp is a cmake build and a model download away from serving tokens; TensorRT-LLM on Jetson requires a JetPack-matched branch, a multi-stage compile, and per-model engine builds that can take 10–90 minutes each. The gap isn’t cosmetic — it reflects two fundamentally different deployment models, and it determines how many engineer-hours you burn before first token.

Figure 1: TensorRT-LLM’s path runs a Hugging Face checkpoint through ModelOpt export and a trtllm-build compile into a device- and version-locked engine before it can serve a single token; llama.cpp loads a GGUF file directly into llama-server with GPU layers offloaded via a runtime flag, with no per-model compile step in between.

The diagram makes the core asymmetry visible: TensorRT-LLM front-loads cost into a compile phase in exchange for a highly optimized, fixed execution plan at serve time, while llama.cpp defers essentially all of that cost, trading some steady-state throughput headroom for the ability to point the same binary at a new model file and start serving within seconds. Neither path is strictly better — which one wins depends entirely on how often the model underneath your service is expected to change.

llama.cpp: clone, build, run

llama.cpp’s Jetson path is close to its path on any other CUDA-capable Linux box. You clone the repo, build with GGML_CUDA=ON, and point the binary at a GGUF file — either downloaded pre-quantized from Hugging Face or produced locally with the convert_hf_to_gguf.py script and llama-quantize. There’s no separate compile step per model; the same binary loads any GGUF file whose architecture is supported by the current build.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=87
cmake --build build --config Release -j 8

./build/bin/llama-server \
  -m models/llama-3.2-3b-instruct.Q4_K_M.gguf \
  -ngl 99 \
  -c 8192 \
  --host 0.0.0.0 --port 8080

-ngl 99 offloads all transformer layers to the GPU; -DCMAKE_CUDA_ARCHITECTURES=87 targets Ampere-generation Orin SMs specifically, which matters for build time and avoids JIT-compiling PTX at first launch. Total time from clean JetPack image to first served token is typically well under an hour, most of which is the CUDA build itself. Teams report OOM and layer-offload edge cases on 8GB Orin Nano boards with larger context windows, but the failure mode is a clear error, not a silent build break.

TensorRT-LLM: JetPack-locked, version-pinned, per-model compiles

TensorRT-LLM’s Jetson story is more constrained than its datacenter story, and the constraint is worth naming explicitly because it’s easy to assume TensorRT-LLM support is uniform across NVIDIA’s product line. It is not. Initial TensorRT-LLM support for Jetson AGX Orin landed via a dedicated v0.12.0-jetson branch tied to JetPack 6.1, documented on the Jetson AI Lab TensorRT-LLM page. Two limitations matter for planning:

• Version lag. Mainline TensorRT-LLM has moved past 1.0 with substantial architecture changes; the Jetson branch is still anchored at 0.12.0. Features, quantization recipes, and bug fixes landing in mainline do not automatically appear on Jetson.
• Device scope. Jetson architectures are explicitly excluded from the TensorRT-LLM main branch, per the project’s own GitHub issue tracking Jetson device support; AGX Orin has the most mature path, while community discussion on Orin NX/Nano support and a standing request for TensorRT-LLM 1.0+ support on Jetson both indicate that support on the smaller boards is narrower and less consistently maintained than on AGX Orin.
• Thor is a different story entirely. On Jetson AGX Thor, classic TensorRT-LLM is not the supported path — NVIDIA has moved developers to TensorRT Edge-LLM, a separate, purpose-built runtime for the Blackwell-generation Jetson boards, confirmed in NVIDIA developer forum threads on TensorRT-LLM and AGX Thor. If your target hardware is Thor, this comparison’s TensorRT-LLM column effectively becomes “TensorRT Edge-LLM,” a related but distinct codebase with its own build tooling.

The build itself, once you’re on the correct branch and JetPack version, follows TensorRT-LLM’s general workflow: export the checkpoint (optionally through NVIDIA’s ModelOpt toolkit for AWQ or INT4/INT8 weight-only quantization, per the project’s quantization documentation), then run trtllm-build to compile the checkpoint into a device-specific engine.

# Export checkpoint with INT4-AWQ (ModelOpt)
python quantize.py \
  --model_dir ./qwen3-4b-instruct \
  --qformat int4_awq \
  --output_dir ./qwen3-4b-awq-ckpt

# Compile the checkpoint into a Jetson-specific TensorRT engine
trtllm-build \
  --checkpoint_dir ./qwen3-4b-awq-ckpt \
  --output_dir ./engines/qwen3-4b-awq \
  --gemm_plugin auto \
  --max_batch_size 1 \
  --max_input_len 4096

That compile step is the crux of the trade-off. Engines are architecture- and version-locked: an engine built on one Orin module for one TensorRT/JetPack combination will not load on a different module revision or a different TensorRT minor version without a rebuild, a constraint explicit in NVIDIA’s own deployment guidance. Budget real time for this — TensorRT-LLM engine compiles commonly run 10–90 minutes per model/precision combination even on workstation-class hardware, and Jetson’s more limited CPU and memory bandwidth push builds toward the higher end of that range. If you swap base models weekly during prototyping, you are compiling, not iterating.

Containerized setup as a third path

Both ecosystems increasingly ship as prebuilt Docker containers rather than bare-metal builds, largely through the community-maintained jetson-containers project, which packages CUDA-toolkit-matched images for llama.cpp, Ollama, and TensorRT-LLM against specific JetPack releases. This shifts the setup burden from “compile CUDA code correctly on your board” to “pull the correctly tagged image,” which is usually faster and more repeatable for llama.cpp, and marginally faster for TensorRT-LLM only in the sense that you skip toolchain installation — the per-model trtllm-build engine compile still has to happen inside the container, on your device, every time. Containers do not eliminate the engine-portability problem discussed below; a TensorRT engine baked into one image still won’t load against a different TensorRT runtime version in a different image.

Quantization, Throughput, and VRAM: What the Numbers Actually Say

Two caveats before the numbers: first, Jetson LLM benchmarks in the wild are overwhelmingly community-reported rather than NVIDIA-audited, so treat every figure below as directional, not a guarantee for your exact model, prompt length, and power mode. Second, TensorRT-LLM and llama.cpp are rarely benchmarked head-to-head on the same model, quantization scheme, and Jetson SKU in public sources — the comparison below stitches together the best available same-family data points and labels every estimate.

Figure 2: How TensorRT-LLM’s fixed engine allocation and llama.cpp’s dynamic mmap-backed allocation both draw from the same LPDDR unified memory pool shared with the OS and CPU workloads on Jetson.

The figure above captures the single biggest mental-model shift for engineers coming from discrete-GPU deployment: on Jetson, “VRAM” and “system RAM” are the same 64GB (or 32GB, 16GB, 8GB) physical pool. There’s no PCIe bus separating a dedicated GPU memory bank from host memory — CPU and GPU contend for the same LPDDR5 controller. That means a model whose weights plus KV cache fit inside total system RAM can, in principle, run fully GPU-resident; conversely, background services, desktop compositors, and OS overhead all eat directly into the budget you’d otherwise hand to the model. On a 64GB AGX Orin the practical rule of thumb reported in the community is that model file size plus roughly 1.5GB of KV cache overhead needs to clear available RAM, which in turn means a 64GB module can plausibly GPU-offload a 70B model at 4-bit quantization, with headroom shrinking fast as context length grows.

Quantization format support

Format	TensorRT-LLM (Jetson)	llama.cpp (GGUF)
INT4 weight-only	Yes, via ModelOpt	Yes, Q4_0/Q4_K_M/Q4_K_S and others
INT4-AWQ	Yes, native ModelOpt path	Not natively; AWQ weights must be re-quantized to GGUF
INT8 weight-only	Yes	Yes, Q8_0
FP8	Supported on Blackwell/Hopper-class TensorRT-LLM; limited on Orin’s Ampere SMs	No FP8 GPU kernel path on Jetson-class Ampere
K-quants (Q4_K_M, Q5_K_M, etc.)	Not applicable — different quantization system entirely	Yes, this is llama.cpp’s primary format family
Mixed CPU/GPU layer split	No — engine is fully compiled for one execution plan	Yes, -ngl controls exact GPU layer count, degrades gracefully

The practical read: llama.cpp’s K-quant family (Q4_K_M in particular) is the de facto standard for community-quantized GGUF releases, so model availability is rarely the bottleneck. TensorRT-LLM’s AWQ and INT4 weight-only paths through ModelOpt can produce tighter, better-calibrated quantization for a given accuracy target, but each new base model requires you to run the export-and-compile pipeline yourself rather than downloading a ready-made artifact — Qwen3-4B-Instruct built with INT4-AWQ on Jetson Orin Nano is a documented example, and NVIDIA’s own guidance notes the 4B model shrinks to roughly 2GB of weights under INT4-AWQ, leaving comfortable KV cache headroom on an 8GB Nano.

Model conversion pain

Getting a model from a Hugging Face checkpoint into each runtime’s native format carries different friction. For llama.cpp, the path is convert_hf_to_gguf.py followed by llama-quantize — a script that understands a wide, actively maintained list of model architectures, and for popular model families the conversion step is often unnecessary entirely because a community member has already uploaded a quantized GGUF to Hugging Face. The main failure mode is architecture support lag: a brand-new model family (a novel attention variant, an unusual MoE routing scheme) may need a llama.cpp code change before conversion works at all, which shows up as GitHub issues within days of a notable model release.

For TensorRT-LLM, conversion runs through ModelOpt’s export tooling into a TensorRT-LLM checkpoint format, then trtllm-build compiles that checkpoint into an engine. This path supports fewer architectures out of the box on the Jetson branch specifically, and because the branch trails mainline TensorRT-LLM, a model architecture that mainline supports may not yet be present in the v0.12.0-jetson codebase — a subtler and easier-to-miss failure mode than llama.cpp’s, since the export step can appear to succeed before the engine-compile step fails on an unsupported operator. In both cases, plan a conversion smoke-test into your model-evaluation pipeline rather than assuming a checkpoint that works on a workstation GPU will convert cleanly on Jetson.

Measured throughput

Figure 3: Community-reported llama.cpp generation throughput across Jetson Orin Nano and AGX Orin at various model sizes and quantization levels. These are single-run, community-sourced figures rather than a controlled benchmark suite — treat them as directional, not a guarantee for your exact configuration.

The most consistent same-hardware-family data available comes from llama.cpp community benchmarks. On a Jetson Orin Nano, one detailed community benchmark run reported:

• Llama 3.2 3B, Q4_K_M: ~580 tok/s prompt-eval, ~28.7 tok/s generation
• Llama 3.2 7B-class model, Q4_K_M: ~285 tok/s prompt-eval, ~14.2 tok/s generation
• Mistral 7B, Q4_K_M: ~301 tok/s prompt-eval, ~15.1 tok/s generation

(Source: community Jetson Orin Nano LLM benchmark writeups; treat as illustrative for Q4_K_M 3B–7B class models rather than a guaranteed floor for your exact stack.)

On the more capable Jetson AGX Orin, forum-reported llama.cpp runs with CUDA 12.9 have hit 300+ tok/s aggregate throughput on a 7B Mistral-class model under favorable conditions (short context, batched decode), and a much larger Qwen3-VL-30B-A3B mixture-of-experts model has been reported around 37 tok/s generation — both figures from the NVIDIA developer forum’s LLM library recommendations thread, and both should be read as single-run community numbers, not vendor-audited benchmarks.

Head-to-head, apples-to-apples TensorRT-LLM-versus-llama.cpp throughput numbers on Jetson specifically are thin in public sources as of this writing — most TensorRT-LLM Jetson coverage focuses on setup and enablement (the Hackster.io TensorRT-LLM on AGX Orin walkthrough is representative) rather than rigorous benchmark tables. Directionally, TensorRT-LLM’s compiled-kernel, fused-op execution plan is expected to outperform llama.cpp’s more general CUDA backend on raw generation throughput for a fixed model and precision — this is consistent with TensorRT-LLM’s design goals and with datacenter-class comparisons — but the exact multiplier on Orin-class Ampere hardware is not something we can cite a solid published number for, and you should benchmark your own model and prompt shape before trusting any specific percentage. Related quantization research (AWQ/TinyChat) reports 1.2–3.0x speedup over baseline systems for 4-bit quantized inference on Jetson Orin, which gives a rough sense of the ceiling compiled, quantization-aware execution can reach relative to a naive baseline — not a direct TensorRT-LLM-vs-llama.cpp delta.

Time-to-first-token and the compile tax

TTFT behaves differently across the two engines in a way raw tok/s tables don’t capture. llama.cpp’s TTFT is dominated by prompt processing speed — the prompt-eval numbers above (285–580 tok/s depending on model size) are a reasonable proxy, since there’s no separate compile phase; the first request after server startup pays only model-load latency. TensorRT-LLM’s effective TTFT for a new model includes the one-time engine build (10–90 minutes), but once an engine is compiled and cached, inference-time TTFT benefits from graph fusion and reduced kernel-launch overhead per decode step — the same properties that help steady-state throughput. For a service that boots once and runs for weeks, the compile tax amortizes to zero. For a research loop that swaps checkpoints hourly, it dominates.

Methodology caveats worth internalizing

None of the community numbers cited here come from a controlled, apples-to-apples harness that fixes prompt length, batch size, power mode, and JetPack/CUDA version simultaneously across both engines — that kind of rigorous, vendor-neutral benchmark suite largely doesn’t exist yet for Jetson-class hardware the way MLPerf does for datacenter GPUs. A separately useful data point comes from an Orin Nano Super power-mode sweep across eight small models, which found the 25W power profile delivered roughly 43% more tokens/sec than the 15W profile while also landing on better tokens-per-joule efficiency than the board’s maximum power setting — a reminder that “fastest” and “most efficient” are not the same optimization target, and that power-mode selection is itself a tuning knob independent of which inference engine you pick. If throughput numbers matter for a purchasing or architecture decision, re-run the benchmark yourself, pin the power mode explicitly, and report it alongside the number.

Trade-offs, Gotchas, and What Goes Wrong

Engine portability is the sharpest edge. A TensorRT-LLM engine compiled on one Orin module is not guaranteed to load on a different Orin module revision, a different JetPack/TensorRT minor version, or a different Jetson SKU — NVIDIA’s own documentation is explicit that engines must be built on the target device. Ship a fleet of boards with slightly different JetPack point releases and you can end up maintaining a build matrix instead of a single artifact. llama.cpp’s GGUF files carry no such constraint; the same file runs across driver and JetPack versions as long as the binary itself is rebuilt against the current CUDA toolkit, which is a much cheaper operation.

The version-lag problem compounds over time. Because the Jetson TensorRT-LLM branch trails mainline (0.12.0 versus 1.0+), quantization recipes, model architecture support, and performance fixes that land upstream can take months to reach Jetson, if they reach it at all. If your target model architecture is very new, check the Jetson-specific branch’s supported model list before committing — don’t assume mainline TensorRT-LLM documentation applies.

Unified memory contention is easy to underestimate. Because GPU and CPU share one LPDDR pool, a background container, a desktop environment, or a second model server can silently steal memory bandwidth and capacity from your inference process. Community reports of OOM failures and CUDA memory allocation errors on Orin Nano boards are common enough to be a recurring forum topic — set explicit -ngl layer counts rather than assuming full offload, and headroom-budget conservatively (model size + ~1.5GB KV cache + OS reserve, not model size alone) before sizing context windows.

Multi-model flexibility favors llama.cpp by a wide margin. Because TensorRT-LLM engines are compiled per model, swapping between three or four candidate models during evaluation means three or four multi-minute-to-hour compile cycles. llama.cpp swaps models by pointing the server at a different GGUF file — seconds, not minutes.

Thor is not a drop-in upgrade path for TensorRT-LLM users. If your roadmap includes moving from Orin to Thor, plan for a runtime migration, not just a rebuild — classic TensorRT-LLM’s Jetson support does not carry forward to Thor the way you might expect from a straightforward architecture bump; TensorRT Edge-LLM is the supported successor there, and it is a separate project with its own tooling and model coverage.

Quantization accuracy is not free on either path. Aggressive INT4 quantization — whether via llama.cpp’s Q4_K_M or TensorRT-LLM’s INT4-AWQ — trades measurable accuracy for throughput and memory headroom; validate task-specific accuracy (not just perplexity) before locking a quantization level into a production config, especially for reasoning-heavy or code-generation workloads where small numerical errors compound across long generations.

Dependency footprint and image size diverge sharply. A llama.cpp build is a handful of compiled binaries and a GGUF file; the whole deployable artifact for a 7B model can sit comfortably under 5GB. A TensorRT-LLM toolchain — CUDA, TensorRT, ModelOpt, and the Python build tooling needed to run trtllm-build — pulls in a much heavier dependency graph, and if you’re building inside a container, the base image alone commonly runs several gigabytes before you’ve added a single model. On boards with limited eMMC or NVMe capacity, that difference affects how many models and engine variants you can realistically keep resident on-device at once, and it lengthens the time needed to provision a fresh board from a flashed JetPack image to a working inference service.

Debugging failure modes differs in kind, not just degree. When llama.cpp fails, it typically fails loudly and immediately — an unsupported GGUF architecture, an out-of-memory allocation, a missing CUDA library — with a stack trace that points at the proximate cause. TensorRT-LLM failures more often surface during the build step itself, sometimes deep inside graph-optimization or plugin-selection logic, and a failed trtllm-build can burn most of an hour before you learn the checkpoint or quantization config was incompatible. Budget debugging time accordingly, and prefer building one model end-to-end before batch-scripting engine builds for several candidates.

Practical Recommendations

Figure 4: A simplified decision path — how often you swap models and which Jetson board you’re targeting are the two variables that matter most; AGX Orin on JetPack 6.1 is the only board with a mature TensorRT-LLM path, and Thor routes to TensorRT Edge-LLM instead of classic TensorRT-LLM entirely.

If you’re standing up a fixed, single-model production deployment on AGX Orin and can tolerate a JetPack version pin, TensorRT-LLM’s compiled-engine path is worth the setup cost — you pay the compile tax once and keep the throughput and latency benefits for the life of the deployment. If you’re on Orin Nano, Orin NX, or need to move fast across several candidate models before locking one in, llama.cpp is the pragmatic default: minutes to first token, painless model swaps, and quantization coverage (Q4_K_M and friends) that tracks the broader open-weights ecosystem closely. If Thor is in your near-term hardware roadmap, prototype on llama.cpp now and evaluate TensorRT Edge-LLM specifically for Thor rather than assuming your Orin-era TensorRT-LLM investment carries forward.

A pattern that works well in practice for teams that genuinely need both: prototype and select the model with llama.cpp, where iteration is cheap, then port only the finalized model to TensorRT-LLM once you’re confident it’s the one going to production on AGX Orin. This front-loads the flexibility where it’s valuable (model selection, prompt-format validation, early accuracy checks) and defers the compile tax to the point where you’re compiling exactly once for exactly one model, rather than paying it repeatedly during exploration. Treat any decision here as reversible for the first few weeks of a project and revisit it once real production traffic patterns — request volume, concurrency, latency SLAs — are known, since those numbers, not benchmark tables, should ultimately decide whether the compiled-engine investment pays for itself.

Checklist before you commit to either path:

• [ ] Confirm your exact Jetson SKU (Orin Nano / NX / AGX Orin / AGX Thor) and JetPack version — TensorRT-LLM support and the correct branch depend on both.
• [ ] Decide how often you expect to swap base models; if it’s more than monthly, weight toward llama.cpp.
• [ ] Budget real engineer-hours for TensorRT-LLM’s per-model compile cycle (10–90 minutes) into your deployment timeline, not just your setup timeline.
• [ ] Set explicit -ngl / layer-offload and context-length limits rather than relying on defaults, given unified-memory contention risk.
• [ ] Benchmark your own model, quantization level, and prompt length on your target board — published numbers above are directional, not guarantees.
• [ ] If targeting Thor, validate TensorRT Edge-LLM’s model support list before assuming feature parity with Orin-era TensorRT-LLM.
• [ ] Validate task-specific accuracy after quantization, not just tokens/sec — a faster wrong answer is not a win.

Frequently Asked Questions

Does TensorRT-LLM run on all Jetson Orin boards?

Not uniformly. Documented, relatively mature support centers on Jetson AGX Orin via a dedicated JetPack 6.1-tied branch. Support for Orin NX and Orin Nano exists in community discussion and testing but is less consistently documented than AGX Orin’s path, and Jetson devices in general are excluded from TensorRT-LLM’s main branch. Always check the specific Jetson branch’s supported-device list before planning a deployment on a smaller module.

Is llama.cpp actually GPU-accelerated on Jetson, or does it fall back to CPU?

It’s genuinely GPU-accelerated when built with GGML_CUDA=ON and run with -ngl set high enough to offload the model’s layers. Because Jetson uses unified memory, that GPU offload draws from the same physical RAM as the CPU rather than a separate VRAM pool, but the compute itself runs on the Ampere SMs via CUDA kernels, not on the CPU cores. If -ngl is left too low, or a build silently falls back to a CPU-only backend because the CUDA toolkit wasn’t detected during compilation, throughput drops sharply and generation speed can fall to a fraction of GPU-offloaded numbers — check nvidia-smi-equivalent tooling like tegrastats during a test run to confirm GPU utilization before trusting a benchmark.

Which engine has better time-to-first-token for a chat application?

For a long-running service, TensorRT-LLM’s compiled kernels typically give favorable per-request latency once the one-time engine build is done. For anything involving frequent model swaps or rapid iteration, llama.cpp’s near-zero setup latency per model usually wins on effective TTFT across a development cycle, even if raw decode-step latency is somewhat higher.

Can I use AWQ-quantized weights with llama.cpp on Jetson?

Not directly — AWQ is TensorRT-LLM/ModelOpt’s native quantization scheme, and llama.cpp’s GGUF format uses its own quantization families (Q4_K_M, Q5_K_M, Q8_0, and so on). You’d need to re-quantize the original full-precision weights into a GGUF format rather than converting an existing AWQ checkpoint directly, since the two formats encode scale factors and grouping differently at the bit level. In practice this means treating AWQ and GGUF as two separate quantization pipelines running off the same base checkpoint, each with its own accuracy-validation pass, rather than expecting a lossless conversion between them.

How much unified memory headroom should I leave on a Jetson board for LLM inference?

A working rule reported in the community is model file size plus roughly 1.5GB for KV cache overhead, with additional headroom for OS and desktop-environment usage — treat that as a starting estimate, not a hard guarantee, and always confirm actual free memory under load with tegrastats or jtop before finalizing a context-length or batch-size setting.

Will TensorRT-LLM engines built for Jetson AGX Orin run on Jetson AGX Thor?

No. Engines are compiled for a specific TensorRT version and GPU architecture, and Thor’s Blackwell-generation SMs are a different target entirely from Orin’s Ampere SMs. Beyond the architecture mismatch, NVIDIA has directed Thor users toward TensorRT Edge-LLM as the supported runtime rather than continuing classic TensorRT-LLM on that platform, so plan for a runtime change, not just a re-compile, when migrating hardware generations. If your organization is standardizing on Thor for a future product line, it’s worth prototyping directly against TensorRT Edge-LLM’s tutorials rather than building institutional expertise in classic TensorRT-LLM workflows that won’t transfer.

Further Reading

• On-device LLM runtimes: llama.cpp vs MLC vs ONNX (2026)
• Jetson Thor vs Hailo-10H vs Coral: edge inference compared (2026)
• NVIDIA Jetson K3s edge AI cluster tutorial (2026)
• TensorRT-LLM build workflow — official NVIDIA documentation
• TensorRT Edge-LLM on Jetson — Jetson AI Lab tutorial

By Riju — about