RT-DETR vs YOLO: NMS-Free Object Detection and Edge Deployment

For most of a decade, the honest answer to “which real-time detector should I ship?” was “a YOLO, and tune the NMS thresholds.” That answer has changed twice. In 2023 Baidu’s RT-DETR showed that a transformer detector could match YOLO accuracy at YOLO speed without non-maximum suppression. In January 2026 Ultralytics answered by releasing YOLO26, a YOLO that also drops NMS. The rt-detr vs yolo question is no longer “transformer or convnet.” It is “which end-to-end design survives contact with my deployment target?”

That matters because post-processing is where edge pipelines quietly break. An NMS step has data-dependent latency, awkward export paths, and thresholds that silently trade accuracy for speed. If your detector runs on a Jetson, an NPU, or a CPU-bound gateway, those details decide whether the model meets its frame budget.

You will leave with a mechanism-level understanding of both families, the published numbers with their caveats, a TensorRT build recipe, and a decision framework you can defend in a design review.

What this covers: why NMS is a deployment problem, how RT-DETR and YOLO26 each remove it, what the sourced benchmarks do and do not show, a TensorRT deployment workflow, failure modes, and a decision tree.

Context and Background

Real-time object detection spent years on a stable recipe. A convolutional backbone and neck produce a dense grid of candidate boxes, a head scores every candidate, and non-maximum suppression (NMS) prunes overlapping duplicates. The YOLO family, from Redmon’s original through YOLOv8 and YOLO11, refined that recipe with better backbones, anchor-free heads, and stronger training losses.

DETR (Detection Transformer, 2020) took the opposite route. It treats detection as set prediction: a transformer decoder emits a fixed number of object queries, and a bipartite (Hungarian) matching loss forces each ground-truth object to be explained by exactly one query. No duplicates are produced, so no NMS is needed. The cost was slow convergence and heavy compute, which kept DETR out of real-time use until RT-DETR.

RT-DETR, introduced in the CVPR 2024 paper “DETRs Beat YOLOs on Real-time Object Detection”, redesigned the encoder and query initialization so that the end-to-end approach became fast enough to compete. RT-DETRv2 then added training and deployment improvements. In parallel, Roboflow’s RF-DETR and academic work such as D-FINE and DEIM pushed DETR-style accuracy higher, and Ultralytics folded an end-to-end head into YOLO26.

Two shifts make 2026 different from 2023. First, both camps now ship NMS-free detectors, so the old “transformers avoid NMS, YOLO needs it” argument is obsolete. Second, deployment tooling matured: TensorRT handles attention layers well on NVIDIA hardware, and runtime choices like those in our ONNX vs TFLite vs ExecuTorch vs Core ML comparison now matter as much as the model.

For readers choosing hardware first, our Hailo-10H vs Jetson Orin Nano comparison covers the accelerator side, and the backbone comparison for industrial inspection covers feature extractors. This post focuses on the detector head and the post-processing contract.

A note on sourcing. Every number below comes from the RT-DETR and RT-DETRv2 papers, Ultralytics documentation, or Roboflow’s published material, and each is attributed. Where a figure is vendor-reported or measured under a protocol I could not verify, I say so. I did not run these benchmarks myself.

Why NMS Is a Deployment Problem, Not Just an Algorithm

The standard framing says NMS is a cheap, harmless cleanup step. The RT-DETR authors measured it and disagreed.

NMS adds latency that depends on the scene

NMS sorts candidate boxes by score, then iteratively removes boxes whose intersection-over-union (IoU) with a higher-scoring box exceeds a threshold. Its cost grows with the number of surviving candidates, which depends on the confidence threshold, the IoU threshold, and how many objects are in the frame.

The RT-DETR paper quantifies this on YOLOv8. At a confidence threshold of 0.001 and an IoU threshold of 0.7, YOLOv8 reaches 52.9 AP, but NMS takes 2.36 ms. Raising the confidence threshold to 0.05 cuts NMS to 1.06 ms, but AP drops to 51.2. The paper’s point is that benchmark speed numbers for NMS-based detectors usually exclude NMS time, or report it at a favourable threshold, so the quoted frames per second flatter the model.

Think about what 2.36 ms means on a 33 ms frame budget (30 fps). It is about 7 percent of the budget for a step that does nothing useful for accuracy. On a model that otherwise runs in 6 ms, it is a third of total latency.

NMS is awkward to export and quantize

NMS is a sequential, data-dependent algorithm. Inside a TensorRT or ONNX graph it becomes either a custom plugin, a dedicated operator with dynamic output shapes, or a separate CPU post-processing function. Each option has a cost: plugin version pinning (see our TensorRT 11 vs 10 plugin migration notes), dynamic shapes that defeat some optimizations, or a GPU-to-CPU copy and a Python loop.

The Ultralytics write-up on removing NMS makes the same argument from the product side: NMS “doesn’t map cleanly to specialized AI accelerators,” and NMS thresholds make production behaviour less predictable. That is a vendor blog, so treat the framing as motivation, not measurement. The engineering claim is nonetheless sound. A graph that ends in a fixed-shape tensor is easier to compile, profile, and verify than one that ends in a variable-length list.

Thresholds become hidden hyperparameters

With NMS, the IoU threshold is a model hyperparameter that is rarely tuned per deployment. Crowded scenes (people in a queue, boxes on a pallet, cells under a microscope) suffer from over-suppression when objects genuinely overlap. Sparse scenes tolerate duplicates. One global threshold cannot serve both, and teams discover this in production.

Figure 2: Where the variable-cost step lives. The classic path ends in NMS whose runtime grows with candidate count. The NMS-free path ends in a fixed top-K read.

Figure 2 contrasts the two post-processing contracts. In the classic path the last stage is data dependent. In the NMS-free path the last stage reads a fixed-size tensor and applies a score cutoff, so its cost is constant per frame.

NMS-free detection does not make this problem vanish; it moves it into training. A one-to-one assignment must learn which of several similar queries “owns” an object. When that learning is imperfect you get duplicates or misses that no threshold can fix. We return to this in the failure-modes section.

How NMS-Free Detection Works: Two Routes to One Box per Object

Both RT-DETR and YOLO26 produce one prediction per object by training with one-to-one label assignment. They differ in how the network is built around that idea.

Figure 1: RT-DETR pipeline. Attention runs only on the smallest feature map, cross-scale fusion stays convolutional, and 300 selected features seed the decoder queries.

Figure 1 shows the RT-DETR data path: a CNN backbone, an efficient hybrid encoder (AIFI then CCFF), top-K query selection, and a transformer decoder that outputs 300 boxes with class scores. The caller keeps predictions above a score threshold. There is no suppression step.

The direct answer

NMS-free detection means the network is trained so that each object produces exactly one confident prediction, which removes the suppression step at inference time. RT-DETR achieves this with a transformer decoder and Hungarian matching. YOLO26 achieves it with a one-to-one head trained alongside a conventional one-to-many head, and exports only the one-to-one path.

RT-DETR: the efficient hybrid encoder

Plain DETR encoders apply self-attention across every position of every feature scale, which is quadratic in the token count and far too slow for real time. The RT-DETR authors observed that most of that compute is redundant, and split the encoder into two parts.

AIFI (attention-based intra-scale feature interaction) applies a single transformer layer only to the S5 feature map, the lowest-resolution and most semantically rich scale. Self-attention on S5 is cheap because it has few tokens, yet it captures the global relations that high-level features need.

CCFF (CNN-based cross-scale feature fusion) then fuses scales using convolutional blocks, each containing three RepBlocks. This is the part YOLO engineers will recognize: it is a PAN-style neck built from re-parameterizable convolutions. The paper’s ablation reports that its chosen variant reaches 47.9 AP at 9.3 ms, compared with 46.4 AP at 12.2 ms for a variant that applied attention across scales. That is a 24 percent latency reduction with higher accuracy, which is the core efficiency win.

For deployment, the useful property is that only one small attention layer sits in the encoder. Most of the encoder is plain convolution that TensorRT fuses aggressively.

RT-DETR: uncertainty-minimal query selection

DETR-style decoders need initial object queries. Early DETRs used learned embeddings, which converge slowly. RT-DETR instead selects the top K = 300 encoder features and uses them as the initial queries, so the decoder starts from locations already likely to contain objects.

The refinement is the selection criterion. Ranking features by classification score alone can pick features with high class confidence but poor localization. The paper’s uncertainty-minimal selection folds the gap between predicted classification and predicted IoU into the loss, so selected features are good at both. Reported effects: AP rises from 47.9 to 48.7 in the ablation, the share of selected features with classification score above 0.5 rises from 0.35 to 0.82, and the share with both classification and IoU above 0.5 rises from 0.30 to 0.67.

RT-DETR: tunable decoder depth

Each decoder layer produces predictions, and the model is trained with auxiliary losses on all of them. At inference you can therefore stop early. The paper reports that using only the fifth layer of the six-layer R50 model costs 0.1 AP (53.1 to 53.0) and cuts latency from 9.3 ms to 8.8 ms, with no retraining.

Ultralytics documents the same knob for its RT-DETR-L implementation, measured with TensorRT 10.11 on a T4. The default (six decoder layers, 300 queries) is listed at 8.0 ms and 52.7 mAP. Four decoder layers gives 7.4 ms and 52.5 mAP. Dropping to 100 queries gives 7.4 ms and 51.7 mAP. This is a real latency dial that YOLO does not have, and it is free in engineering terms.

YOLO26: one-to-one head without a decoder

YOLO26 keeps a convolutional detector and changes how it is supervised and exported. According to the Ultralytics YOLO26 documentation, the model has two heads. The default one-to-many head outputs a tensor of shape (N, nc + 4, 8400) and needs NMS. The one-to-one head, selected with nms=False, outputs up to 300 detections per image in a tensor of shape (N, 300, 6). The documentation says the one-to-many head typically gives slightly higher accuracy.

The reason this works is training. A one-to-many head assigns several positive anchors to each ground-truth object, which gives richer gradients but produces duplicates. A one-to-one head assigns exactly one, which suppresses duplicates but gives sparse supervision. Training both together lets the shared backbone learn from the dense signal while the one-to-one head learns to emit a single answer. At export, you ship only the one-to-one path.

Ultralytics also lists three other changes: removal of Distribution Focal Loss (DFL) from box regression, a progressive loss that shifts emphasis toward the inference-time head, and small-target-aware label assignment (STAL). It adds a MuSGD optimizer, described as a hybrid of SGD and Muon.

A caveat about sources. The most detailed independent-looking analysis, an arXiv paper titled “YOLO26: An Analysis of NMS-Free End to End Framework for Real-Time Object Detection”, is explicitly a secondary analysis built on Ultralytics documentation. Its author states that some architectural interpretations are his own abstractions, and the paper contains internal inconsistencies (for example, different mAP values for the same models in two tables). I use it only for claims that match Ultralytics’ own docs.

Why DFL removal matters for the edge

DFL predicts each box edge as a discrete probability distribution and takes its expectation, which needs a softmax over bins. Softmax-heavy heads are awkward to quantize to INT8 and can map poorly to NPUs and DSPs. The Ultralytics blog also connects DFL removal to making one-to-one assignment workable. The argument that this reduces edge latency is plausible but, in the sources I found, asserted rather than measured on a stated NPU. Test it on your target.

What the Published Numbers Actually Show

Numbers are where comparison posts go wrong, so this section is deliberately conservative.

RT-DETR and RT-DETRv2 results

From the original paper, on COCO val2017 with TensorRT FP16 on a T4 at 640 by 640:

Model	AP	FPS (T4, FP16)	Params	GFLOPs
RT-DETR-R50	53.1	108	42M	136
RT-DETR-R101	54.3	74	76M	259
YOLOv8-L (paper’s comparison)	52.9	71	43M	165
YOLOv8-X (paper’s comparison)	53.9	50	68M	257

The authors state RT-DETR-R50 beats YOLOv8-L by 0.2 AP and runs 52.1 percent faster, and R101 beats YOLOv8-X by 0.4 AP and runs 48.0 percent faster. With Objects365 pre-training, R50 reaches 55.3 AP and R101 reaches 56.2. The YOLOv8 FPS figures include NMS, which is the paper’s methodological point.

RT-DETRv2 (arXiv 2407.17140) keeps the architecture and changes the decoder’s deformable attention plus training. Its bag-of-freebies are selective multi-scale sampling (different numbers of sampling points per scale), a discrete sampling operator that replaces grid_sample with rounded offsets for easier deployment, dynamic data augmentation that is switched off in the last two epochs, and scale-adaptive learning rates. Reported results versus RT-DETR:

Variant	Backbone	Params	FPS (bs=1)	AP
S	ResNet18	20M	217	47.9
M	ResNet34	31M	161	49.9
M*	ResNet50	36M	145	51.9
L	ResNet50	42M	108	53.4
X	ResNet101	76M	74	54.3

Gains over RT-DETR are largest for small models (+1.4 AP for S) and vanish at X. The discrete sampling operator, which removes the grid_sample deployment barrier, costs roughly 0.3 to 0.5 AP in the authors’ ablation. If your runtime lacks efficient grid_sample, that is a fair price; on TensorRT it usually is not needed.

YOLO26 results

From the Ultralytics YOLO26 documentation (COCO, 640 px, speeds with the NMS-free head):

Model	mAP 50-95	mAP (end-to-end)	T4 TensorRT10 (ms)	CPU ONNX (ms)	Params	FLOPs
YOLO26n	40.9	40.1	1.7	38.9	2.4M	5.5B
YOLO26s	48.6	47.8	2.5	87.2	9.5M	20.9B
YOLO26m	53.1	52.5	4.7	220.0	20.4M	68.4B
YOLO26l	55.0	54.4	6.2	286.2	24.8M	86.8B
YOLO26x	57.5	56.9	11.8	525.8	55.7M	194.4B

Two details deserve attention. The end-to-end mAP column is consistently 0.6 to 0.8 points lower than the standard column, which quantifies the accuracy cost of the one-to-one head on this model family. And Ultralytics reports up to 43 percent faster CPU ONNX inference for YOLO26n versus YOLO11n on an Intel Xeon at 2.00 GHz; this is a vendor claim for the nano model only.

Roboflow’s RF-DETR

Roboflow reports RF-DETR built on a DINOv2 backbone, with Nano to Large under Apache 2.0 and XL and 2XL under Roboflow’s own PML license. Its published figures include Nano at 48.4 mAP and 2.3 ms on a T4, and Large at 60.1 AP and 6.8 ms. Precision, batch size, and measurement method are not stated in the article I read, and the article comes from the vendor. I include RF-DETR because it matters to the landscape, but I would not rank it against the other numbers here without re-measuring.

What you cannot conclude from these tables

You cannot read a clean winner across the three tables. The protocols differ. The RT-DETR paper uses FPS from a T4 with TensorRT FP16. The Ultralytics RT-DETR page lists RT-DETR-L at 53.0 AP and 114 FPS and RT-DETR-X at 54.8 AP and 74 FPS without stating the runtime, and separately lists 8.0 ms and 52.7 mAP for RT-DETR-L under TensorRT 10.11. The YOLO26 table uses TensorRT 10. These are close in spirit but not guaranteed to share settings.

With that caveat, one comparison is fair enough to discuss: YOLO26l lists 55.0 mAP (54.4 end-to-end) at 6.2 ms on T4, while Ultralytics’ RT-DETR-L lists 52.7 mAP at 8.0 ms. On the vendor’s own numbers, the new YOLO is both faster and more accurate than the first-generation RT-DETR at similar size. That is a real result and it should dampen any assumption that transformers automatically win on accuracy at the same latency. RT-DETRv2 and newer DETRs narrow the gap, and measuring on your own data is the only way to settle it.

Deploying on TensorRT: A Walk-through for Edge GPUs

The architecture decides what is possible. The export path decides what ships. This section walks through a workflow that works for either family, with the places each one differs.

Figure 3: Export and validation workflow. The accuracy gate after the FP16 build decides whether INT8 is attempted, and attention layers are the first suspects when it fails.

Figure 3 shows the order of operations. Export to ONNX with a pinned opset, check the graph, build an FP16 engine, gate on accuracy, optionally calibrate INT8, then measure on the actual target device. Each step catches a different class of bug.

Step 1: export with the end-to-end contract

For YOLO26, the Ultralytics documentation lists TensorRT, ONNX, CoreML, LiteRT, and OpenVINO as export formats and states that exported speeds use the NMS-free head. For RT-DETR, the Ultralytics docs show an export(format="engine", device=0, quantize=16) call and note that decoder-depth and query-count settings carry into the export.

The sketch below is illustrative. Argument names reflect the Ultralytics documentation I fetched, but check your installed version, since export options change between releases.

from ultralytics import YOLO, RTDETR

# YOLO26: ship the one-to-one path. Weights name is illustrative.
yolo = YOLO("yolo26s.pt")
yolo.export(format="onnx", imgsz=640, opset=17, simplify=True)

# RT-DETR: optionally trade accuracy for latency BEFORE export.
rtdetr = RTDETR("rtdetr-l.pt")
head = rtdetr.model.model[-1]
head.decoder.eval_idx = 3     # use 4 decoder layers
head.num_queries = 300        # keep 300 unless you validated fewer
rtdetr.export(format="onnx", imgsz=640, opset=17)

The eval_idx and num_queries attributes come from the Ultralytics RT-DETR documentation. Per that page, max_det only limits returned detections, and pretrained checkpoints cap output at 300 because they use 300 queries.

Step 2: confirm the graph has no NMS

Open the ONNX file in Netron or run a short script to list operator types. You want to see the final output as a plain tensor: (1, 300, 6) for the YOLO26 one-to-one head, or boxes plus scores of 300 rows for RT-DETR. If you see a NonMaxSuppression operator, you exported the one-to-many head and your pipeline is NMS-based again.

import onnx
m = onnx.load("yolo26s.onnx")
ops = {n.op_type for n in m.graph.node}
print("NMS in graph:", "NonMaxSuppression" in ops)
print("outputs:", [(o.name, [d.dim_value for d in o.type.tensor_type.shape.dim])
                   for o in m.graph.output])

Step 3: build the engine and measure honestly

TensorRT’s trtexec builds and benchmarks an engine. The flags below are standard trtexec options; verify them against your TensorRT version.

trtexec --onnx=yolo26s.onnx --saveEngine=yolo26s_fp16.engine --fp16 \
        --warmUp=500 --iterations=1000 --useCudaGraph --noDataTransfers

Two measurement traps matter. --noDataTransfers excludes host-device copies, which flatters the number compared with a real pipeline that must upload each frame. And a warm, clocked-up GPU on a desk differs from a Jetson in a hot enclosure. On Jetson, lock clocks and power mode before measuring, and record the mode. Our TensorRT 11 vs 10 migration notes cover plugin and API drift between versions that can change results.

The fair way to report latency for an NMS-based model is end to end: preprocess, inference, NMS, and copy. For an NMS-free model the same end-to-end number is simpler because the tail is a constant-time read.

Step 4: build a latency histogram, not a mean

The strongest practical argument for NMS-free detection is tail latency, not mean latency. Mean frame time hides the crowded frame that misses its deadline. Measure the distribution on a dataset that includes your worst scenes.

import time, numpy as np

def bench(run_once, frames, warmup=50):
    for f in frames[:warmup]:
        run_once(f)
    lat = []
    for f in frames:
        t0 = time.perf_counter()
        run_once(f)           # includes pre/post-processing
        lat.append((time.perf_counter() - t0) * 1e3)
    a = np.array(lat)
    return {"mean": a.mean(), "p50": np.percentile(a, 50),
            "p99": np.percentile(a, 99), "max": a.max()}

For an NMS-based model, group frames by object count and plot p99 against it. If the curve slopes upward, NMS is contributing. For an NMS-free model the curve should be flat apart from preprocessing. I have not run this on the models discussed here; it is a method, not a result. Whether the slope matters depends on your scenes. In a warehouse camera with three objects it will not. In a crowd scene with hundreds it can.

Step 5: FP16 first, INT8 with suspicion

FP16 engines are usually safe. INT8 needs calibration data representative of deployment, and transformer attention is the typical casualty. Softmax and layer normalization are sensitive to quantization error. Common practice is mixed precision: quantize convolutional layers and keep attention and the decoder in FP16. Our INT4 vs INT8 vs FP8 edge NPU quantization post goes through the trade-offs.

This is also where the DFL argument gets tested. YOLO26’s removal of softmax-based box regression is claimed to ease quantization. I did not find a published INT8 accuracy comparison between YOLO26 and RT-DETR. Treat the claim as a hypothesis, build both engines, and compare INT8 accuracy drop on your validation set. Our FPGA vs GPU vs NPU edge inference comparison shows why NPU operator support, not FLOPs, is often the binding constraint.

Step 6: validate parity between PyTorch and the engine

Run a few hundred images through the PyTorch model and the TensorRT engine and compare detections, not just mAP. Check that box counts match within tolerance and score distributions have not shifted. For NMS-free models, watch for duplicate rate (two high-score boxes on one object) and miss rate, because those are the signature failures of imperfect one-to-one training.

Deeper Analysis: Where Each Design Wins

Mechanism tells you where to expect differences. Measurement confirms them. Here are the places the two families should diverge.

Crowded and overlapping scenes

NMS removes boxes with high IoU against a higher-scoring box. When two real objects overlap heavily, NMS can delete a true positive. Soft-NMS and tuned thresholds mitigate this but add parameters. NMS-free models avoid the hard rule, so they have a structural advantage when objects touch: stacked boxes, people in queues, packed produce.

The advantage is real only if the model learned to separate the objects. A one-to-one head trained on sparse data can fail the other way and emit duplicates. The RT-DETR paper’s results on COCO support the general claim, but COCO is not your factory. Test on your densest realistic frames.

Small objects and high-resolution input

DETR variants historically struggled with small objects because decoder queries attend to coarse features. RT-DETR’s multi-scale encoder and deformable attention decoder address this, but resolution still matters. YOLO26’s STAL targets small objects by lowering the assignment IoU threshold for them (a paper analysis cites an example drop from 0.5 to 0.10, again a secondary source). If your targets are a few pixels wide, such as defects on a 4K line, both families need tiling or higher input resolution, and the cost grows quickly with input size.

Domain shift and few-shot fine-tuning

Roboflow argues that DINOv2-based RF-DETR adapts quickly to new domains, citing one study in which it plateaued within ten epochs on single-class tasks, and notes that results vary by dataset. I treat this as promising but unproven in general. Pretrained backbone quality tends to dominate small-dataset fine-tuning, so the backbone comparison in our vision transformer vs ConvNeXt vs Swin vs YOLO backbones post is relevant here.

Latency tuning knobs

RT-DETR gives you decoder depth and query count at inference time. YOLO gives you model size (n, s, m, l, x) and input resolution. These are different shapes of control. A decoder-depth change preserves the same weights and nearly the same accuracy, so you can tune on deployed hardware without a new training run. A size change requires choosing a different checkpoint and, usually, re-validating.

The documented RT-DETR-L numbers show the shape of the trade: dropping from six to four decoder layers saved 0.6 ms (8.0 to 7.4 ms) for 0.2 mAP on a T4. Cutting queries to 100 saved the same 0.6 ms but cost 1.0 mAP. Depth is the better knob to turn first.

Hardware fit

Transformer decoders run well on GPUs and on accelerators with mature attention support. CNN-heavy models fit NPUs and CPUs better. This is why Ultralytics emphasizes CPU and edge speed for YOLO26 (the 43 percent CPU figure) and why the RT-DETR docs emphasize CUDA with TensorRT. If your target is a Hailo, an Ethos NPU, or a microcontroller-class device, check operator support for attention before you commit to a DETR. Our Arm Ethos-U85 TinyML architecture piece shows how narrow NPU operator sets can be.

License and total cost

Licensing is part of the architecture decision. Per the Ultralytics docs, YOLO26 is AGPL-3.0 or an Enterprise license. Roboflow lists RF-DETR Nano through Large as Apache 2.0, and XL and 2XL under its PML license. The Ultralytics RT-DETR page does not state a license for the RT-DETR models themselves, so check the repository you actually pull weights from; RT-DETR’s original release comes from Baidu. If you ship a closed-source device, AGPL has consequences that your counsel should evaluate. I am not giving legal advice, only flagging that this belongs in the matrix.

Trade-offs, Gotchas, and What Goes Wrong

NMS-free is not a free lunch. These are the failure modes I would test for before approving either model.

The end-to-end accuracy tax

YOLO26’s own table shows the end-to-end mAP 0.6 to 0.8 points below the one-to-many mAP across sizes. You are trading a sliver of accuracy for simpler deployment. For most applications that is a good trade, but if you are chasing the last half point in a benchmark-driven contract, know what you give up. The documentation notes you can still run the one-to-many head with NMS.

Fixed query count caps detections

RT-DETR emits 300 predictions; YOLO26’s one-to-one head emits up to 300 detections. Dense scenes with more than 300 objects, such as cell counting or aerial crowd imagery, hit a hard ceiling. Retraining with a larger query budget is possible but it is a modelling change, not a flag.

Duplicates and phantom boxes

An imperfectly trained one-to-one model can produce two moderately confident boxes on one object. With NMS, that would be cleaned up. Without it, you get double counts. Monitor the duplicate rate in production and consider a cheap IoU-based dedupe as a safety net. That reintroduces a post-processing step, but a trivial one on at most 300 boxes.

Calibration and score thresholds

DETR-style scores are trained under a matching loss and are not calibrated probabilities. A threshold of 0.5 means different things across models. Choose operating points from a precision-recall curve on your own data. Do not carry thresholds from a YOLO deployment to an RT-DETR deployment.

Benchmark protocol mismatch

The most common error in model selection is comparing numbers from different tools. The T4 TensorRT FP16 figure in a paper, an Ultralytics-measured TensorRT 10 figure, and a vendor blog figure with unstated precision are not interchangeable. Always rebuild both candidates with your toolchain on your device before deciding.

Training cost and convergence

DETR models have historically needed long schedules. The RT-DETR paper uses a 6x schedule (72 epochs in its main tables) for final results, with Objects365 pre-training adding further epochs. Fine-tuning from pretrained weights is much cheaper, but if you train from scratch on a niche dataset, budget more GPU time than for a YOLO.

Tooling drift

TensorRT versions, ONNX opsets, and Ultralytics releases move quickly. An engine is tied to a TensorRT version and GPU architecture, so an engine built on a workstation does not run on a Jetson. Pin versions in CI, build engines on the target or an identical image, and keep the ONNX as the portable artifact.

Choosing Between Them: A Decision Framework

Figure 4: Decision tree for rt-detr vs yolo selection. Hardware class comes first, then scene difficulty, then license.

Figure 4 encodes the order in which constraints should bite. Hardware class is the hardest constraint, scene difficulty is the main accuracy driver, and licensing is the tiebreaker that sometimes overrides everything.

Situation	Lean toward	Reason	Verify by
CPU-only gateway or NPU with limited operator set	YOLO26 n or s	Convolutional graph, vendor-reported CPU gains	Measure CPU ONNX latency on the real SoC
NVIDIA Jetson or discrete GPU, balanced scenes	YOLO26 m or l, or RT-DETRv2 L	Both are competitive on vendor-reported T4 numbers	Rebuild both with trtexec, compare p99
Crowded, overlapping objects	RT-DETR family or RF-DETR	No hard suppression rule	Test duplicate and miss rate on dense frames
Few labeled images, domain shift	RF-DETR or RT-DETR with strong pretraining	Pretrained backbone dominates	Fine-tune both on 10 percent of data
Closed-source commercial device	RT-DETRv2 or RF-DETR small	Apache-licensed options exist	Confirm license of the exact weights
Need runtime latency tuning without retrain	RT-DETR	Decoder depth and query count	Sweep eval_idx on target
More than 300 objects per frame	Neither as shipped	Fixed query budget	Retrain with larger budget or tile

Read the table as a prior, not a verdict. The “verify by” column is the real instruction.

Practical Recommendations

Start by deciding the hardware and license envelope, because those eliminate options for free. Then pick one candidate from each family at roughly the same size, fine-tune both on a representative slice of your data, and compare them under identical conditions.

Measure latency as a distribution on your device, including preprocessing and any remaining post-processing. Report p50 and p99, not just the mean. Compare accuracy at the operating point you will actually deploy, not at the benchmark’s low confidence threshold.

Prefer FP16 first. Attempt INT8 only after FP16 is validated, and keep attention layers in higher precision if accuracy slips. Keep the ONNX file as the source of truth and rebuild engines per target.

For a production rollout, add a cheap dedupe and a detection-count alarm, and log the score distribution so you notice drift. Plan to re-run the comparison when a new TensorRT, JetPack, or model release lands; the numbers in this post are a snapshot of October 2026 sources.

A short checklist:

Confirm the weights’ license for your distribution model.
Export the end-to-end head and verify the graph contains no NMS operator.
Build FP16 engines on the target and record clocks and power mode.
Benchmark p50 and p99 on your hardest scenes, end to end.
Compare duplicate rate and miss rate, not only mAP.
Gate INT8 on a validation set; keep attention in FP16 if needed.
Pin TensorRT, CUDA, and exporter versions in CI.
Revisit after each major runtime release.

Frequently Asked Questions

Is RT-DETR better than YOLO?

Neither wins universally. The original RT-DETR paper reports RT-DETR-R50 at 53.1 AP and 108 FPS on a T4 with TensorRT FP16, ahead of YOLOv8-L at 52.9 AP and 71 FPS including NMS. But Ultralytics’ newer YOLO26l lists 55.0 mAP at 6.2 ms on a T4, which beats its RT-DETR-L figures. Protocols differ, so re-measure on your data and hardware.

What does NMS-free detection mean?

It means the model is trained so each object yields one confident prediction, so no non-maximum suppression step is needed at inference. RT-DETR does this with a transformer decoder and bipartite matching. YOLO26 does it with a one-to-one head, exposed with nms=False, which outputs up to 300 detections as a fixed-shape tensor. The result is simpler export and more predictable post-processing time.

Does YOLO26 really run without NMS?

Yes, in its end-to-end mode. Per Ultralytics, the one-to-one head outputs a (N, 300, 6) tensor and needs no NMS, while the default one-to-many head outputs (N, nc + 4, 8400) and does. The documentation lists end-to-end mAP slightly below the standard figure, for example 54.4 versus 55.0 for YOLO26l, so you accept a small accuracy cost for simpler deployment.

Can RT-DETR run on a Jetson with TensorRT?

TensorRT supports the operators RT-DETR needs, and Ultralytics documents TensorRT export for it and says the model performs well on CUDA with TensorRT. I did not find a vendor-published Jetson latency table for RT-DETR in the sources I read, so build the engine on your Jetson and measure. Lock clocks and power mode first, and expect numbers different from a T4.

Is transformer-based detection too heavy for edge devices?

Not necessarily. RT-DETR confines attention to the smallest feature map and keeps fusion convolutional, which is why it is fast on GPUs. RT-DETRv2-S has 20M parameters and a reported 217 FPS on a T4 at batch size 1. The real risk is NPUs and microcontrollers with limited attention operator support, where a CNN-based YOLO is safer.

Which is better for commercial products, YOLO26 or RT-DETR?

It depends on licensing. Ultralytics lists YOLO26 under AGPL-3.0 or an Enterprise license. Roboflow lists RF-DETR Nano through Large as Apache 2.0, and the original RT-DETR comes from Baidu, so confirm the license of the specific weights and code you use. Have counsel review distribution obligations for closed-source devices; this post is not legal advice.