Linux 7.1 PREEMPT_RT on ARM: Build, Measure, Compare

For years, anyone who wanted a real-time Linux kernel on a 32-bit ARM board had to carry patches. You downloaded a kernel, found the matching patch-x.y.z-rtN.patch.xz, applied it, fixed the rejects, and hoped the vendor BSP did not collide with it. That ritual is over for a large class of boards. Linux 7.1 PREEMPT_RT on ARM works from a vanilla kernel tree: the last ARM-specific real-time patches were merged before 7.1-rc1, and the 7.1 release landed on 14 June 2026.

This matters now because the installed base of industrial gateways is overwhelmingly 32-bit ARM: i.MX6, i.MX7, AM335x, Cortex-A7 and Cortex-A9 parts that will run for another decade. Mainline support means security updates, long-term-support trees and distribution kernels can all carry real-time behaviour without a private fork.

In this tutorial you will build the kernel, configure it for deterministic behaviour, measure latency with cyclictest and rt-tests, and then decide, with a decision framework rather than a slogan, whether Linux RT, Zephyr or a dual-kernel approach fits your gateway.

What this covers: what changed in 7.1, how PREEMPT_RT works inside the kernel, a cross-compile and boot walk-through, runnable measurement commands, tuning, failure modes, and a Linux RT vs Zephyr vs Xenomai decision guide.

Context and Background

PREEMPT_RT began life as Ingo Molnar’s real-time preemption patch set in 2004 and was maintained out of tree for two decades, largely by Thomas Gleixner, Sebastian Andrzej Siewior, Steven Rostedt and a rotating cast of contributors. Its goal never changed: make the kernel preemptible almost everywhere, so that the worst-case time between an event and the highest-priority task reacting to it is bounded and small. The bulk of the work merged gradually (threaded interrupts, high-resolution timers, priority-inheriting mutexes, the printk rework), and in Linux 6.12 the final switch landed so that CONFIG_PREEMPT_RT became a selectable option on architectures that declared support.

Architecture support then spread one port at a time. According to Phoronix’s reporting on the 7.1 merge, mainline already supported real-time kernels on x86_64, ARM64 and RISC-V from 6.12, and on LoongArch from 6.13. The remaining gap was 32-bit ARM. A cleanup of ARM fault handling merged shortly before 7.1-rc1 removed the blocker, and the v7.1-rc1-rt1 patch release dropped all ARM patches, noting that everything required had reached mainline. Three items remain out of tree as of that reporting: fixes for the Intel i915 DRM driver, a patch adding a /sys/kernel/realtime entry that tells userspace it is on an RT kernel, and pending PowerPC enablement. None of these affects a typical ARM gateway, though the missing sysfs flag means your scripts should check uname -v instead.

Why does 32-bit ARM still matter in 2026? Because replacement cycles in industrial equipment are measured in decades. A protocol gateway sitting between a PLC cell and a cloud broker is rarely replaced because a better SoC exists; it is replaced when it fails or when it can no longer be patched. A mainline real-time option lowers the cost of keeping those boards patchable. The determinism requirements, however, are not new. The same jitter concerns that shape PLC scan cycles under IEC 61131-3 apply when you move a control-adjacent workload (a Modbus poller, an OPC UA server, a protocol translator) onto a Linux gateway.

The competing options each occupy a different point on the determinism-versus-capability curve. Microcontroller RTOSes such as Zephyr, FreeRTOS and NuttX give you microsecond-class interrupt latency on small cores but little userland. We compared the first three in our Zephyr vs FreeRTOS vs NuttX industrial RTOS analysis. Dual-kernel systems such as Xenomai put a small real-time core beside Linux. Mainline PREEMPT_RT sits in the middle: Linux’s full ecosystem with bounded, but not hard-guaranteed, latency. For the authoritative description of what the patch set does and how to configure it, see the Linux kernel’s real-time documentation and the Linux Foundation real-time wiki.

A note on rigor. This article states the release facts above from published reporting and kernel.org-linked sources. Every latency figure that appears below is illustrative: it shows the shape of results you should expect, not a benchmark I ran. Real numbers depend on the SoC, clock, cache and memory configuration, drivers, and load. You must measure on your own hardware, and the method in this post is how.

What Linux 7.1 Changes for 32-bit ARM Real-Time

In one paragraph: Linux 7.1 lets you select CONFIG_PREEMPT_RT for 32-bit ARM from a vanilla tree. With it, spinlocks become sleeping, priority-inheriting locks, interrupt handlers run as schedulable threads, and the scheduler can preempt almost any kernel code path, bounding the delay before a SCHED_FIFO task runs.

Figure 1: Mainline PREEMPT_RT architecture support timeline, ending with 32-bit ARM in Linux 7.1 and the three items still out of tree.

Figure 1 shows how architectures joined mainline RT over four releases. The path matters for risk assessment: x86_64 and ARM64 have had years of production exposure, while 32-bit ARM mainline RT is new as of 7.1. Treat the first kernels as a platform you validate, not one you assume.

What actually blocked ARM

The reporting describes the enabling change as a cleanup of ARM fault handling. The reason that is plausible as a blocker is mechanical: on an RT kernel, page-fault and exception paths may take sleeping locks, so they must run with interrupts and preemption in a state that permits sleeping. Code that disabled preemption or interrupts around fault handling would trigger “sleeping function called from invalid context” errors and, worse, latency spikes. I have not read the individual commits for this post, so treat the exact mechanism as an informed description rather than a citation; the reported fact is only that a fault-handling cleanup was the enabling change.

What you gain by being mainline

There are four concrete gains. First, you can use git and a normal stable-kernel workflow: pull a 7.1.y stable release and rebuild, with no patch rebase. Second, distribution and vendor kernels can enable real-time with a config flag instead of a fork. Third, bugs are reported against one tree, so the community that already debugs RT on x86 and ARM64 also looks at ARM32. Fourth, security fixes arrive at the same cadence as everything else.

What you do not gain is a guarantee. Mainline RT bounds latency in the engineering sense; it does not prove a worst case. Drivers, firmware, SMIs on x86, cache and TLB behaviour, bus contention and DVFS all participate in the tail. On a 32-bit ARM SoC the same caveat applies: your vendor’s GPU, Ethernet, USB and flash drivers are where the surprises live.

How PREEMPT_RT achieves bounded latency

Four mechanisms do most of the work, and understanding them tells you what to tune.

Sleeping spinlocks. In a non-RT kernel, spinlock_t disables preemption while held. On PREEMPT_RT it is implemented on top of rt_mutex, so a task holding it can be preempted and a contending task sleeps instead of spinning. Truly atomic sections use raw_spinlock_t, which keeps the old behaviour and must be short. The result is that critical sections that used to be non-preemptible are now preemptible, shrinking the longest window in which a high-priority wakeup can be delayed.

Threaded interrupt handlers. With RT enabled, most hard-IRQ handlers are forced into kernel threads (irq/NN-name). The hardirq part only acknowledges and wakes the thread. Because those threads have scheduler priorities (default SCHED_FIFO 50), you can place your control task above or below a given device’s IRQ thread, which is a powerful lever and a trap if misused.

Priority inheritance. When a high-priority task blocks on an rt_mutex held by a lower-priority task, the holder is temporarily boosted to the waiter’s priority so a medium-priority task cannot starve it. This avoids unbounded priority inversion, the failure that famously affected a Mars lander in 1997. Userspace gets the same via PTHREAD_PRIO_INHERIT on POSIX mutexes, which is non-optional for real-time code.

Preemptible RCU, softirq handling and high-resolution timers. Read-side RCU sections are preemptible, softirqs run in the context of the thread that raised them (or ksoftirqd), and hrtimers give you timer events at hardware resolution rather than tick granularity.

Figure 2: Threaded interrupt wakeup and priority inheritance on an rt_mutex, showing why a medium-priority task cannot starve the lock holder.

The diagram above traces one wakeup. The timer interrupt only wakes a threaded handler; the handler wakes your RT task; your task blocks on a mutex held by a low-priority task; priority inheritance boosts the holder past a medium-priority task; and when the lock is released the RT task runs. Every step in that chain is a place where latency can accumulate, and each corresponds to something you can measure.

Building a Mainline PREEMPT_RT Kernel for 32-bit ARM

The steps below use a generic multi-platform ARMv7 configuration so they apply to boards such as i.MX6, AM335x and Allwinner H3 class devices. Where your SoC needs a vendor defconfig or device tree, substitute it. The commands are standard kernel build practice; the only 7.1-specific part is that PREEMPT_RT appears in the menu without patching.

Figure 3: Build, boot, measure and tune loop for a PREEMPT_RT 32-bit ARM kernel, ending in a latency budget check.

Figure 3 is the loop you will repeat. The build is the cheap part; the discipline is in the measure-and-tune cycle and in refusing to ship until the worst-case figure under load is inside your budget.

Step 1: toolchain and source

On a Debian or Ubuntu build host, install the hard-float cross compiler and kernel build dependencies, then fetch the 7.1 source. Use the stable tree so you receive fixes.

sudo apt install gcc-arm-linux-gnueabihf make bc bison flex libssl-dev \
     libncurses-dev u-boot-tools device-tree-compiler git

git clone --depth 1 --branch v7.1 \
  https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git linux-7.1
cd linux-7.1
export ARCH=arm
export CROSS_COMPILE=arm-linux-gnueabihf-

If a v7.1.y stable tag exists when you read this, prefer the newest one. Check kernel.org for the current stable release; I confirmed the 7.1 release date (14 June 2026) from Phoronix’s release article but did not enumerate stable point releases.

Step 2: base configuration

Start from the multi-platform ARMv7 defconfig, or from your vendor defconfig if the SoC is not covered. Then enable the RT preemption model.

make multi_v7_defconfig        # or imx_v6_v7_defconfig, omap2plus_defconfig ...
make menuconfig

In menuconfig, go to General setup, Preemption Model and choose Fully Preemptible Kernel (Real-Time). On some trees the option is hidden until CONFIG_EXPERT is enabled; if you do not see it, enable Expert under General setup first. In the saved .config you want:

CONFIG_EXPERT=y
CONFIG_PREEMPT_RT=y
CONFIG_PREEMPT_COUNT=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_NO_HZ_IDLE=y
CONFIG_HZ_250=y

Then turn things off. Debug options that add instrumentation are the most common cause of “the RT kernel is slow” reports:

./scripts/config --disable DEBUG_PREEMPT
./scripts/config --disable LOCKDEP
./scripts/config --disable PROVE_LOCKING
./scripts/config --disable DEBUG_KMEMLEAK
./scripts/config --disable KASAN
./scripts/config --disable FUNCTION_TRACER   # re-enable only for debugging runs
./scripts/config --disable CPU_FREQ_DEFAULT_GOV_ONDEMAND
./scripts/config --enable  CPU_FREQ_DEFAULT_GOV_PERFORMANCE
make olddefconfig

Two choices deserve explanation. Tracing and lock-debugging options inflate worst-case latency by large factors, so keep two configs: a lean production config and a diagnostic config with FTRACE, IRQSOFF_TRACER and PREEMPT_TRACER, used only to hunt a spike. And the CPU frequency governor matters: a governor that changes frequency under load creates latency jitter and shifts timing between runs. Fix the frequency (the performance governor, or a pinned frequency) for the measurement runs, then decide whether the product can tolerate scaling.

Verify the result before building:

grep -E 'CONFIG_PREEMPT_RT|CONFIG_PREEMPT=|CONFIG_HZ=' .config

If CONFIG_PREEMPT_RT=y is missing after olddefconfig, something in your config depends on an option RT does not support, or the architecture symbol is not selected. Read the dependency with / in menuconfig instead of editing .config by hand.

Step 3: build and install

make -j"$(nproc)" zImage dtbs modules
make INSTALL_MOD_PATH=$PWD/out modules_install
cp arch/arm/boot/zImage out/
cp arch/arm/boot/dts/<vendor>/<board>.dtb out/

Copy zImage and the .dtb to your boot partition, and copy out/lib/modules/ to the root filesystem. Boot, then confirm the kernel is the one you built:

uname -a
# Linux gw01 7.1.0 #1 SMP PREEMPT_RT ... armv7l GNU/Linux
cat /proc/version
zcat /proc/config.gz | grep PREEMPT_RT   # needs CONFIG_IKCONFIG_PROC

The PREEMPT_RT token in uname -v is the reliable runtime check, since the /sys/kernel/realtime flag was reported as still out of tree. Write your provisioning scripts to parse uname -v.

Step 4: install the measurement tools

sudo apt install rt-tests stress-ng
# or build from source
git clone https://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git
make -C rt-tests

rt-tests ships cyclictest, hackbench, signaltest, ptsematest and others. The rtla tool in the kernel tree (tools/tracing/rtla) offers timerlat, which separates timer-IRQ latency from thread latency and is the better tool for root-causing a spike once cyclictest has told you one exists.

Measuring Latency with cyclictest

Cyclictest measures the difference between when a timer should have woken a thread and when the thread actually ran. It reports minimum, average and maximum, plus an optional histogram. The maximum under heavy load is the number that matters, not the average.

A repeatable measurement recipe

The following command pins one thread per CPU, locks memory, uses SCHED_FIFO priority 95, wakes every 1000 microseconds, and runs for ten minutes:

sudo cyclictest --mlockall --smp --priority=95 --interval=1000 \
     --distance=0 --duration=10m --histogram=400 --histfile=hist.txt -q

For a single core you want to protect:

sudo cyclictest -m -a 1 -t 1 -p 95 -i 1000 -D 10m -q

Flags worth knowing: -m locks all memory so page faults cannot pollute the result; -a sets CPU affinity; -t sets the thread count; -p is the real-time priority; -i is the wakeup interval in microseconds; -D the duration; -h or --histogram records a histogram up to the given bucket in microseconds. Use a shorter interval such as 200 microseconds to stress the timer path more often, at the cost of more CPU overhead.

Generate load while you measure

An idle system tells you nothing. Run load in a second terminal for the whole measurement:

# CPU, memory, and scheduler pressure
stress-ng --cpu 4 --vm 2 --vm-bytes 128M --io 2 --timeout 600s &
hackbench -l 100000 &
# I/O and network pressure matching your real workload
dd if=/dev/zero of=/tmp/big bs=1M count=2000 oflag=direct &
iperf3 -c <peer> -t 600 &

The load must resemble what the gateway will actually do. A Modbus poller, an MQTT publisher and a TLS handshake storm stress different kernel paths (network stack, crypto, flash I/O), and the worst-case latency usually comes from the path you forgot to load. Run for hours, not minutes, before drawing conclusions about the tail; the distribution of rare events is exactly what a ten-minute run misses.

Reading the output

A typical line looks like:

T: 0 ( 612) P:95 I:1000 C: 600000 Min:      9 Act:   14 Avg:   17 Max:      88

Min, Avg and Max are in microseconds. The numbers above are illustrative of the format and of a plausible order of magnitude, not a measured result. Published community results for tuned PREEMPT_RT kernels on embedded ARM cores generally land in tens of microseconds typical and low hundreds of microseconds worst case under stress, while an untuned non-RT kernel can show multi-millisecond maxima; but you should never quote such a range for your product without your own histogram.

To interpret a histogram, plot the counts per bucket on a log scale. A clean RT system shows a tight peak near the minimum and a thin tail. A long, flat tail or a sudden isolated spike points to a specific cause: a driver holding interrupts disabled, a cache or memory-bandwidth contention event, an SD-card write stall, or thermal or frequency changes. Use rtla timerlat top or the irqsoff and preemptoff tracers on the diagnostic kernel to catch the culprit with a stack trace.

sudo rtla timerlat top -a 100 -c 1   # -a: auto-analysis threshold in us on CPU 1

Defining a pass criterion

Before you test, write down a budget. Example: the control loop runs every 1 ms and must complete within 800 microseconds; the budget for kernel scheduling latency is therefore less than 100 microseconds at the 99.999th percentile, with a hard ceiling for anything beyond. Tests then have an unambiguous pass or fail. Without a budget, “low latency” means whatever you observed.

Tuning the Gateway: Isolation, IRQ Threads, and Application Discipline

A mainline RT kernel gives you the mechanism. Determinism comes from how you configure the system around it. There are four levers: where interrupts run, which CPUs the scheduler uses, how your application behaves, and what else shares the machine.

CPU isolation and IRQ affinity

On a multi-core SoC such as a dual-core i.MX6 or a quad-core Cortex-A7, reserve a core for the real-time workload. On the kernel command line:

isolcpus=domain,managed_irq,1 nohz_full=1 rcu_nocbs=1 irqaffinity=0

Here isolcpus removes CPU 1 from the general scheduler domains, nohz_full stops the periodic tick there when a single task is running, rcu_nocbs moves RCU callback processing off it, and irqaffinity=0 steers default interrupts to CPU 0. Note that isolcpus flags and nohz_full have specific semantic requirements; read the kernel-parameters document for your 7.1 tree, and confirm that nohz_full is supported on your ARM configuration (it depends on CONFIG_NO_HZ_FULL and context tracking support, which I have not verified for every 32-bit ARM platform). On a single-core part you cannot isolate, so priority design carries the whole load.

Then bind device interrupts deliberately:

cat /proc/interrupts
echo 1 | sudo tee /proc/irq/<N>/smp_affinity      # mask 1 = CPU 0
ps -eLo pid,cls,rtprio,comm | grep irq/
sudo chrt -f -p 70 $(pgrep -f 'irq/[0-9]+-eth0')   # raise the NIC IRQ thread

Priority layout

Because interrupt handlers are threads, you decide who preempts whom. A defensible layout for a gateway is:

Priority (SCHED_FIFO)	Thread	Rationale
90	Fieldbus or timer IRQ thread for the control path	Must run before anything in userspace that consumes it
80	Your control or acquisition task	The reason the device exists
60	NIC IRQ thread	Needed, but must not starve the control task
50	Default IRQ threads	Kernel default
40	Network stack housekeeping, ksoftirqd	Below hardware and control
SCHED_OTHER	MQTT client, OPC UA server, web UI, logging	Best-effort work

The principle: never place an unbounded or CPU-hungry task above the IRQ threads that the system needs to make progress. A SCHED_FIFO task that spins at priority 99 can starve the kernel’s own threads, including the ones that feed the watchdog, and lock up the board. The kernel throttles real-time tasks through sched_rt_runtime_us (default 950000 out of a 1000000 microsecond period); keep that protection on in production unless you have a hardware watchdog and have tested it.

Application discipline

The application must cooperate or the kernel’s work is wasted. A minimal real-time loop in C looks like this:

#define _GNU_SOURCE
#include <pthread.h>
#include <sched.h>
#include <sys/mman.h>
#include <time.h>
#include <stdint.h>
#include <stdio.h>

#define PERIOD_NS 1000000L   /* 1 ms */

static void add_ns(struct timespec *t, long ns) {
    t->tv_nsec += ns;
    while (t->tv_nsec >= 1000000000L) { t->tv_nsec -= 1000000000L; t->tv_sec++; }
}

int main(void) {
    struct sched_param sp = { .sched_priority = 80 };
    if (sched_setscheduler(0, SCHED_FIFO, &sp)) { perror("sched"); return 1; }
    if (mlockall(MCL_CURRENT | MCL_FUTURE))      { perror("mlock"); return 1; }

    struct timespec next;
    clock_gettime(CLOCK_MONOTONIC, &next);
    for (;;) {
        add_ns(&next, PERIOD_NS);
        clock_nanosleep(CLOCK_MONOTONIC, TIMER_ABSTIME, &next, NULL);
        /* control work: bounded, no malloc, no blocking I/O, no printf */
    }
}

Four rules are embedded in those few lines. Use CLOCK_MONOTONIC with absolute-time sleeps so jitter does not accumulate. Lock memory with mlockall so a page fault never stalls the loop. Pre-allocate everything before the loop; malloc can enter the kernel and block. And keep the loop body bounded; log through a lock-free ring buffer to a lower-priority thread rather than calling printf. If the loop shares data with lower-priority threads, protect it with a pthread_mutex_t initialised with PTHREAD_PRIO_INHERIT, otherwise priority inheritance in the kernel will not help you in userspace.

pthread_mutexattr_t a;
pthread_mutexattr_init(&a);
pthread_mutexattr_setprotocol(&a, PTHREAD_PRIO_INHERIT);
pthread_mutex_init(&m, &a);

Taming drivers and shared hardware

Most real-time failures on ARM boards are driver and hardware problems, not scheduler problems. Check for these in your validation plan: SD or eMMC write stalls from flash garbage collection (put logs on tmpfs or a separate device); GPU or display drivers that disable interrupts for long periods (disable the display stack on headless gateways); USB host controllers and Wi-Fi drivers with long interrupt-off sections; DMA bursts that saturate the memory controller; and cache-coherency effects on multi-core parts, where a noisy neighbour core evicts your working set. The i915 note in the 7.1 coverage is a reminder that out-of-tree fixes can still be needed for a specific driver; apply the same suspicion to your SoC vendor’s drivers, and test them under the RT config, not just the default one.

Linux RT vs Zephyr vs Xenomai: A Decision Framework

The question “which is real-time?” is the wrong one. All three can be, with different guarantees. The right questions are: what is the deadline, what is the consequence of missing it, what else must the device do, and who maintains the software for ten years.

Figure 4: Decision tree for choosing Linux PREEMPT_RT, Zephyr on an MCU, or a dual-kernel Xenomai design based on deadline and platform needs.

Figure 4 encodes the reasoning. The branch thresholds (50 microseconds, 100 microseconds to 1 millisecond) are rules of thumb that I use as starting points, not standardised limits. Calibrate them with your own measurements.

Linux PREEMPT_RT

Linux RT is the right choice when the deadline is in the hundreds of microseconds to milliseconds range and the device also needs a real operating system: a TCP/IP stack with TLS, a container runtime, package management, SSH, over-the-air updates, filesystems, and a driver for almost every peripheral. It is a soft-to-firm real-time system. Its latency is bounded by engineering and testing, not by proof, and certification for safety functions is a separate and difficult conversation. With 7.1, the ARM32 build now sits in the mainline workflow, which is its main advantage for an installed fleet.

Zephyr

Zephyr is a small, scalable RTOS under the Linux Foundation, with a priority-based preemptive scheduler, a tickless kernel option, and a large set of board ports. On a Cortex-M class part, interrupt latency and context-switch times are typically in the low microseconds, and the kernel is small enough to analyse. The cost is capability: networking exists but is far thinner than Linux, there is no general-purpose filesystem ecosystem comparable to Linux, and you write more yourself. Zephyr is also available on some application-class cores, but the typical industrial pattern is different: put Zephyr on a microcontroller to own the hard loop, and let Linux handle everything else. Compare Zephyr with FreeRTOS and NuttX in our industrial RTOS comparison.

Many modern SoCs, including heterogeneous parts that combine Cortex-A and Cortex-M cores, make this a single-chip decision. Linux runs on the A cores; Zephyr runs on the M core; an inter-core mailbox such as RPMsg over OpenAMP links them. That architecture gives you microsecond-class determinism for the loop and a rich Linux gateway, with a narrow, auditable interface between them.

Xenomai and dual-kernel designs

Xenomai takes a different approach: a separate real-time core handles real-time tasks and interrupts at higher priority than the Linux kernel, so Linux’s own latency does not affect them. Xenomai 3 pioneered the “Cobalt” core on the older I-pipe patch. The current generation, Xenomai 4, is built around the EVL core, which uses the Dovetail interface and an interrupt pipeline. According to the EVL project’s published port status, which I checked for this article, EVL covers ARM64, 32-bit ARM and x86_64; the ten 32-bit ARM SoCs listed (including TI AM335x, Raspberry Pi chips, Altera Cyclone V, Exynos 5422, Allwinner H3 and NXP i.MX6 and i.MX7 parts) pass its test suite, though on test kernels ranging from v5.2-rc7 to v6.16, and the page states the current target is Linux v7.2 and cautions that 32-bit ARM may need more work on unlisted SoCs.

The takeaway is that the dual-kernel path remains alive and ports 32-bit ARM, but it requires an out-of-tree patch tied to specific kernel versions and a separate API for real-time tasks. That is exactly the maintenance cost mainline PREEMPT_RT removes. Choose a dual-kernel approach only when measured PREEMPT_RT latency on your hardware does not meet the budget and you cannot add a microcontroller.

Comparison matrix

Criterion	Linux PREEMPT_RT (mainline)	Zephyr on MCU	Xenomai 4 / EVL
Typical deadline class	Hundreds of us to ms	Single-digit us to ms	Tens of us to ms
Determinism basis	Bounded by tuning and test	Small kernel, analysable	Separate RT core above Linux
Userland and networking	Full Linux	Minimal, library-based	Full Linux for non-RT side
Kernel maintenance	Mainline, stable trees	Zephyr LTS	Out-of-tree patch set
32-bit ARM status	Mainline from 7.1	Many Cortex-M boards	Ports listed, older test kernels
Best fit	Protocol gateways, soft control	Motor, sensor, safety-adjacent loops	Linux apps needing tighter jitter

Rows describing “typical deadline class” are my engineering judgement, not published guarantees. Verify them on your hardware.

An Industrial Gateway Use Case

Consider a protocol gateway: a dual-core Cortex-A9 board reads 200 Modbus RTU registers from drives every 10 ms, runs a local rules engine, publishes aggregated data over MQTT with TLS, and exposes an OPC UA endpoint. Cloud connectivity, container updates and diagnostics are non-negotiable, which rules out a bare-metal approach.

On stock Linux the system works until a burst of TLS handshakes, a flash write or a USB event delays the Modbus poller by tens of milliseconds, causing a stale read and, in the worst case, a drive communication fault. With mainline PREEMPT_RT the architecture changes in four ways. The Modbus poller runs as a SCHED_FIFO thread on isolated CPU 1, with its UART interrupt thread pinned and prioritised above it. TLS, MQTT and OPC UA run as SCHED_OTHER on CPU 0. Logs go to tmpfs and are flushed by a low-priority task. And a hardware watchdog supervises the poller’s heartbeat, so a bug becomes a controlled restart instead of a hang.

This is where the relationship to the PLC world becomes concrete. A PLC’s scan cycle has a fixed, monitored period and a watchdog on overrun; our article on PLC scan cycles and jitter explains why. Your Linux RT loop should be designed with the same idea: a fixed period, a measured worst case, and an explicit overrun policy. If the gateway ever moves from observing to actuating, revisit whether Linux RT’s non-provable bound is acceptable or whether the loop should move to a Zephyr microcontroller.

There is also a software-supply-chain angle. As more drivers move to Rust, the mainline kernel’s memory-safety profile improves in exactly the subsystems gateways depend on; see our analysis of Rust in the Linux kernel and its embedded IoT driver implications. Mainline real-time plus a maintained stable tree is a better long-term security posture than a frozen vendor fork.

Trade-offs, Gotchas, and What Goes Wrong

PREEMPT_RT trades throughput for latency. Making locks sleepable and interrupts threaded adds overhead, so peak throughput and average latency can be somewhat worse than a non-RT kernel on the same hardware. How much depends on workload; measure your own application before and after rather than trusting a number from someone else’s board.

New port, new bugs. 32-bit ARM mainline RT is brand new. Expect to find platform and driver issues that x86 and ARM64 users shook out years ago. Pin a stable release, run a soak test of days with your real workload, and subscribe to the linux-rt-users list for reports.

Priority inversion in userspace. Kernel priority inheritance protects rt_mutex and PI futexes. It does not protect spinlocks you write yourself, condition variables used incorrectly, or locks inside libraries such as the C allocator or a logging library. If your RT thread calls into code you do not control, audit it.

Real-time starvation. A runaway SCHED_FIFO thread can starve everything else, including the networking and watchdog-feeding code you need. Keep sched_rt_runtime_us throttling on, test with a deliberately misbehaving task, and use a hardware watchdog.

Hidden latency sources outside the kernel. Memory-bandwidth contention, cache eviction, DVFS transitions, thermal throttling, bootloader-configured secure-world firmware (SMC calls into TrustZone can block a core for long periods on some platforms) and shared buses all land in your tail. None of them is fixed by a kernel option.

Anti-patterns. Do not set every thread to SCHED_FIFO 99. Do not rely on usleep for periodic work. Do not measure with the display or debug UART spewing output. Do not benchmark with FUNCTION_TRACER enabled and call the result final. Do not test with an idle system. And do not claim a hard real-time guarantee based on a test: a test shows what you saw, not what is possible.

Certification. If a missed deadline causes injury or equipment damage, PREEMPT_RT is not a drop-in substitute for a certified safety RTOS or a safety PLC. Treat it as a tool for performance-critical, not safety-critical, functions, and keep safety functions on certified hardware.

Practical Recommendations

Use mainline PREEMPT_RT on 32-bit ARM when your deadline is in the hundreds of microseconds or above, when the device needs a full Linux stack, and when you want to stay on a mainline-tracking kernel for years. Keep Zephyr on a coprocessor or a separate MCU when the hard loop needs microsecond-class, analysable behaviour. Reserve dual-kernel Xenomai for cases where measured results demand it and you accept an out-of-tree patch set.

Sequence the work this way. First, write the latency budget with a percentile and a duration. Second, build the lean RT kernel from a 7.1 stable tree and confirm it boots on your board. Third, run cyclictest with your real load for hours and keep the histogram. Fourth, tune IRQ affinity, isolation and priorities until the budget passes with margin. Fifth, run the application itself and measure end-to-end loop jitter, not just kernel latency. Finally, bake the kernel config, command line and priority layout into your image build so the result is reproducible.

Checklist before shipping:

[ ] uname -v shows PREEMPT_RT on the production image
[ ] Debug, lockdep and tracing options are off in the production config
[ ] CPU frequency policy is fixed and tested
[ ] A latency budget exists and a multi-hour, loaded cyclictest run passes it
[ ] RT threads use mlockall, pre-allocation and PI mutexes
[ ] IRQ threads for critical devices are prioritised deliberately
[ ] A hardware watchdog supervises the RT loop
[ ] The run was repeated on every board revision and storage vendor

Frequently Asked Questions

Does Linux 7.1 really support PREEMPT_RT on 32-bit ARM without patches?

Yes, according to Phoronix’s reporting on the merge: after a cleanup of ARM fault handling shortly before 7.1-rc1, you can build a PREEMPT_RT kernel for 32-bit ARM from mainline, and the v7.1-rc1-rt1 patch release dropped all ARM patches. Remaining out-of-tree items are Intel i915 fixes, a /sys/kernel/realtime flag and PowerPC enablement. Always confirm that your specific SoC and drivers work, since mainline support does not mean every vendor driver is RT-clean.

How do I check that I am running a real-time kernel?

Run uname -v and look for PREEMPT_RT; uname -a and /proc/version show the same string. You can also check /proc/config.gz for CONFIG_PREEMPT_RT=y if your kernel exposes it. Do not rely on /sys/kernel/realtime, because the patch that adds it was reported as still out of tree. In scripts, parse uname -v and fail provisioning when it is missing.

What latency can I expect from cyclictest on ARM?

It depends on the SoC, clock, memory system, drivers and load, so no honest universal number exists. Tuned PREEMPT_RT kernels on embedded ARM commonly show typical latencies in the tens of microseconds, with worst cases under stress in the low hundreds, but treat that as a rough expectation only. Measure with cyclictest -m -p95 -i1000 under realistic load for hours, and judge the maximum and high percentiles of the histogram, not the average.

Is PREEMPT_RT hard real-time?

Not in the formal sense. Hard real-time means a provable upper bound on response, usually from a small analysable kernel and worst-case execution analysis. PREEMPT_RT drastically reduces and bounds latency by design, and in practice meets demanding deadlines, but the kernel is large, drivers vary, and hardware effects exist, so the bound is established by testing. For safety-critical deadlines use a certified RTOS or a safety-rated controller.

When should I choose Zephyr instead of Linux RT?

Choose Zephyr when the deadline is in single-digit to tens of microseconds, when the loop runs on a Cortex-M class core, when you need a small, auditable code base, or when you want minimal power. Choose Linux RT when the device also needs full networking, TLS, containers, filesystems and broad driver support. Many designs use both, with Zephyr on a microcontroller core and Linux on application cores, joined by a message interface.

Is Xenomai still worth using now that PREEMPT_RT is mainline?

Sometimes. The EVL core, the basis of Xenomai 4, still lists ports for 32-bit ARM and tracks mainline kernels, and a dual-kernel design can deliver lower jitter than PREEMPT_RT on some hardware. The cost is an out-of-tree patch, tight kernel-version coupling and a separate real-time API. Measure PREEMPT_RT first; adopt dual-kernel only if your measured tail exceeds the budget and a microcontroller is not an option.