Multi-Sensor Fusion Architecture for Autonomous Robots (2026)

A single sensor never tells an autonomous robot where it is. An inertial measurement unit updates a thousand times a second but drifts within seconds. A lidar returns centimetre-accurate geometry but goes blind in a featureless corridor. A camera is rich and cheap yet cannot recover metric scale from one lens, and it fails the moment a cloud passes the sun. Multi-sensor fusion is the discipline of combining these flawed, complementary streams into one state estimate that is more accurate, higher rate, and far more robust than any input alone. Get it right and your robot localises smoothly through a warehouse; get the time synchronisation wrong by fifteen milliseconds and the same filter diverges into a wall.

This article is a mechanism-level reference architecture, not a survey. It walks the full pipeline — drivers, synchronisation, calibration, prediction, update, and outlier rejection — and shows exactly where each design choice earns its keep or quietly breaks.

What this covers: why complementary sensors beat any single one; the Bayesian estimation core (KF, EKF, UKF, and the error-state Kalman filter); a component-by-component fusion pipeline; a decision matrix across filters and factor graphs; and the failure modes that actually sink real robots.

Context and Background

Multi-sensor fusion sits at the join between perception and control, and it is the layer that turns raw measurements into the pose and velocity every downstream module assumes it can trust. The field grew out of aerospace navigation in the 1960s — the Apollo guidance computer ran a Kalman filter — and it reached mobile robotics through the localisation and SLAM work of the 1990s and 2000s. The canonical text remains Thrun, Burgard, and Fox’s Probabilistic Robotics (2005), which frames the whole problem as recursive Bayesian belief propagation.

Two broad families dominate practice in 2026. Filtering methods maintain a single running estimate and its covariance, updating it measurement by measurement in constant time per step. Smoothing methods, built on factor graphs, keep a window (or the whole history) of past states and re-optimise them jointly whenever new information arrives. Filtering wins on latency and embedded cost; smoothing wins on accuracy and on recovering gracefully after a bad stretch. The incremental smoothing algorithm iSAM2, implemented in the widely used GTSAM library, narrowed the gap by re-solving only the affected part of the graph.

Modern stacks rarely fuse in isolation. The estimator feeds — and is fed by — the mapping and localisation layer described in our SLAM architecture reference, and it publishes the transform tree that motion planners such as Nav2 in a warehouse setting consume every cycle. Fusion is the quiet contract underneath both. When it is healthy nobody notices; when it drifts, everything above it inherits the error.

The Reference Fusion Architecture

Multi-sensor fusion for a mobile robot is a pipeline: sensor drivers feed a time-synchronisation stage, calibration corrects the geometry and biases, an IMU-driven prediction step propagates the state at high rate, and asynchronous measurement updates from lidar, camera, GNSS, and wheel odometry correct it — each gated against outliers before it is allowed to touch the estimate.

That single sentence is the whole design. The direct answer to “how should I fuse sensors on a robot?” is: run a recursive estimator whose fast prediction comes from the IMU and whose corrections come from every slower, more absolute sensor you have, and never let an ungated measurement into the update. Everything below is the detail that makes that safe.

Figure 1: The multi-sensor fusion reference pipeline. Raw driver output is time-aligned, geometrically corrected, propagated by the IMU, then corrected by gated measurement updates before the fused state reaches localization, control, and planning.

Figure 1 reads left to right. Drivers deliver raw, timestamped samples. The synchronisation stage puts every sample on one clock and one time base. Calibration converts each sensor’s private coordinates into the shared body frame and removes known systematic errors. The prediction block runs the motion model forward — on a mobile robot this is almost always IMU integration — and the measurement-update block folds in each slower observation. A feedback path carries every correction back into the next prediction, which is what makes the estimate converge rather than wander.

Why fuse at all: complementary error signatures

The reason to fuse is that sensor errors are complementary in time and in geometry, so their weaknesses cancel. This is the whole justification for the architecture, and it is worth stating precisely rather than as a slogan.

An IMU measures angular rate and specific force at high rate — typically 100 to 1000 Hz — with very low latency, but integrating those signals accumulates unbounded drift, and its bias walks with temperature. Wheel odometry gives a decent local velocity estimate but lies whenever a wheel slips, and it says nothing about the vertical or lateral world. Lidar returns dense, metrically accurate geometry, yet it is comparatively slow (typically 10–20 Hz), sparse in the vertical, and degenerate in self-similar spaces — a long straight corridor gives a scan-matcher no way to fix its position along the corridor’s axis. A camera is information-rich and cheap, but a single lens cannot recover absolute scale, and any camera is at the mercy of lighting, motion blur, and texture. GNSS gives an absolute global position but is noisy, arrives slowly, and is unavailable or multipath-corrupted indoors and in urban canyons.

Fusion buys three things. It improves observability: a state one sensor cannot see (monocular scale, corridor-axis position) becomes observable once another sensor constrains it. It improves robustness: when one sensor drops out or lies, the others hold the estimate. And it improves rate: IMU prediction delivers a smooth, high-frequency pose between the slow, accurate corrections, which is exactly what a control loop needs.

The Bayesian core: predict and update

Every fusion estimator is a recursive Bayesian filter with two alternating operations — a prediction that moves belief forward through a motion model, and an update that sharpens belief with a measurement. Under a Gaussian assumption this belief is fully described by a mean and a covariance, and the recursion becomes the Kalman filter.

The prediction step advances the state estimate x through the motion model f and grows the covariance P by the process noise Q, because moving forward without new evidence makes the robot less certain. The update step computes the innovation — the difference between the actual measurement z and the measurement the model predicted, h(x) — and corrects the state in proportion to the Kalman gain K, which weighs how much to trust the measurement (noise R) versus the prediction (covariance P). Large sensor noise means a small gain and a cautious update; a confident prediction likewise resists a noisy measurement. That weighting, computed optimally every step, is the entire magic of the Kalman filter.

The plain Kalman filter assumes f and h are linear, which robot motion and range/bearing measurements never are. The extended Kalman filter (EKF) handles this by linearising f and h about the current estimate at every step, using their Jacobians F and H — the matrices of partial derivatives. The extended Kalman filter fusion approach remains the default in production because it is cheap, well understood, and good enough when the estimate stays near the true state so the linearisation holds.

When linearisation is the weak link

The EKF’s Achilles heel is that linearising about a wrong estimate can inject error and, in the worst case, drive the filter to diverge. Two well-established alternatives address this from different directions.

The unscented Kalman filter (UKF) skips Jacobians entirely. Instead of linearising, it deterministically samples a small set of “sigma points” around the mean, pushes each through the true nonlinear function, and recovers the transformed mean and covariance from the propagated points. It captures the nonlinearity to higher order than the EKF, costs modestly more, and spares you the error-prone job of deriving Jacobians by hand. For strongly nonlinear measurement models it is often worth the price.

The more important idea for robot fusion is the error-state Kalman filter (ESKF), also called the indirect Kalman filter, which we treat in depth next because orientation is where naive filters quietly go wrong.

Deeper Walk-through: The Error-State Filter and the Fusion Loop

The error-state Kalman filter is the workhorse of IMU-centric fusion because it estimates a small error on top of a nominal state that lives on the correct manifold, which keeps orientation valid and the linearisation honest. Rather than filtering the full state directly, the ESKF integrates the IMU to maintain a high-rate nominal trajectory, and it runs a Kalman filter only on the small deviation between that nominal state and reality.

Figure 2: The ESKF recursive loop. The IMU propagates the nominal state and grows covariance; each measurement’s innovation is gated; accepted measurements update the error state, which is injected into the nominal state and then reset to zero.

Figure 2 shows why this indirection matters. Orientation is not a vector — it lives on the rotation group SO(3), and you cannot simply add a correction to a quaternion and stay on the unit sphere. The ESKF sidesteps this. The nominal state carries the full orientation as a quaternion or rotation matrix; the error state carries orientation error as a small three-element rotation vector in the tangent space, where ordinary linear algebra is valid. After each update the estimated error is injected into the nominal state through the correct manifold operation (a quaternion multiplication, not an addition) and then reset to zero. Sola’s widely cited report “Quaternion kinematics for the error-state Kalman filter” (2017) is the definitive derivation and the reference every implementer should keep open.

Why the error state is small — and why that helps

The error state stays near zero, so linearising about zero is accurate and the filter behaves far better than a full-state EKF. This is the practical payoff. Because the deviation between the nominal trajectory and truth is small and slowly varying, the Jacobians are evaluated at a point where the linear approximation genuinely holds. The dangerous feedback loop of the direct EKF — linearise about a bad estimate, get a worse estimate, linearise about that — is broken.

The typical state a mobile-robot ESKF tracks is position, velocity, orientation, and the two IMU biases — accelerometer bias and gyroscope bias. Estimating the biases inside the filter is not optional; an uncorrected gyro bias of even a fraction of a degree per second integrates into a heading error that will walk your robot off any map within a minute. The filter observes those biases indirectly: whenever lidar or GNSS corrects the position, the correlation the covariance has built up between position error and bias error lets the update nudge the bias too.

The recursive loop, step by step

Each measurement traverses a fixed sequence: predict, compute innovation, gate, update, inject, reset. Understanding that sequence is understanding the filter.

Between measurements the IMU runs the prediction alone, integrating acceleration and angular rate to push the nominal pose forward at full IMU rate and grow the covariance. When a slower measurement arrives — a lidar scan-match result, a set of visual feature observations, a GNSS fix, a wheel-odometry velocity — the filter forms the innovation, the gap between what it observed and what it expected. It then gates that innovation (the next section explains how), and only if the measurement passes does it compute the gain, update the error state, inject it into the nominal state, and reset. The corrected nominal state is what the rest of the robot consumes, at IMU rate, smoothed and drift-corrected.

Loose versus tight coupling

Coupling describes how much raw sensor detail enters the filter, and it is the single biggest architectural fork in visual-inertial and lidar-inertial fusion. In loose coupling each sensor is processed to a pose or velocity by its own pipeline first — a visual odometry front-end outputs a pose, a lidar matcher outputs a pose — and the filter fuses those derived poses. It is modular, easy to build, and fails softly, but it throws away information and cannot exploit a sensor that is only partly constrained.

In tight coupling the raw features enter the estimator directly — individual visual feature tracks, individual lidar residuals — and are fused jointly with the IMU in one estimator. This is strictly more powerful: a monocular camera plus an IMU can recover metric scale only when coupled tightly, because scale emerges from the interplay of visual parallax and inertial acceleration. The Multi-State Constraint Kalman Filter (MSCKF) of Mourikis and Roumeliotis (2007) is the classic tightly coupled visual-inertial filter, and VINS-Mono (Qin, Li, and Shen, 2018) is the classic tightly coupled optimisation-based estimator. Tight coupling demands excellent time sync and calibration; loose coupling forgives sloppiness at the cost of accuracy.

Time Synchronisation, Calibration, and Coordinate Frames

Before any filter math is trustworthy, three unglamorous prerequisites must hold: every sample must sit on a common clock, every sensor’s geometry and biases must be calibrated, and every measurement must be expressed in a well-defined coordinate frame. Skip any one and the estimator will produce confident nonsense.

Time synchronisation

Fusion assumes it knows when each measurement was taken, and an error there maps directly into a position error scaled by the robot’s speed. A robot moving at two metres per second with a ten-millisecond timestamp error places a lidar return two centimetres from where it belongs — enough to smear a map and bias a scan-match.

The gold standard is hardware synchronisation: a common trigger line or a Precision Time Protocol (PTP, IEEE 1588) clock shared across sensors, so each sample carries a timestamp from the same clock domain. Where hardware sync is unavailable, software timestamping at the driver is the fallback, and it must account for the sensor’s internal latency and bus transport delay rather than stamping on arrival. Because sensors run at different rates and phases, the filter also interpolates: to fuse a camera frame captured between two IMU samples, you interpolate the IMU-propagated state (or the rotation, via spherical linear interpolation) to the exact capture instant. Getting the effective timestamp right, latency included, matters more than raw clock precision.

Calibration: intrinsics and extrinsics

Calibration tells the filter how each sensor is built and where it sits, and an error here is a constant bias the filter cannot distinguish from real motion. Two kinds matter. Intrinsic calibration models a single sensor: a camera’s focal length, principal point, and lens distortion; an IMU’s bias, scale-factor, and axis-misalignment terms. Extrinsic calibration is the rigid transform between sensors — the lidar-to-IMU, camera-to-IMU, and camera-to-lidar six-degree-of-freedom poses.

Extrinsic error is insidious because it looks like signal. A camera-to-IMU rotation error of one degree will be read by a tight visual-inertial filter as a small persistent motion, corrupting the bias estimates. This is why tools such as Kalibr exist to jointly estimate camera intrinsics, camera-IMU extrinsics, and time offset from a single calibration sequence, and why serious stacks re-verify calibration after any mechanical change. The hand-eye calibration formulation (solving AX = XB for the sensor-to-body transform) is the classical method for recovering an extrinsic from synchronised motion. Some estimators go further and treat the extrinsics and time offset as slowly varying states inside the filter — online calibration — so they track thermal and mechanical drift rather than trusting a one-time bench result.

Coordinate frames and the TF tree

Every measurement lives in some frame, and fusion is largely the disciplined bookkeeping of transforming between them, so a robot maintains an explicit tree of frames. The ROS community codified this in REP-105, which every practitioner should treat as canonical.

Figure 3: The REP-105 transform tree. The map frame is a fixed world reference; odom is a smooth but drifting frame; base_link is the robot body; each sensor hangs off base_link by a fixed extrinsic. Localization corrects map to odom; the estimator drives odom to base_link continuously.

Figure 3 captures the convention. The map frame is world-fixed and metrically consistent but can jump when a global correction (a lidar relocalisation or a GNSS fix) arrives. The odom frame is smooth and continuous — it never jumps — but it drifts without bound. base_link is rigidly attached to the robot, and each sensor frame (lidar_link, camera_link, imu_link) hangs off base_link by its calibrated extrinsic. The crucial design rule from REP-105 is that the fused high-rate estimate publishes the continuous odom-to-base_link transform, while the global localiser publishes the correcting map-to-odom transform. Controllers consume the smooth frame so they never see a discontinuity; planners consume the consistent one. Confusing which estimate belongs in which frame is one of the most common integration bugs in real stacks.

Trade-offs, Gotchas, and What Goes Wrong

Fusion fails in a small number of well-understood ways, and every one of them traces back to a violated assumption — about time, geometry, noise, or observability. Knowing the failure catalogue is more useful than knowing the happy path, because a filter that works in the lab and diverges in the field almost always tripped one of these.

Bad time sync is the most common and most punishing. A constant timestamp offset couples rotation into apparent translation and biases the whole estimate; a jittery offset injects noise the filter cannot model. It is the first thing to check when a working stack degrades after a hardware change.

Uncalibrated or drifting extrinsics masquerade as motion, so the filter “corrects” a phantom, corrupting bias and scale estimates. Thermal expansion and vibration slowly change extrinsics over a long mission, which is the argument for online calibration.

IMU bias walk and thermal effects are relentless. If the filter’s process-noise model for the biases is too tight, it stops tracking the walking bias and the heading drifts; too loose, and the estimate gets noisy. Tuning Q for the bias states is a real engineering task, not a default.

Over-optimistic covariances are a classic path to divergence. If you tell the filter a measurement is more precise than it is (R too small) or that its prediction is better than it is (P grown too little), the filter becomes overconfident, gates out the very measurements that would correct it, and locks onto a wrong trajectory — an inconsistent filter. Consistency is checked formally with the Normalised Estimation Error Squared (NEES) in simulation and the Normalised Innovation Squared (NIS) online; both should sit inside their chi-square bounds. A filter that is statistically inconsistent will eventually diverge even if it looks fine on a short run.

Degenerate geometry defeats even a perfect filter. A lidar in a long featureless corridor cannot observe position along the corridor; a camera facing a blank wall gives no parallax. The fix is to detect degeneracy (for example, by inspecting the eigenvalues of the scan-matching information matrix) and to lean on the complementary sensor while it lasts, rather than trusting a rank-deficient update.

Single points of failure and sensor dropout must be planned for. A robust stack treats every sensor as droppable: if the camera saturates or the lidar returns nothing, the filter should widen its covariance and coast on the survivors, then re-acquire cleanly. GNSS deserves special caution — multipath in urban canyons returns confident, wrong fixes, so GNSS updates need aggressive gating rather than blind trust.

Outlier rejection is the mechanism that contains most of these. Every measurement is tested with a Mahalanobis-distance gate: the innovation, scaled by its own covariance, must fall below a chi-square threshold for the measurement’s degrees of freedom, or the measurement is discarded. Front-ends add RANSAC to reject spurious feature matches, and optimisation-based estimators wrap residuals in a robust cost such as the Huber loss so a few bad correspondences bend rather than break the solution. Gating is not optional garnish; it is the difference between a filter that survives a glass door and one that chases a reflection into it.

The last structural trade-off is the choice of estimator itself, shown in Figure 4.

Figure 4: Choosing the estimator. Whether you need the full trajectory, how tight your compute budget is, and whether you have raw features or per-sensor poses together decide between a filter and a factor graph, and between loose and tight coupling.

Practical Recommendations

Build the pipeline in the order the errors compound: fix time and geometry first, then choose an estimator, then tune, then harden against dropout. Teams that reverse this — reaching for a fancier filter to paper over a sync bug — burn weeks. The estimator can only be as good as the timestamps and calibration feeding it.

Start with an error-state Kalman filter fusing IMU with wheel odometry and lidar; it is the highest robustness-per-unit-effort configuration for a ground robot and it gives you a smooth, high-rate pose immediately. Add a camera only once you can justify the tight-coupling complexity, and add GNSS only with hard gating. Treat every extrinsic and time offset as something to verify after any mechanical change, and prefer online calibration for long-running platforms.

Instrument consistency from day one — log NIS per sensor and watch for a filter that gates out too much or too little. Budget compute honestly: an ESKF runs comfortably on an embedded CPU, while a tightly coupled sliding-window optimiser or an iSAM2 factor graph may want a Jetson-class GPU or a dedicated core. When in doubt, favour the estimator you can debug over the one with the best benchmark.

A practical pre-deployment checklist:

• [ ] Every sensor timestamped from a common clock (PTP or hardware trigger) with latency accounted for.
• [ ] Intrinsics, extrinsics, and time offsets calibrated and re-verified after mechanical changes.
• [ ] TF tree conforms to REP-105; the smooth estimate publishes odom→base_link, the global fix publishes map→odom.
• [ ] IMU biases estimated inside the filter, with tuned process noise.
• [ ] Mahalanobis gating on every update; RANSAC or Huber on feature front-ends.
• [ ] NEES/NIS consistency logged; covariances neither over- nor under-confident.
• [ ] Sensor-dropout and degeneracy behaviour tested, not assumed.

Decision matrix: EKF vs UKF vs ESKF vs factor graph

Criterion	EKF	UKF	ESKF	Factor graph (iSAM2)
Handles nonlinearity	Linearise via Jacobians	Sigma points, higher order	Linearise about small error, accurate	Full nonlinear re-optimisation
Orientation on SO(3)	Awkward, drifts	Awkward	Correct, native manifold handling	Correct, native
Needs Jacobians by hand	Yes	No	Yes, but simple error dynamics	Uses analytic/auto factors
Compute cost	Lowest	Low–moderate	Low	Highest (bounded by iSAM2)
Latency	Constant, lowest	Constant	Constant	Higher, re-solves window
Recovers after bad stretch	Poorly	Poorly	Moderately	Best (relinearises past)
Loop closure / re-visits	No	No	No	Yes, native
Typical use	Simple odometry fusion	Strongly nonlinear models	IMU-centric mobile robots	VIO/SLAM, offline-quality online

Read the matrix as a gradient, not a ranking. The ESKF is the pragmatic default for a real-time mobile robot; the factor graph is where you go when accuracy and loop closure justify the compute; the UKF is the answer when Jacobians are painful or the model is sharply nonlinear; the plain EKF still earns its place in lightweight, well-behaved odometry fusion. This same filter-versus-smoother tension shapes the estimator inside a full humanoid control stack, where state must be both fast enough for balance and accurate enough for manipulation.

Frequently Asked Questions

What is multi-sensor fusion in robotics?

Multi-sensor fusion is the process of combining measurements from several sensors — commonly an IMU, wheel encoders, lidar, cameras, and GNSS — into a single estimate of the robot’s state, typically its position, velocity, and orientation. It works because sensor errors are complementary: fast-but-drifting inertial data is corrected by slow-but-accurate geometric data, and a quantity one sensor cannot observe becomes observable once another constrains it. The result is a state estimate that is more accurate, higher rate, and more robust to any single sensor failing than any input on its own.

What is the difference between an EKF and an error-state Kalman filter?

A standard extended Kalman filter estimates the full robot state directly and linearises its motion and measurement models about the current — possibly wrong — estimate, which can inject error and cause divergence. An error-state Kalman filter instead keeps a high-rate nominal state on the correct manifold (integrating the IMU) and runs the Kalman filter only on the small error between that nominal state and reality. Because the error stays near zero, linearisation is accurate, and orientation is handled correctly on SO(3) via inject-and-reset. For IMU-centric fusion the ESKF is more numerically stable and is the common production choice.

Why is time synchronisation so critical in sensor fusion?

Because fusion assumes it knows exactly when each measurement was taken, a timestamp error maps directly into a position error scaled by the robot’s speed. A ten-millisecond error on a robot moving two metres per second misplaces a measurement by two centimetres, which smears maps and biases scan-matching. Worse, a constant offset couples rotation into apparent translation and corrupts bias estimates. Hardware synchronisation (a shared trigger or PTP clock) is the gold standard; software timestamping must still account for sensor and bus latency rather than stamping on arrival.

What is the difference between loose and tight coupling?

Loose coupling processes each sensor into a pose or velocity independently and then fuses those derived outputs; it is modular and forgiving but discards information. Tight coupling feeds raw measurements — individual visual features or lidar residuals — directly into one joint estimator alongside the IMU. Tight coupling is strictly more powerful: a monocular camera plus an IMU can recover metric scale only when tightly coupled. The price is that tight coupling demands excellent time synchronisation and calibration, whereas loose coupling tolerates sloppiness at the cost of accuracy.

When should I use a factor graph instead of a Kalman filter?

Choose a factor graph (with an incremental solver such as iSAM2) when you need the full trajectory rather than just the latest pose, when loop closures and re-visits matter, and when you can afford the extra compute. Factor graphs re-optimise past states jointly, so they recover far better after a bad stretch and handle loop closure natively. Choose a Kalman filter — usually an ESKF — when you need constant-time, low-latency updates on a tight compute budget and only the current state matters, as in most real-time control loops on embedded hardware.

How do robots reject bad sensor measurements?

The core mechanism is a Mahalanobis-distance gate: each measurement’s innovation is scaled by its covariance, and if that scaled distance exceeds a chi-square threshold for the measurement’s degrees of freedom, the measurement is discarded rather than fused. Feature-based front-ends add RANSAC to reject spurious matches, and optimisation-based estimators use robust cost functions such as the Huber loss so a few outliers bend the solution instead of breaking it. Gating is essential for surviving reflections, multipath GNSS, and moving objects that violate the static-world assumption.

Further Reading

• SLAM architecture for autonomous robots: localization and mapping — how the fused estimate feeds and is corrected by simultaneous localization and mapping.
• ROS 2 and Nav2 for autonomous mobile robot warehouse navigation — the planning and control stack that consumes the transform tree fusion publishes.
• Humanoid robot control stack architecture — state estimation under the harder real-time constraints of a balancing, manipulating robot.
• Thrun, Burgard, and Fox, Probabilistic Robotics (MIT Press, 2005) — the foundational text on recursive Bayesian estimation for robots.
• Sola, “Quaternion kinematics for the error-state Kalman filter” (2017, arXiv:1711.02508) — the definitive ESKF derivation for orientation.
• Mourikis and Roumeliotis, “A Multi-State Constraint Kalman Filter for Vision-aided Inertial Navigation” (ICRA 2007) — the canonical tightly coupled visual-inertial filter.
• Qin, Li, and Shen, “VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator” (IEEE T-RO, 2018) — the reference optimisation-based VIO system.
• Kaess et al., “iSAM2: Incremental Smoothing and Mapping Using the Bayes Tree” (IJRR, 2012) and GTSAM — factor-graph smoothing in practice.
• ROS REP-105: Coordinate Frames for Mobile Platforms — the map/odom/base_link convention.

By Riju — about