Hand-Eye Calibration for Robot Arms: Eye-in-Hand vs Eye-to-Hand and AX=XB

A robot arm can repeat a joint trajectory to a fraction of a millimetre, and a camera can find a part to within a pixel. Put the two together without a correct hand-eye calibration and the gripper still misses the part by centimetres. The camera reports where the part is in the camera’s frame, the controller wants it in the robot’s base frame, and the rigid transform between those frames is something nobody measured. Every pick-and-place, bin-picking, visual-servoing or learned-policy deployment depends on that one transform being right.

This matters more in 2026 because vision-driven manipulation is spreading from fixed cells into mobile and learning-based systems, where the camera is bolted on by hand and re-mounted often. Calibration is the cheapest accuracy you will ever buy, and the most common thing teams get subtly wrong.

You will leave with the geometry, the AX=XB and AX=ZB formulations, an honest comparison of the classical solvers, a runnable OpenCV example on synthetic data, and a verification routine you can trust.

What this covers: coordinate frames and transforms, intrinsics, eye-in-hand versus eye-to-hand setups, solver choice, pose selection and conditioning, error budgets, a ROS 2 workflow, verification, and failure modes.

Context and Background

Calibration for a vision-guided arm is really three separate problems stacked on top of each other, and confusing them is the root of most bad results.

First, camera calibration recovers the intrinsics: focal lengths, principal point and lens distortion. This is a property of the camera and lens alone and does not involve the robot. Second, robot kinematic calibration corrects the arm’s own geometric model, because real link lengths and joint offsets differ from the CAD values. Third, hand-eye calibration recovers the fixed rigid transform between the camera and the robot. It assumes the first two are already good enough, which is why a poor intrinsic calibration quietly corrupts everything downstream.

The classical formulation dates to the late 1980s. Tsai and Lenz published an efficient method for the eye-in-hand problem in 1989, and later work by Park and Martin, Horaud and Dornaika, Andreff and colleagues, and Daniilidis added closed-form and dual-quaternion alternatives. OpenCV exposes five of these through a single function, cv::calibrateHandEye, with the methods Tsai, Park, Horaud, Andreff and Daniilidis, and Tsai as the default (OpenCV calib3d documentation). It also provides cv::calibrateRobotWorldHandEye for the related robot-world formulation, with the Shah and Li methods.

On the ROS side, the MoveIt Calibration package offers a GUI panel in RViz that supports both eye-in-hand and eye-to-hand mounts, defaults to the Daniilidis solver, and uses an ArUco-style marker target (MoveIt hand-eye calibration tutorial). Its tutorial notes that a minimum of five samples is required before a result is computed, and that quality typically plateaus around twelve to fifteen. Treat those as the package’s own guidance, not a universal law. The tutorial also contains build steps that look like ROS 1 tooling, so check them against your distribution; our ROS 2 Jazzy to Kilted migration guide covers the distribution side.

Hand-eye calibration also sits beside other perception work on this site. If your camera is part of a mapping stack, the extrinsics you calibrate here feed the sensor frames discussed in our SLAM architecture for autonomous robots post, and if you train policies in simulation, an accurate camera pose is one of the cheapest ways to narrow the gap covered in sim-to-real transfer and domain randomization.

The plan for the rest of this post: first the frames and the two classical setups, then the algebra, then code, then the practical engineering that decides whether your number is trustworthy.

Frames, Transforms and the Two Hand-Eye Setups

Hand-eye calibration finds one fixed rigid transform, either camera-to-gripper or camera-to-base, depending on where the camera is mounted. In an eye-in-hand setup the camera rides on the end effector and the unknown is the gripper-to-camera pose. In an eye-to-hand setup the camera is fixed in the workcell and the unknown is the camera-to-base pose. Both reduce to solving a matrix equation from several robot poses.

Homogeneous transforms in one paragraph

A pose in 3D is a rotation matrix R in SO(3) and a translation vector t. We pack them into a 4×4 homogeneous matrix T so that composing poses is matrix multiplication. The notation ^aT_b maps coordinates expressed in frame b into frame a. Chains read left to right: ^baseT_cam = ^baseT_gripper · ^gripperT_cam. Getting the direction convention right is the single most frequent source of “my calibration is mirrored” bugs. OpenCV names its arguments R_gripper2base and R_target2cam, which means the transform that takes a point in the gripper frame to the base frame, and a point in the target frame to the camera frame.

Four frames matter. The base frame is fixed to the robot’s foot. The gripper (or tool flange) frame moves with the last link and its pose comes from forward kinematics, so it is only as accurate as the arm’s kinematic model. The camera frame has its origin at the optical centre. The target frame is attached to a calibration board, and its pose in the camera frame comes from a perspective-n-point solution on detected corners. If you are new to how the gripper pose is computed from joint angles, our inverse kinematics and Jacobian guide covers the forward and inverse maps.

Eye-in-hand: the AX=XB problem

In the eye-in-hand case the board sits still in the world and the robot moves the camera to several poses. Between any two stations i and j, two relative motions can be computed independently. The gripper moved by A = (^baseT_gripper,i)^-1 · ^baseT_gripper,j, read from the robot. The camera moved by B = ^camT_target,i · (^camT_target,j)^-1, read from the vision system. Because the camera is rigidly attached to the gripper through the unknown X = ^gripperT_cam, the same physical motion expressed in two frames must satisfy

A X = X B

This is the well-known AX=XB equation. It splits into a rotation part, R_A R_X = R_X R_B, and a translation part, R_A t_X + t_A = R_X t_B + t_X. The rotation part alone determines R_X, and then the translation part becomes a linear system in t_X.

Figure 1: Frames and unknowns for eye-in-hand calibration. The board stays fixed, the camera moves with the gripper, and X is the constant transform between them.

The figure shows why the problem is solvable and also why it can fail. One relative motion gives two rotation constraints that fix the rotation axis alignment but leave a one-parameter ambiguity about it. At least two relative motions whose rotation axes are not parallel are needed to pin down R_X uniquely. In practice you collect many more, because every measurement carries noise. This requirement of non-parallel axes is the origin of the advice, repeated in the MoveIt tutorial, to rotate around at least two axes between samples.

Eye-to-hand: the AX=ZB formulation

When the camera is fixed and the board is held by the gripper, the roles shift. The board is rigidly attached to the gripper through an unknown transform, and the camera sits at an unknown pose in the base frame. Chaining base to gripper to board to camera gives ^baseT_gripper,i · ^gripperT_board = ^baseT_cam · ^camT_board,i for every station i.

There are two unknown constant transforms here: the camera-in-base pose and the board-in-gripper pose. You can treat it two ways. The first rewrites it as an AX=XB problem on relative motions, exactly like the eye-in-hand case, by inverting the robot motion. OpenCV’s calibrateHandEye supports this: for eye-to-hand you pass the inverse of the gripper-to-base poses, i.e. base-to-gripper, and the result is the camera expressed in the base frame. The second approach solves the AX=ZB (robot-world, hand-eye) problem directly for both unknowns simultaneously, which is what calibrateRobotWorldHandEye does with the Shah or Li methods. That function returns the base-to-world transform and the gripper-to-camera transform together.

Property	Eye-in-hand	Eye-to-hand
Camera mount	On the end effector	Fixed in the cell
Unknown	Gripper to camera	Camera to base
Field of view	Changes with the arm	Constant and wide
Occlusion by arm	Rare	Common
Accuracy near target	Excellent, close view	Limited by distance
Recalibrate when	Camera or lens bumped	Camera or base moved
Typical use	Bin picking, inspection	Workcell monitoring, handover

Where the choice leaves a signature in error

Eye-in-hand gives high resolution on the object because the camera can approach it, but its pose error is amplified by the arm: a small rotational error in X turns into a translation error proportional to the camera-to-object distance. Eye-to-hand fixes the camera and removes that lever arm from the pick, but the object is usually imaged from farther away, so the pixel footprint is coarser and the arm itself may block the view. Neither is universally better, and many production cells use both.

Intrinsics, Targets and the Solver Landscape

Before any hand-eye step, the camera must be calibrated, because the target pose in the camera frame comes from a perspective-n-point solve that depends on the intrinsic matrix and distortion model. An error here is not random noise that averages out; it is a systematic bias that moves with the viewing angle, and AX=XB will happily absorb it into X.

Pinhole model, distortion and target choice

The pinhole model maps a 3D point in camera coordinates to pixels through the intrinsic matrix K, which holds the focal lengths fx and fy and the principal point cx and cy. Real lenses add radial and tangential distortion, which OpenCV models with polynomial coefficients (k1, k2, p1, p2, k3 and optionally more). Wide-angle and fisheye lenses need the dedicated fisheye model, since the standard polynomial does not describe them well at large field angles.

Intrinsics are recovered from many views of a known planar pattern. A plain checkerboard works, but a ChArUco board, which interleaves a chessboard with ArUco markers, is usually better for robotics. It remains usable when partially occluded or viewed at a steep angle, and its corner detections carry identifiers, so you never get a flipped or rotated assignment. Print it on a rigid flat substrate, because a warped print breaks the planar assumption, and measure the printed square size with calipers rather than trusting the file.

Use the same focus and resolution during intrinsic calibration and during operation. Autofocus lenses change their effective focal length when they refocus, so lock focus. Report the reprojection error from the intrinsic step as a sanity check, but remember that a low value does not prove the model is correct; it only shows the model fits the views you gave it.

The classical solvers, compared

All of the classical solvers estimate rotation first and translation second, differing in how they represent rotation and whether they decouple the two.

Figure 2: Calibration pipeline. Each station yields one robot pose and one target pose; the solver consumes all stations and the result is checked with an independent test.

Solver	Rotation representation	Rotation and translation	Notes
Tsai-Lenz (1989)	Rotation axis and angle	Decoupled, linear	Fast, historically the default; sensitive to rotation noise
Park and Martin (1994)	Lie group logarithm	Decoupled, least squares	Closed form, good conditioning with few poses
Horaud and Dornaika (1995)	Quaternions	Nonlinear in some variants	Quaternion-based; behaviour depends on the variant implemented
Andreff et al. (1999)	Linear in Kronecker form	Linear system	Linear formulation; check the paper for its motion requirements
Daniilidis (1999)	Dual quaternions	Simultaneous	Estimates rotation and translation together; MoveIt’s default

The descriptions of method families above come from the original papers and surveys; I have not reproduced any quantitative claim about which is “most accurate”, because published comparisons disagree and depend on noise levels, motion ranges and implementations. Decoupled methods propagate rotation error into the translation estimate, which is the main argument for simultaneous methods. On the other hand simultaneous methods are not automatically better when the data are poorly conditioned.

A practical position: run two or three solvers on the same data and compare. If the results agree to within your tolerance, the dataset is probably well conditioned. If they disagree by a lot, the solver choice is not the problem; the data are. This is the cheapest diagnostic available, and OpenCV lets you do it in a loop of a few lines.

Closed-form solutions minimise algebraic errors, not the quantity you care about. After the initial estimate, a nonlinear refinement that minimises reprojection error of the board corners (or the pose residuals over all stations) typically improves the result, particularly when pose noise is not isotropic. Camera pose error is much larger along the optical axis than across it, so a good cost function weights those directions differently. OpenCV does not expose this refinement for hand-eye directly, so teams either use Ceres or SciPy least-squares starting from the closed-form solution, or use a ROS package that bundles it.

From joint angles to gripper pose

The gripper pose in the base frame comes from forward kinematics of the robot’s nominal model, or from the controller’s reported tool-centre-point pose. This is where a hidden assumption lives: hand-eye calibration treats the robot pose as ground truth. If the arm’s absolute accuracy is poor, the calibration quietly absorbs that error into X and the result is valid only in the region where you sampled. Keep the sampled workspace similar to the workspace you will operate in.

Walk-through: A Runnable Calibration on Synthetic Data

The fastest way to understand what a calibration can and cannot recover is to generate data where you know the answer. The script below builds a ground-truth gripper-to-camera transform, simulates twenty robot stations looking at a fixed board, adds small noise to the vision poses, and then recovers X with the Park and Martin closed-form method, written in plain NumPy so there is nothing to install. It then runs the consistency check that matters in the real world: if X is right, the board position computed from every station through T_base_gripper · X · T_cam_target should be the same point.

import numpy as np
rng = np.random.default_rng(7)

def hat_exp(w):                      # axis-angle vector -> rotation matrix
    th = np.linalg.norm(w)
    if th < 1e-12: return np.eye(3)
    k = w / th
    K = np.array([[0, -k[2], k[1]], [k[2], 0, -k[0]], [-k[1], k[0], 0]])
    return np.eye(3) + np.sin(th) * K + (1 - np.cos(th)) * K @ K

def log_so3(R):                      # rotation matrix -> axis-angle vector
    th = np.arccos(np.clip((np.trace(R) - 1) / 2, -1, 1))
    if th < 1e-12: return np.zeros(3)
    w = np.array([R[2, 1] - R[1, 2], R[0, 2] - R[2, 0], R[1, 0] - R[0, 1]])
    return th / (2 * np.sin(th)) * w

def T_of(R, t):
    T = np.eye(4); T[:3, :3] = R; T[:3, 3] = t; return T

def park_martin(T_g2b, T_t2c):       # solves A X = X B
    A, B = [], []
    for i in range(len(T_g2b) - 1):
        A.append(np.linalg.inv(T_g2b[i]) @ T_g2b[i + 1])   # gripper motion
        B.append(T_t2c[i] @ np.linalg.inv(T_t2c[i + 1]))   # camera motion
    M = sum(np.outer(log_so3(b[:3, :3]), log_so3(a[:3, :3])) for a, b in zip(A, B))
    w, V = np.linalg.eigh(M.T @ M)
    RX = V @ np.diag(w ** -0.5) @ V.T @ M.T
    C = np.vstack([a[:3, :3] - np.eye(3) for a in A])
    d = np.concatenate([RX @ b[:3, 3] - a[:3, 3] for a, b in zip(A, B)])
    return T_of(RX, np.linalg.lstsq(C, d, rcond=None)[0])

X_true = T_of(hat_exp(np.array([0.1, -0.05, 0.3])), [0.04, -0.02, 0.09])
T_b_t = T_of(hat_exp(np.array([np.pi, 0, 0.2])), [0.5, 0.1, 0.0])

def station(rot_sigma, trans_sigma):
    t = np.array([0.5, 0.1, 0.45]) + rng.uniform(-0.12, 0.12, 3)
    R = hat_exp(np.array([np.pi, 0, 0.2])) @ hat_exp(rng.uniform(-0.5, 0.5, 3))
    T_g = T_of(R, t)
    T_c_t = np.linalg.inv(T_g @ X_true) @ T_b_t           # what the camera sees
    T_c_t[:3, :3] = hat_exp(rng.normal(0, rot_sigma, 3)) @ T_c_t[:3, :3]
    T_c_t[:3, 3] += rng.normal(0, trans_sigma, 3)          # vision noise
    return T_g, T_c_t

st = [station(0.002, 0.001) for _ in range(20)]
X = park_martin([s[0] for s in st], [s[1] for s in st])
ang = np.degrees(np.arccos(np.clip((np.trace(X[:3, :3].T @ X_true[:3, :3]) - 1) / 2, -1, 1)))
print(f"rot err {ang:.3f} deg, trans err {np.linalg.norm(X[:3,3]-X_true[:3,3])*1000:.2f} mm")
pos = np.array([(g @ X @ c)[:3, 3] for g, c in st])
print("board scatter mm", (pos.std(axis=0) * 1000).round(2))

With the seed above, I ran this and got a rotation error of about 0.32 degrees and a translation error of about 0.8 millimetres, with a board-position scatter near 1 millimetre per axis. Those numbers are specific to this synthetic noise level (2 milliradians of rotation noise, 1 millimetre of position noise) and should not be read as expected real-world accuracy. They illustrate the mechanism: vision noise of a millimetre in, roughly a millimetre of residual scatter out.

Using OpenCV for the same job

In a real project you would feed OpenCV’s solver instead. The inputs are lists of rotation matrices and translation vectors, one per station: gripper-to-base from the robot, and target-to-camera from your board pose estimation. The call is as documented in the API reference, and the returned pair is the camera-to-gripper transform for eye-in-hand.

import cv2
R_cam2grip, t_cam2grip = cv2.calibrateHandEye(
    R_gripper2base, t_gripper2base,   # lists of 3x3 and 3x1 arrays, from the robot
    R_target2cam,  t_target2cam,      # lists of 3x3 and 3x1 arrays, from solvePnP
    method=cv2.CALIB_HAND_EYE_DANIILIDIS)

For eye-to-hand, pass the inverse robot poses (base-to-gripper) in the first two arguments and read the result as the camera pose in the base frame. The OpenCV hand-eye interface is stable across the 4.x series; I did not exercise this specific call in the environment used to draft this post, so test it on your installed version. Run it in a loop over the five method constants and print the translation of each. Disagreement larger than your accuracy budget is a data problem, not a solver problem.

Getting the board pose right

Each station needs a good T_target_cam. Detect ChArUco corners with the cv2.aruco module, then estimate the board pose from corners and the intrinsic matrix with a PnP solve. Three practical rules improve results. Reject frames with few detected corners, since PnP on four points is fragile. Keep the board reasonably oblique, because a fronto-parallel view gives weak depth information. Also average several consecutive frames at each station, after the arm has fully stopped, to reduce sensor noise without introducing motion blur.

Time synchronisation matters as much as detection. If the robot pose is read while the arm is still settling, or the image is captured with a latency of tens of milliseconds, the pair is inconsistent and the error is proportional to the speed at the moment of capture. Move, wait for the controller to report zero velocity, then capture and read the pose together.

Pose Selection, Conditioning and the Error Budget

The solver is the easy part. The quality of a calibration is decided by which poses you collect, how well the geometry constrains the unknowns, and how large the other error sources are compared with the one you are trying to remove.

Why rotation diversity dominates

The rotation of X is observable only through relative motions whose rotation axes differ. If every motion rotates about nearly the same axis, the matrix M in the Park and Martin derivation becomes nearly rank-deficient, and R_X along the missing direction is determined by noise. Translation inherits the problem, because the linear system for t_X uses the matrices (R_A − I), which lose rank when rotation angles are small.

Figure 3: Good and poor pose sets. Rotation about two or more independent axes with large angles conditions the problem; small, same-axis motions leave directions unobserved.

A concrete rule of thumb that follows from the algebra: relative rotation angles should be large enough that rotation signal dwarfs vision noise. If your board-pose noise is a few tenths of a degree, a 5 degree inter-station rotation gives a poor signal-to-noise ratio, while 20 to 40 degrees gives a good one. Do not exceed what the board detector can handle, because beyond roughly 60 degrees of tilt corner detection degrades and the PnP error grows. Treat these figures as engineering heuristics from the conditioning argument, not as numbers from a standard.

A pose-selection recipe

Choose a workspace region matching the real task, then define a dome of viewpoints over the board.
Command stations that vary roll, pitch and yaw by at least tens of degrees each, not just position.
Vary the camera-to-board distance across a modest range, but keep the board large enough in the image for stable corner detection.
Collect more stations than the minimum. The MoveIt tutorial says computation begins after five samples and that results usually plateau around twelve to fifteen, which is a reasonable operating target for a first attempt.
Score each candidate set by solving on subsets and checking variance across subsets; a set whose leave-one-out solutions differ by more than your tolerance is under-constrained.

Automated pose generators help, because humans tend to cluster samples near a comfortable pose. They also help make recalibration a ten-minute routine instead of an afternoon of jogging, which in turn means it actually gets done after a bump.

Repeatability versus accuracy

Industrial arm datasheets quote repeatability, how closely the arm returns to a previously visited pose, and it is usually far better than absolute accuracy, how closely the arm reaches a commanded Cartesian pose in space. Hand-eye calibration only needs the robot pose to be consistent with the pose it reports, but that report comes from the kinematic model, and the model is wrong in small ways. The consequence is that your calibrated X is a best fit over the sampled region, and the residual error grows as you leave it. That is why calibrating in the same region you will work in matters more than adding stations far away.

The error budget

Treat the end-to-end positioning error as a sum of independent contributions, roughly combined in quadrature when they are uncorrelated:

Camera model error: intrinsics and distortion residuals, worst at the image edges.
Board pose noise: corner localisation, depth ambiguity along the optical axis, board print errors.
Robot kinematic error: link-length and joint-offset deviations from the nominal model, gear backlash, payload deflection.
Hand-eye solver error: conditioning and noise amplification discussed above.
Thermal drift: link expansion and joint warm-up, which many controllers do not compensate. Allow the arm to run through a warm-up routine before calibrating, and recalibrate if the operating temperature differs from the calibration temperature.
Mount stability: a camera clamped on a flexible bracket changes pose under acceleration. This one is mechanical, not mathematical, and no solver fixes it.
Timing: image and pose skew, scaling with arm velocity.

The practical insight is that the budget is usually dominated by the largest one or two terms. If your arm’s absolute accuracy is a millimetre or two, spending days squeezing hand-eye error below 0.1 millimetre does not improve the pick. Measure the budget before optimising any term.

Rotation error is amplified by distance

A small angular error in X multiplies by the lever arm. With the camera 0.3 metres from the object, an angular error of 0.5 degrees (about 8.7 milliradians) in the camera orientation moves the computed object position sideways by roughly 2.6 millimetres, since arc length is angle times radius. At 0.6 metres the same angular error produces about 5.2 millimetres. This simple geometry explains why translation accuracy tends to be good near the calibration board and degrades at longer working distances, and why you should calibrate at the distance you will work at.

Practical ROS 2 Workflow and Verification

A calibration is only useful if its result ends up in the robot’s transform tree and has been checked independently of the data that produced it.

The workflow in a ROS 2 cell

Figure 4: A ROS 2 workflow. Capture synchronized samples, solve, publish the result as a static transform, and gate deployment on an independent touch-up test.

The MoveIt Calibration project provides an RViz panel that wraps this flow (tutorial). Per its documentation, you create and measure a marker target, enter its dimensions, select the image and camera-info topics, set the sensor, object, end-effector and base frames, then take samples at different arm poses while rotating about at least two axes. The panel computes a result as samples arrive, defaults to the Daniilidis solver, and can export a launch file with a static transform publisher through a “Save camera pose” action. That is a convenient way to bootstrap, and the same logic is easy to reproduce in your own node if you need unattended recalibration.

Whatever tool you use, three ROS-level details cause most deployment bugs. Publish the result as a static transform between the correct parent and child, keeping the parent-child direction consistent with what the solver returned: eye-in-hand gives the camera relative to the gripper (or tool flange) frame, eye-to-hand gives the camera relative to the base. Make sure the camera optical frame, which follows the convention of z forward, x right, y down, is the frame you calibrated, not the mechanical camera_link frame, which is oriented differently. Version-control the calibration file alongside the cell configuration, and record the intrinsics, the board geometry, the solver and the date.

Verification that is independent of the solver

A low residual on the training stations tells you the fit is internally consistent, not that it is right. Run at least two independent checks.

Consistency check. Compute the board position in the base frame from each station, as in the code above, and report the standard deviation. A high scatter flags bad stations, bad intrinsics or timing skew. Remove outliers and re-solve, but keep a note of why they were removed.

Touch-up test. Place a different calibrated target, or a part with known features, at several positions that were not used for calibration. Have the vision system compute the target point, command the robot to touch it with a pointed tool or a dial indicator, and measure the miss distance with a ruler, feeler gauge or probe. Report the mean and worst-case error over, say, ten positions covering the operating volume. This is the number that predicts task success, and it is the only one that includes the whole chain: intrinsics, hand-eye transform, kinematics and your own tool-centre-point definition.

Hold-out test. Split the stations into a training half and a test half. Solve on one, then check the board-position scatter on the other. A large gap between training and hold-out scatter is the classic signature of over-fitting to a poorly distributed pose set.

Set an acceptance threshold up front from the task tolerance. If a gripper tolerates a 3 millimetre positioning error, a verified 1 millimetre worst case is a pass; a clean-looking solver output with an unknown worst case is not.

Trade-offs, Gotchas, and What Goes Wrong

Most failed calibrations trace back to a short list of causes, and a few of them look like solver problems when they are not.

Convention mismatches. The most frequent failure is a flipped direction: passing camera-to-target where target-to-camera was expected, or a robot pose that is tool-to-base while the code assumed base-to-tool. The symptom is a result that looks plausible in magnitude but places the camera in the wrong place, often mirrored. Sanity check with a ruler: the translation of X should match the physical camera offset to within a centimetre or so. A second check is unit consistency; many robot controllers report millimetres and degrees, while OpenCV and ROS expect metres and radians.

Euler angle ambiguity. Robot vendors report orientation as Euler angles with vendor-specific axis orders, or as rotation vectors. Converting incorrectly produces rotation errors that are consistent per pose but wrong, and the solver cannot detect it. Convert through a rotation matrix or quaternion and test the conversion with a known pose.

Poor intrinsics hidden by good-looking solves. If the distortion model is wrong at the image periphery, the board pose estimate is biased for oblique views, which are exactly the views that hand-eye calibration wants. Calibrate intrinsics with views covering the whole field of view, and re-run after any lens change or refocus.

Insufficient rotation diversity. Small or same-axis motions give a near-singular system. The solver still returns an answer, and its translation may be wildly off. Compare multiple solvers and subsets to expose this.

Flexible or loose mounts. A wrist camera on a cable-carrying bracket can shift when the cable pulls. A calibration that was valid at lunchtime is not valid after the cable snagged. Use a rigid mount, strain-relieve the cable, and add the touch-up test to a periodic check.

Treating the number as global. Because kinematic errors are position dependent, a hand-eye transform calibrated in one corner of the workspace degrades elsewhere. Solutions are to sample the whole operating volume, to calibrate the robot kinematics first if the application demands sub-millimetre absolute accuracy, or to use a local correction such as a visual servoing final approach that closes the loop on the image and so becomes insensitive to small calibration errors.

Thermal and payload changes. Joint warm-up and different tool payloads shift the arm’s effective geometry. For tight tolerances, calibrate in the same thermal state and with the same tool as production.

Over-trusting the residual. Hand-eye residuals can be small simply because the data underdetermine X. This is why the hold-out and touch-up checks above exist.

The trade-off between approaches is worth stating plainly. A one-time precise calibration is cheaper to run but ages; an automated recalibration routine costs integration effort but turns drift into a monitored quantity. For learned policies that take camera images and output actions, calibration error can be hidden by a policy that learned to compensate for a particular fixed offset, which breaks when the camera is replaced. Recording the calibration alongside the training data avoids that trap.

Practical Recommendations

Start by deciding the accuracy you need, then work backwards. If the gripper or suction cup tolerates several millimetres, a clean eye-in-hand calibration with a ChArUco board and a dozen well-spread stations will usually meet it. If you need sub-millimetre positioning across the workspace, plan for robot kinematic calibration, a thermally stable cell, and a closed-loop final approach.

Prefer eye-in-hand when the task benefits from close, high-resolution views and the arm can approach the part. Prefer eye-to-hand when you need a stable overview of the workcell and the arm does not block the line of sight. Use both when a cell needs coarse localisation followed by a close inspection.

Use the methods that are available in OpenCV as a cross-check on each other. Start with the Daniilidis or Park solver, run the others on the same data, and investigate any disagreement larger than your tolerance before blaming the algorithm.

A short checklist:

Calibrate intrinsics first, with locked focus, covering the full field of view, and store the result with the camera serial number.
Print the ChArUco board on a rigid substrate and measure the square size with calipers.
Collect at least twelve to fifteen stations with large roll, pitch and yaw variation, waiting for the arm to stop before capturing.
Solve with two or three methods and compare. Refine nonlinearly if you need the last millimetre.
Compute board-position scatter in the base frame and run a hold-out check.
Publish a static transform in the correct frame convention and commit it with its metadata.
Verify with a physical touch-up test at positions not used in calibration, and set a pass threshold from the task tolerance.
Schedule recalibration after any mount change, collision, lens refocus or significant temperature change.

Frequently Asked Questions

What is hand-eye calibration in robotics?

Hand-eye calibration determines the fixed rigid transform between a camera and a robot, so that points seen by the camera can be expressed in the robot’s frame. In an eye-in-hand setup it recovers the camera pose relative to the gripper; in an eye-to-hand setup it recovers the camera pose relative to the robot base. It is solved from several robot poses and matching views of a calibration target, usually through the AX=XB equation.

What is the difference between eye-in-hand and eye-to-hand?

In eye-in-hand the camera is mounted on the end effector and moves with the arm, which gives close, high-resolution views of the workpiece. In eye-to-hand the camera is fixed in the workcell and observes the arm and the workspace from outside, giving a stable wide view but sometimes suffering occlusion by the arm. The unknown transform and the mathematical setup differ slightly, but the same solvers handle both by choosing which robot poses to supply.

What does AX=XB mean?

AX=XB relates two measurements of the same physical motion between two stations. A is the relative motion of the gripper computed from robot kinematics, B is the relative motion of the camera computed from vision, and X is the unknown constant transform between the gripper and the camera. The equation splits into a rotation part and a translation part, and at least two motions with non-parallel rotation axes are required for a unique solution.

Which OpenCV hand-eye method should I use?

OpenCV’s calibrateHandEye offers Tsai, Park, Horaud, Andreff and Daniilidis, with Tsai as the default. No single method wins in all published comparisons because results depend on noise and pose distribution. A sensible practice is to run several on the same data: if they agree, the data are well conditioned, and if they differ widely, collect more diverse poses. The MoveIt Calibration package defaults to Daniilidis.

How many poses do I need for hand-eye calibration?

The mathematical minimum is two motions with non-parallel rotation axes, which means at least three stations. In practice you need many more to average noise. The MoveIt Calibration tutorial requires five samples before computing a result and says quality usually plateaus around twelve to fifteen. Diversity matters more than count: large rotations about at least two axes beat dozens of near-identical views.

How do I know if my hand-eye calibration is accurate?

Check it with data that were not used to compute it. Measure the scatter of the target position in the base frame across stations, perform a hold-out test, and then do a physical touch-up test at several new positions, recording the worst-case miss. Compare that figure with your task tolerance. A small solver residual alone does not prove accuracy, because poorly conditioned data can produce a small residual with a wrong transform.

Hand-Eye Calibration for Robot Arms: Eye-in-Hand vs Eye-to-Hand, AX=XB and Practical Accuracy (2026)