Visual Servoing for Robots: IBVS vs PBVS, the Interaction Matrix and Hybrid 2.5D Control

A robot that plans a grasp from one camera frame and then executes it blind is only as accurate as its calibration chain: camera intrinsics, hand-eye transform, kinematic model, and joint encoders. Every link contributes error, and the errors add up at the gripper. Visual servoing removes most of that chain by closing the control loop on the image itself, so the robot keeps moving until the picture looks the way it should, regardless of whether the arm’s absolute pose estimate is a few millimetres off.

The idea is four decades old, yet it is having a second life in 2026. Cheap global-shutter cameras, GPU feature extractors and learned keypoint detectors make the sensing side easy, while force-controlled and compliant arms make the contact side forgiving. What remains hard is the control theory: choosing the feature vector, deriving the interaction matrix, avoiding the well-known failure modes, and keeping a loop with 50 to 100 ms of vision latency stable. You will leave this article with the derivation of the point-feature Jacobian, a clear rule for choosing between IBVS, PBVS and 2.5D schemes, a delay-stability bound verified numerically, and a runnable numpy simulation.

What this covers: eye-in-hand versus eye-to-hand configurations, error definitions, a full derivation of the image Jacobian, the control law, IBVS and PBVS pathologies, Malis 2.5D hybrid control, feature choices, robot-level mapping, latency and stability, ViSP and ROS 2 integration, learning-based servoing, and a worked simulation.

Context and Background

Visual servoing, also called vision-based control, uses image measurements directly inside a feedback loop. The canonical reference for the field is the two-part tutorial by Chaumette and Hutchinson: Part I, “Basic approaches” appeared in IEEE Robotics and Automation Magazine, vol. 13, no. 4, pp. 82-90, in December 2006, and Part II on advanced approaches followed in 2007. Part I formulates the problem, presents the two archetypal schemes, image-based and position-based, and discusses their stability and performance. Almost every taxonomy in this article traces back to that framing.

The first design decision is where the camera sits. In eye-in-hand configurations the camera is mounted on the end effector, so camera motion equals tool motion and the camera sees the target grow, shrink and rotate as the arm approaches. In eye-to-hand configurations the camera is fixed in the workspace and observes both the target and the end effector; the relevant Jacobian then contains the robot-to-camera transform, and the loop regulates the relative image of the tool and the object. Eye-in-hand gives high resolution at the point of contact and simple kinematics, but the target can leave the field of view. Eye-to-hand never loses the scene overview, but resolution at the gripper is poor and the arm may occlude the target. Most of the theory below is written for eye-in-hand and carries over with an additional fixed transform.

The second decision is what the error is made of. Define the task error as e(t) = s(t) – s, where s is a vector of features measured in the image or computed from it, and s is the value of those features at the desired pose. If s is built from image measurements such as pixel coordinates of points, the scheme is image-based visual servoing (IBVS). If s is built from a 3D pose estimate of the camera relative to the target, the scheme is position-based visual servoing (PBVS). Schemes that mix both are called hybrid, and the best known one is the 2.5D method of Malis, Chaumette and Boudet, published as “2 1/2 D Visual Servoing” in IEEE Transactions on Robotics and Automation, vol. 15, no. 2, pp. 238-250, April 1999.

Why does this matter for production systems now? Bin picking, charging-port insertion, aircraft assembly and lab automation all share a property: the target pose is known only in the image or only approximately in the world. A grasp planner such as the one covered in our 6-DoF grasp detection architecture outputs a target pose from a depth-camera snapshot, and the arm then needs a closed-loop terminal phase to absorb calibration drift, conveyor motion or slight object shifts. Visual servoing is that terminal phase. It also composes naturally with the joint-level machinery discussed in our posts on damped least squares inverse kinematics and impedance and admittance control, since the servo loop outputs a Cartesian velocity that lower layers must track compliantly.

Open-source tooling is mature. The ViSP library from Inria’s Lagadic lineage is a cross-platform toolkit that, per its repository, supports visual tracking and visual servoing and provides control-law computation, feature tracking and simulation tools; it is licensed GPL-2.0. We return to ViSP and its ROS packaging near the end.

The Reference Architecture: Features, Interaction Matrix and the Control Law

A visual servoing loop reduces to three pieces: a feature vector s measured every frame, a matrix L_s that tells you how s changes when the camera moves, and a velocity law that inverts that relationship to drive the error to zero. Concretely, the camera twist v = (vx, vy, vz, wx, wy, wz) is chosen as v = -lambda L_s^+ e, where L_s^+ is the Moore-Penrose pseudo-inverse and lambda is a positive gain, giving an exponential decrease of the error when the model is accurate.

Figure 1: The eye-in-hand visual servoing loop. Features s are compared with s, the interaction matrix L is estimated from the features and depth, the control law produces a camera twist, and the robot Jacobian turns it into joint velocities.*

The diagram shows two parallel flows. The measurement path goes camera, feature extraction, error. The model path feeds the same features, plus depth values, into the interaction matrix estimate, which is the only place where scene geometry enters the loop. The control law multiplies them, and the result is mapped to joint velocities and executed by the robot’s low-level velocity controller, which moves the camera and closes the loop physically.

Deriving the interaction matrix for a point

The interaction matrix, also called the image Jacobian (not to be confused with the robot’s geometric Jacobian), is defined by the relation s_dot = L_s v, where v is the camera’s spatial velocity expressed in the camera frame. For a point feature it can be derived in a few lines, and the derivation exposes why depth appears in the matrix.

Take a 3D point P = (X, Y, Z) expressed in the camera frame, and normalized image coordinates x = X/Z, y = Y/Z, which equal pixel coordinates after removing intrinsics (x = (u – u0)/fx, y = (v – v0)/fy). If the camera moves with translational velocity (vx, vy, vz) and angular velocity (wx, wy, wz), a static point seen from the camera moves as P_dot = -v_t – w x P. Written component by component:

X_dot = -vx – wy Z + wz Y
Y_dot = -vy – wz X + wx Z
Z_dot = -vz – wx Y + wy X

Differentiating x = X/Z gives x_dot = X_dot/Z – X Z_dot/Z^2. Substituting the three equations and replacing X/Z by x and Y/Z by y yields:

x_dot = -vx/Z + x vz/Z + x y wx – (1 + x^2) wy + y wz

and, by the same steps for y,

y_dot = -vy/Z + y vz/Z + (1 + y^2) wx – x y wy – x wz.

Stacking the two rows, the 2×6 interaction matrix of one point is

L_point = [ -1/Z, 0, x/Z, x y, -(1 + x^2), y ; 0, -1/Z, y/Z, 1 + y^2, -x y, -x ].

Three facts follow immediately. First, only the three translational columns depend on the unknown depth Z, so the rotational part of the loop is depth-free. Second, one point gives two equations, so at least three points are needed to constrain six degrees of freedom, and four are used in practice to remove ambiguities. Third, the matrix is a local linearization: it is exact for the instantaneous velocity, but the mapping between image-plane motion and camera motion is strongly nonlinear over large displacements, which is the root of most IBVS pathologies.

Stacking k points gives a 2k x 6 matrix. With the classical four-point choice the matrix is 8 x 6, which is over-determined; the pseudo-inverse then yields the least-squares camera velocity that reduces the error as much as possible in the L2 sense.

The control law and the three ways of choosing L

Requiring an exponential decay e_dot = -lambda e together with e_dot = L_s v gives v = -lambda L_s^+ e. In practice the true matrix is unavailable, so a model or estimate L_hat^+ is used, and the closed loop becomes e_dot = -lambda L_s L_hat^+ e. Chaumette and Hutchinson discuss three standard choices, and all three are cheap to implement:

Current matrix. Evaluate L at the current features, using a current depth estimate for each point. This tends to give nearly straight image trajectories but needs Z at every iteration.
Desired matrix. Evaluate L once at s with the desired depths Z. This is constant, needs no online depth, and is robust, but convergence is slower for large errors.
Mean matrix. Use L = 0.5 (L_current + L_desired), a compromise that often has the best behaviour for large displacements, at the cost of still needing Z.

Stability for the perfect-model case is easy to state. Taking the Lyapunov candidate V = 0.5 ||e||^2 gives V_dot = -lambda e^T L_s L_hat^+ e. If the product L_s L_hat^+ is positive definite, V decreases, and the loop is globally asymptotically stable. But with k >= 4 points the product is a 2k x 2k matrix of rank at most 6, so it is only positive semi-definite and the argument fails. The practical result is local asymptotic stability near s*, a statement that holds when L_hat^+ L_s is positive definite on the six-dimensional space. Large displacements, bad depth estimates and nonlinear camera motion all fall outside this guarantee.

From camera twist to robot joints

The control law outputs the camera twist. For an eye-in-hand arm the robot Jacobian J(q) maps joint rates to the end-effector twist, and a fixed twist transform ^cV_e built from the hand-eye calibration (rotation ^cR_e and translation ^ct_e) maps end-effector twist to camera twist. The joint command is q_dot = (^cV_e J(q))^+ v. When the arm is near a kinematic singularity, the pseudo-inverse produces large joint rates; the damped least-squares remedies from our inverse kinematics article apply unchanged. For eye-to-hand setups the matrix is augmented by the fixed transform between the camera and the robot base, and the sign of the velocity mapping flips because the camera no longer moves with the tool.

Hand-eye calibration error is not a source of steady-state error here, because the loop terminates when the image error vanishes. It does act as a gain error and a rotation of the commanded velocity; the closed-loop matrix is then a perturbation of the identity, and the loop remains stable as long as that perturbation keeps L_s L_hat^+ positive definite. In plain language, a calibration error of tens of degrees can destabilize convergence, but a few degrees merely slow it down or bend the path.

IBVS vs PBVS vs 2.5D: Three Ways to Define the Error

Direct answer: in IBVS the features are image coordinates, so the image trajectory is near-straight but the camera path in 3D can be erratic and may retreat; in PBVS the features are a reconstructed 3D camera pose, so the Cartesian path is a straight line but the image path is uncontrolled and the target can leave the view. The 2.5D scheme of Malis, Chaumette and Boudet mixes image coordinates with partial pose to decouple translation and rotation.

Figure 2: The three scheme families branch from the same image. Each yields different trajectory properties and a different dependency: depth for IBVS, calibrated model for PBVS, homography decomposition for 2.5D.

IBVS: straight in the image, unpredictable in space

With IBVS the exponential decay e_dot = -lambda e means each feature coordinate converges along a straight line in the image, provided the model is good. That is attractive because it keeps the target in the field of view if both the initial and desired images contain it, since the convex hull of the points is roughly preserved. The costs are a coupled, nonlinear 3D motion and a set of well-documented failure modes.

Camera retreat. Consider a pure rotation of the camera about its optical axis by an angle approaching 180 degrees. The straight-line image paths of the points pass through the image centre, which the controller interprets as a request to shrink the pattern, and so it commands a large translation along the optical axis, away from the target. In the limit of exactly 180 degrees the commanded motion is a pure backward translation and the loop never terminates. The classical illustration, due to Chaumette, is exactly this rotation. The simulation later in the article reproduces it numerically, and the numbers are instructive.

Local minima and the rank problem. Because 2k > 6, there are error vectors e != 0 for which L_hat^+ e = 0, meaning the commanded velocity is zero while the error is non-zero. These are local minima, and they exist for the standard four-point features. Chaumette’s 1998 analysis of potential stability and convergence problems in image-based and position-based servoing, which I am citing from memory and have not re-verified this run, characterized configurations of this type. Reaching a local minimum looks like the robot stopping well short of the target while the image error stays stubbornly constant.

Singularities of L. For three points, the interaction matrix is singular at specific configurations, for example when the camera optical centre lies on a particular cylinder through the three points (a result associated with Michel and Rives, again cited from memory). With four coplanar points the matrix has rank six in general, but its conditioning can be poor, as the simulation below shows.

Depth dependence. The translation block needs Z. With a depth camera or a known model you can measure it; with a monocular camera you either use Z everywhere, estimate it by an observer on the camera motion, or recover it from a homography. Using Z everywhere is surprisingly robust near convergence and poor far from it.

In PBVS the feature vector is a pose. The usual choice is s = (^{c}t_c, theta u), the translation of the current camera frame expressed in the desired frame plus the axis-angle rotation between them, with s = 0. The interaction matrix is block diagonal: L_s = [ ^{c}R_c, 0 ; 0, L_theta-u ], with L_theta-u = I – (theta/2)[u]x + (1 – sinc(theta)/sinc^2(theta/2)) [u]_x^2, where [u]_x is the skew-symmetric matrix of the axis. This matrix is invertible for rotations below 180 degrees, and L_theta-u^{-1} theta u = theta u exactly, so the control law reduces to the very clean v_t = -lambda ^cR{c} ^{c*}t_c, w = -lambda theta u.

The result is a decoupled loop. The camera origin travels along a straight line to the goal and the orientation follows the geodesic on SO(3). The controller is globally stable if the pose estimate is perfect, and it has none of the retreat pathologies. The weakness is that nothing in the loop looks at the image. Pose estimation from points (PnP) or from a model-based tracker is sensitive to calibration, to the target model, and to image noise, and a pose bias becomes a steady-state positioning error in the world because the loop converges on the estimated, not the true, relative pose. In addition, a straight 3D path does not guarantee that the target stays visible: a camera sweeping sideways toward a close target can easily lose some of the features near the image border.

2.5D: using what each scheme does well

Malis, Chaumette and Boudet’s insight was to estimate only the part of the pose that is cheap and robust to obtain, and to use image data for the rest. Per their abstract, the method estimates a partial camera displacement from the current to the desired pose at each iteration, using visual features extracted from that displacement to build a decoupled control law for all six degrees of freedom. It does not need a 3D model of the object, and the authors report a larger stability and convergence domain than classical IBVS or PBVS, with necessary and sufficient conditions for local stability under calibration errors and sufficient conditions for global stability.

In the standard formulation, the rotation ^{c}R_c and the scaled translation are obtained by decomposing the homography between the current and the reference image of a planar target (or by the epipolar geometry for a non-planar one). The feature vector is then s = (x, y, log rho, theta u). Here (x, y) are the normalized image coordinates of a chosen reference point, rho = Z/Z is the depth ratio from the homography, and theta u is the rotation as in PBVS. Since the rotation part is controlled exactly as in PBVS, the interaction matrix is block-triangular. The translation block, as I recall it from the paper (check the equations before reuse), takes the form -(1/(Z* rho)) times a simple matrix involving x and y and has a nonzero determinant for any Z > 0, so it cannot become singular within the working range. The reference point’s image trajectory is a straight line, so it stays in view if the point is chosen near the centre of the target, while the rotation follows the geodesic.

Two practical caveats apply. First, the homography must be decomposed on every frame, which adds noise: a planar decomposition has two solutions and needs a consistency rule, and a near-fronto-parallel target with a small baseline gives an ill-conditioned translation. Second, only one image point is controlled directly, so other points may still leave the view when the rotation is large.

Decision matrix

Criterion	IBVS	PBVS	2.5D
Needs 3D target model	No	Yes (or pose estimator)	No (planar homography suffices)
Needs camera intrinsics	Weakly (features can be in pixels)	Strongly	Moderately
Image trajectory	Straight, field of view kept	Uncontrolled	Reference point straight
3D trajectory	Coupled, may retreat	Straight line plus geodesic	Decoupled translation and rotation
Large rotation (>120 degrees)	Poor, retreat	Good	Good
Sensitivity to depth error	Moderate (gain error)	None, but pose error shifts goal	Moderate
Typical risk	Local minimum, singularity	Features lost, bias in world	Homography noise

The table is qualitative and follows the arguments in the cited papers rather than a head-to-head benchmark; there is no single dataset on which all three win in all conditions.

Worked Example: A Four-Point IBVS Simulation in NumPy

The simulation below places a 0.2 m square target on the plane Z = 0, starts the camera 1 m above it with an offset and a 30-degree roll, and runs IBVS with each of the three interaction-matrix choices. Camera model: pinhole with normalized coordinates, so the intrinsics drop out. The update integrates the commanded twist in the camera frame using the exponential map for rotation. The script is the exact code I ran; its output is reported after it.

import numpy as np

def skew(w):
    return np.array([[0,-w[2],w[1]],[w[2],0,-w[0]],[-w[1],w[0],0]])

def expm_so3(w):
    th = np.linalg.norm(w)
    if th < 1e-12:
        return np.eye(3)
    K = skew(w / th)
    return np.eye(3) + np.sin(th) * K + (1 - np.cos(th)) * K @ K

def project(P_w, R, t):
    """Camera pose (R, t): camera frame -> world. Returns normalized coords and depths."""
    Pc = (R.T @ (P_w - t).T).T
    return Pc[:, :2] / Pc[:, 2:3], Pc[:, 2]

def interaction_matrix(xy, Z):
    rows = []
    for (x, y), z in zip(xy, Z):
        rows.append([-1/z, 0, x/z, x*y, -(1+x*x), y])
        rows.append([0, -1/z, y/z, 1+y*y, -x*y, -x])
    return np.array(rows)

def ibvs(R0, t0, P_w, s_star, Z_star, lam=0.5, dt=1.0, iters=300, mode="current"):
    R, t = R0.copy(), t0.copy()
    hist = []
    for k in range(iters):
        xy, Z = project(P_w, R, t)
        e = (xy - s_star).ravel()
        hist.append((np.linalg.norm(e), Z.min(), t.copy()))
        if np.linalg.norm(e) < 1e-8:
            break
        if mode == "current":
            L = interaction_matrix(xy, Z)
        elif mode == "desired":
            L = interaction_matrix(s_star, Z_star)
        else:
            L = 0.5*(interaction_matrix(xy, Z) + interaction_matrix(s_star, Z_star))
        v = -lam * np.linalg.pinv(L) @ e
        R = R @ expm_so3(v[3:] * dt)
        t = t + R @ (v[:3] * dt)
    return hist, R, t

a = 0.1
P_w = np.array([[-a,-a,0],[a,-a,0],[a,a,0],[-a,a,0]], float)
R_star = np.diag([1.0, -1.0, -1.0]); t_star = np.array([0, 0, 1.0])
s_star, Z_star = project(P_w, R_star, t_star)
R0 = R_star @ expm_so3(np.array([0, 0, np.deg2rad(30)]))
t0 = t_star + R_star @ np.array([0.15, -0.10, -0.25])
for mode in ["current", "desired", "mean"]:
    hist, R, t = ibvs(R0, t0, P_w, s_star, Z_star, mode=mode)
    print(mode, "iters", len(hist), "final err %.2e" % hist[-1][0],
          "pos err %.2e" % np.linalg.norm(t - t_star))

Running it produced convergence below 1e-8 in feature error in 26 iterations for the current matrix and 27 for both the desired and mean matrices, with final position errors on the order of 1e-8 m or smaller. These counts depend on the gain (lambda = 0.5 per iteration with dt = 1, so the error roughly halves per step in the ideal case), not on wall-clock time; they validate the derivation of L but say nothing about real hardware. The translation of the desired camera frame is found at (0, 0, 1) with the target square seen at normalized coordinates of plus or minus 0.1 at the desired pose; at a focal length of 600 pixels that corresponds to 60 pixels from the centre.

What the simulation reveals about conditioning

Evaluating the singular values of L at the desired pose gives 2.843 (twice), 0.283 (twice) and 0.014 (twice), a condition number of about 202. The weakest directions are not an accident. In the matrix, the vx column is roughly (-1, 0, -1, 0, …) and the wy column is roughly (-1.01, 0, -1.01, …) because -(1 + x^2) is close to -1 when x = 0.1. A sideways translation and a small rotation about the vertical axis therefore move the four points almost identically; the controller can tell them apart only through the 0.01 difference and through the depth-dependent terms. The same holds for vy and wx. Making the target larger in the image helps sharply: with a 0.6 m square at the same distance (x = 0.3) the condition number drops to about 24. This is the numerical form of a familiar rule: a small, distant target makes translation-rotation ambiguity a noise amplifier, and in practice you want the target to occupy a good part of the field of view at the final pose.

Reproducing camera retreat

Rotating the initial pose about the optical axis, with the camera otherwise at the desired position, gives a sweep that I ran with the same code. For 90, 110, 120 and 130 degrees, the loop converged in 26 to 27 iterations while the camera drifted away from the target by 0.71, 1.34, 1.76 and 2.25 m respectively along its optical axis before returning. At 140 degrees it still converged, but needed 44 iterations and the camera retreated by about 17 m. At 150 degrees the run failed to converge in 400 iterations, with a stuck feature error of about 0.28 and a camera that had left the scene, and at 160 degrees the excursion exceeded any physical workspace by many orders of magnitude. A real robot would hit joint limits or its workspace boundary long before. This is the retreat problem in numbers, and it appears in the rotation range where PBVS and 2.5D have no difficulty. It also shows why “converges in simulation” is not a sufficient acceptance test: the path matters.

Latency, Delay and Stability in the Real Loop

Everything above assumed that the feature measured now is the feature that exists now. In a real system it is not. The camera exposes for a few milliseconds, the sensor reads out, the frame crosses USB or GigE or a ROS transport, the detector or tracker runs, and only then does the controller see s. Add the command path through the robot driver and the low-level velocity loop, and the loop easily carries two to four sampling periods of delay. The exact figures are system-specific and you should measure them with a timestamped LED or checkerboard test rather than trusting datasheets.

Figure 3: Timing of one servo cycle. The camera exposes frame k, vision processing and transport add delay, the servo law computes a velocity from stale features, and the robot has already moved by the time the command is applied.

A delay bound you can check

Model each feature direction as a scalar integrator, e_{k+1} = e_k – lambda_d e_{k-n}, where lambda_d = lambda Delta-t is the dimensionless per-sample gain and n is the number of extra samples of delay. The characteristic equation z^{n+1} – z^n + lambda_d = 0 has all roots inside the unit circle exactly when lambda_d < 2 sin(pi / (2(2n + 1))). This closed form is a standard result for the delayed discrete integrator; I checked it numerically rather than relying on memory. Simulating the recursion for four delays and gains at 90% and 110% of the bound gave the following:

Extra delay n (samples)	Bound on lambda_d	Gain at 90% of bound	Gain at 110% of bound
0	2.000	decays to ~1e-39	diverges (~5e31)
1	1.000	decays to ~8e-11	diverges (~2e8)
2	0.618	decays to ~3e-6	diverges (~4e4)
3	0.445	decays to ~1e-5	diverges (~3e3)

The key design consequence is that the permitted gain falls fast as delay grows. At a 30 Hz camera (Delta-t of about 33 ms) with two samples of extra delay, the bound lambda_d < 0.618 corresponds to a continuous gain below roughly 18.5 per second, an error time constant no shorter than about 54 ms, and in practice you would stay well below that to have margin. Lowering lambda makes the loop slower but safer; raising the camera frame rate, trimming processing latency and using the exposure timestamp in the controller all buy back bandwidth.

Compensating delay

If delay cannot be reduced, there are three classical remedies. A Smith predictor feeds the controller a prediction of the undelayed feature by running a model of the plant forward by the delay and correcting it with the measured, delayed output; because the plant here is close to an integrator with known gain, the model is simple, but its accuracy depends on L_hat and on the delay estimate, and a wrong delay estimate can make it worse than no predictor. A Kalman filter or velocity feed-forward estimates target motion and the robot’s own motion, then extrapolates the feature to the current time. And timestamp alignment uses the joint states recorded at exposure time rather than at processing time, which matters most for eye-to-hand systems and moving targets. For a moving target, the plain law leaves a steady tracking error proportional to target velocity divided by lambda; adding an estimate of the target feature velocity as a feed-forward term removes it.

Stability under model errors, restated

The delay analysis and the model-error analysis combine in a simple rule of thumb. The product L L_hat^+ perturbs the effective gain in each direction, so the worst-case direction sets the gain bound. Because the condition number of L at the goal can be in the hundreds (about 202 in the small-target example above), the pseudo-inverse amplifies noise in weak directions: a one-pixel-class error in the weakly observable combination of vx and wy can produce large commanded velocities. Regularization helps. Damped pseudo-inverses, truncated singular value decomposition of L, and a velocity saturation on the commanded twist are all inexpensive and trade a little convergence speed for noise robustness.

Choosing Features Beyond Points

Image points are the default because they are easy to detect and their interaction matrix is simple. They are not the only option, and the choice of s shapes both the conditioning and the basin of convergence.

Lines, circles and cylinders. Geometric primitives have known interaction matrices and are tracked robustly in structured environments such as factories, where edges are plentiful. They use more of the image than a handful of points and average out noise.

Image moments. Treating the region as a blob and using combinations of its moments, as proposed by Tahri and Chaumette (their work on point-based and region-based moments for planar objects, which I am citing from memory), lets you design features that are decoupled and have good linearization. The cost is a more elaborate derivation, and the segmentation must be clean.

Photometric and direct methods. Collewet and Marchand’s photometric visual servoing takes the whole image intensity as the feature vector, avoiding any feature extraction at all, at the price of a very nonlinear cost, a narrow convergence basin and sensitivity to lighting changes. A related family uses mutual information or the Fourier phase. These are good for fine terminal alignment, not for large motions, and they are best seen as a final stage after a coarse feature-based phase.

Event cameras and DVS. Dynamic vision sensors output asynchronous brightness changes with microsecond-scale timestamps, which shrinks sensing latency dramatically and removes motion blur. Published work has demonstrated servoing and tracking with event streams, but the interaction relations are built on events rather than frames, and tooling is less mature. I flag this as a research direction, not a drop-in replacement for a global-shutter camera.

Learned features. A detector network that outputs keypoints, object centres or embeddings can supply s. The risk is that the interaction matrix for such a feature is not known analytically. Two patterns are common: keep the keypoints geometric (the network detects a point, the classical L applies), or learn L or the whole policy end-to-end, as discussed below.

Mixing and partitioning

Because the six degrees of freedom do not need to be treated identically, partitioned schemes assign different features to different subsets. The classic example separates the optical-axis motions, vz and wz, from the other four and drives them with dedicated features such as the square root of the target’s image area and the orientation of a line between two points. This directly cures the retreat problem, because the translation along z is controlled by the apparent size rather than by the point positions. Switching schemes go further: start with PBVS or 2.5D to cover large displacements, then hand over to IBVS near the goal for precision. The switch must be designed with hysteresis so that the loop does not chatter between schemes when the estimates disagree.

Learning-Based Servoing

Learning enters visual servoing at three levels, and it is useful to keep them separate. At the perception level, a network replaces the hand-crafted detector, which is the least controversial use: it gives robust points on textureless parts or under occlusion while the control law remains the analytic one above. At the model level, a network estimates depth, the interaction matrix, or the full image Jacobian online, which loosens the dependence on calibration and the target model. At the policy level, a network maps images directly to velocities, trained by imitation or reinforcement learning, as in the many end-to-end grasping systems of recent years.

The trade-off is classical. Analytic servoing gives you a proof of local stability, a known convergence rate and diagnosable failures. A learned policy can handle clutter, partial views and novel objects, but offers no such guarantee and is data-hungry; its failure modes are distribution-shift failures that are hard to anticipate. A pragmatic hybrid is the layered design used in modern manipulation stacks: a learned grasp or keypoint model supplies the target and features (see the grasp detection architecture post), and an analytic IBVS or 2.5D law handles the last few centimetres where precision matters and the safety envelope can be reasoned about. I am not aware of a verified benchmark that compares these approaches across tasks under identical conditions, so treat claims of one family dominating another as task-specific.

ViSP and ROS 2 Integration

ViSP is the reference implementation for the techniques in this article. Its repository describes a cross-platform toolkit supporting Linux, Windows, macOS, iOS and Android, with visual tracking, visual features, control-law computation and simulation tools, under GPL-2.0. The Debian packaging path I found referenced ViSP 3.7.0, but I could not confirm the current release number from the repository page, so check the project’s release notes before pinning a version. ViSP provides classes for the point, line, moment and pose features discussed above, and the servo task class assembles features, desired values and an interaction matrix choice (current, desired or mean) into the control law, so the three variants in the simulation correspond to options in the library.

On the ROS side, the situation needs care. The ROS index page for the visp_ros package lists version 1.0.0 under GPLv2, marked unmaintained with a last-updated date of January 2022, built with catkin and compatible with the ROS 1 distributions Kinetic, Melodic and Noetic; the index showed no released version for ROS 2 distributions such as Humble or Jazzy. The GitHub repository, however, uses a default branch named rolling, and its installation instructions mention colcon and the Humble and Rolling distributions, so ROS 2 support appears to exist in source form. I could not verify a released binary for ROS 2 or its completeness, so for a production deployment you should build from source, test it on your robot model, and be prepared to maintain a fork.

A pragmatic ROS 2 architecture does not depend on any single package. A camera driver publishes images with exposure timestamps; a vision node publishes feature messages with the same stamp; a servo node subscribes, computes the twist with ViSP or your own numpy or Eigen code, and publishes a geometry_msgs/TwistStamped; and a Cartesian velocity controller or a servoing layer such as MoveIt Servo converts the twist to joint commands with limits and singularity handling. Keep the servo node in a real-time-friendly executor, avoid copying large images between processes (use composable nodes with intra-process transport), and log the end-to-end latency per cycle. The compliance machinery in the impedance and admittance control article sits one level below: it ensures that when the final approach makes contact, the arm yields rather than fights the vision loop.

Trade-offs, Gotchas, and What Goes Wrong

The first trap is validating only on converge-or-not. A servo that converges after the camera has retreated 17 m, or after joint 3 hit its limit, has failed. Always log the camera path, the minimum feature margin to the image border, the joint-limit margin and the manipulability, and fail a test if any crosses a threshold.

The second is field-of-view loss. Classical laws do not include visibility constraints. If a feature leaves the image the interaction matrix changes size or the detector fails, and the error jumps. Remedies include potential-field terms that push features away from the border, task-priority schemes where visibility is a high-priority secondary task, and model predictive control that includes constraints at the cost of compute.

The third is depth. IBVS with Z* everywhere is a good default near the goal, but a gross depth error for a large rotation can flip the sign of the effective gain in some directions. Estimate depth when you can, and cap the commanded velocity.

The fourth is scale and conditioning, as the 202-versus-24 condition-number comparison showed. A target that is too small at the final pose makes the last millimetres slow and noisy. Choose a bigger fiducial, a larger camera focal length, or a closer final stand-off.

The fifth is the pose-estimation bias in PBVS, which is invisible in the loop: the controller reports zero error while the tool sits several millimetres off, because the intrinsics or the target model are wrong. Validate PBVS accuracy against an external reference such as a touch probe or a laser tracker.

The sixth is moving targets and rolling-shutter distortion, which add a feature-dependent time offset within a single frame. Use a global shutter or model the readout time.

Figure 4: A decision flow for scheme selection. Trust in the model and calibration, together with the size of the initial rotation, selects the scheme; safety limits, a delay-aware gain and a ROS 2 node complete the design.

The final trap is regulatory and organisational rather than mathematical. A vision-guided arm that works near people must still respect the speed and separation limits of the applicable safety standards for collaborative robots; a visual servo does not replace a safety-rated monitored stop. Treat the vision loop as a performance feature and the safety function as a separate, independently verified layer.

Practical Recommendations

Start from the failure you can least afford. If the initial pose can differ from the goal by a large rotation, and your model and calibration are trustworthy, use PBVS or 2.5D. If the object model is unavailable or calibration is poor, use IBVS, with the mean interaction matrix, and a partitioned or switching extension if rotations may exceed roughly 90 degrees. In my simulation the retreat grew sharply beyond 120 degrees, so treat that as an order-of-magnitude warning for the four-point case rather than a universal threshold.

Treat latency as a design parameter from day one. Measure it, then set the gain with the delay bound as a ceiling. Saturate the twist, apply damped inversion, and watch the condition number of L at the goal as a design metric. Pick a final stand-off and target size that keeps it below about 30 if you can.

Finally, layer. Let a learned or geometric planner give you the coarse target, let a servo loop do the closed-loop terminal approach, and let a compliant joint-level controller absorb contact. A short checklist:

Choose the scheme from the initial rotation range and calibration trust, not from habit.
Measure end-to-end latency; set lambda below the delay bound with margin.
Use the mean or current interaction matrix with a depth estimate; fall back to Z* near the goal.
Add twist saturation, damped least squares and joint-limit avoidance.
Add visibility constraints, or at least detection of lost features with a safe stop.
Log path, margins and conditioning; accept tests on path quality, not only on final error.
Validate absolute accuracy against an external reference when using PBVS.
Keep the safety function independent of the vision loop.

Frequently Asked Questions

What is visual servoing in robotics?

Visual servoing is a control method that uses camera measurements directly in the feedback loop to move a robot until image features, or a pose derived from them, match a desired configuration. The error e = s – s* is driven to zero by commanding a camera twist computed from the interaction matrix. Because the loop is closed on the image rather than on joint encoders alone, it tolerates calibration and positioning errors that would defeat open-loop, look-then-move execution.

What is the difference between IBVS and PBVS?

In IBVS the features are measured in the image, such as point coordinates, and the control law uses the image Jacobian, producing near-straight image paths but coupled 3D motion that can retreat or stall. In PBVS the features are the camera pose estimated relative to the target, giving straight 3D paths but no image control, and a dependence on accurate calibration and a target model. Hybrids such as 2.5D combine them.

What is the interaction matrix or image Jacobian?

The interaction matrix L_s relates the rate of change of a feature vector to the camera’s spatial velocity: s_dot = L_s v. For a point with normalized image coordinates (x, y) and depth Z it is a 2 by 6 matrix whose translational columns scale with 1/Z and whose rotational columns depend only on x and y. It is distinct from the robot Jacobian, which maps joint rates to end-effector velocity.

What is 2.5D visual servoing?

2.5D visual servoing, introduced by Malis, Chaumette and Boudet in 1999, estimates part of the camera displacement between the current and desired views, usually through a homography, and combines it with image coordinates of a reference point. The result is a decoupled law: the rotation follows a geodesic and the translation is controlled through the image point and the depth ratio, avoiding the retreat of IBVS without needing a 3D model.

Why does IBVS cause camera retreat?

Image-plane straight-line paths are a poor model of the camera motion when the required rotation about the optical axis is large. For a rotation near 180 degrees, the straight paths of the points pass through the image centre and the controller interprets this as the pattern needing to shrink, so it commands translation along the optical axis away from the target. In my four-point simulation the camera excursion grew from 0.7 m at 90 degrees to about 17 m at 140 degrees.

How does time delay affect visual servoing stability?

Delay reduces the maximum stable gain. For a scalar integrator loop with n extra samples of delay, the discrete gain must satisfy lambda_d < 2 sin(pi / (2(2n + 1))), which is 2 for no delay, 1 for one sample, about 0.618 for two and about 0.445 for three. Faster cameras, lower processing latency, timestamp alignment, Smith predictors and feed-forward all improve the margin.

Visual Servoing for Robots: IBVS vs PBVS, the Interaction Matrix and Hybrid 2.5D Control (2026)