Projection Workflow
Projecting a 3D point onto a camera image involves three steps:World to camera coordinates
The 3D point is transformed from the LiDAR frame into the camera’s coordinate system using the extrinsic calibration (rotation and translation from TF messages or calibration data in the recording).
Camera to pixel coordinates
The transformed 3D point is projected onto the 2D image plane using the camera’s intrinsic model — either pinhole or double-sphere. This produces normalized image coordinates.
Pinhole Camera Model
The pinhole model represents a standard perspective camera. It is the most common model for automotive and robotics cameras with moderate fields of view (up to approximately 120 degrees).Parameters
| Parameter | Symbol | Description |
|---|---|---|
| Focal length X | fx | Horizontal focal length in pixels |
| Focal length Y | fy | Vertical focal length in pixels |
| Principal point X | cx | Horizontal optical center in pixels |
| Principal point Y | cy | Vertical optical center in pixels |
| Radial distortion | k1, k2, k3, k4 | Lens radial distortion coefficients |
| Tangential distortion | p1, p2 | Lens tangential distortion coefficients |
How It Works
For a 3D point(X, Y, Z) in camera coordinates:
- Normalize — Divide by depth:
x = X / Z,y = Y / Z - Map to pixels —
u = fx * x + cx,v = fy * y + cy
Lens distortion correction using the radial (
k1—k4) and tangential (p1, p2) coefficients is not yet applied in the pinhole projection path. The distortion parameters are stored and will be used in a future update. For most use cases with low-distortion lenses, the ideal pinhole projection (without distortion correction) provides accurate results.(u, v) is the pixel location on the image where the 3D point appears.
Points behind the camera (
Z <= 0) are rejected before projection. Points that project outside the image boundaries are also discarded.Double-Sphere Camera Model
The double-sphere model handles wide-angle and fisheye lenses with fields of view exceeding 180 degrees. It uses two additional parameters beyond the standard focal length and principal point to model the extreme distortion of fisheye optics.Parameters
| Parameter | Symbol | Description |
|---|---|---|
| Focal length X | fx | Horizontal focal length in pixels |
| Focal length Y | fy | Vertical focal length in pixels |
| Principal point X | cx | Horizontal optical center in pixels |
| Principal point Y | cy | Vertical optical center in pixels |
| Xi | xi | First sphere parameter (controls fisheye distortion strength) |
| Alpha | alpha | Second sphere parameter (balances the two spheres, range 0—1) |
How It Works
The double-sphere model projects points through two conceptual spheres, which naturally handles the extreme bending of light rays in fisheye lenses:- First sphere — Compute the distance from the origin to the 3D point:
d1 = sqrt(X^2 + Y^2 + Z^2). This maps the point onto a unit sphere. - Shift by xi — Offset along the optical axis:
xi * d1 + Z. Thexiparameter controls how strongly the fisheye effect warps the projection. - Second sphere — Compute a second radial distance incorporating the shifted coordinate. This second sphere adds another layer of distortion modeling.
- Blend with alpha — The
alphaparameter blends the contributions of the two spheres. Whenalpha = 0, the model reduces to a single sphere; whenalpha = 1, the second sphere dominates. - Map to pixels — The blended result is scaled by focal length and offset by the principal point, producing the final pixel coordinates
(u, v).
Multi-Camera Setup
Recordings often include multiple cameras covering different directions (front, left, right, rear). Avala handles multi-camera projection automatically.Calibration Data
Camera calibration is extracted from the recording’s transform (TF) messages:| Source | Data Provided |
|---|---|
tf2_msgs/TFMessage | Extrinsics: position and orientation of each camera relative to the LiDAR frame |
foxglove.FrameTransform | Extrinsics: same as above, Foxglove schema |
| Camera info messages | Intrinsics: focal length, principal point, distortion model and coefficients |
Camera Selection
When projecting a point onto an image (as in image projection visualization mode), the system must choose which camera to use. The selection process:- For each point, project it into every available camera’s image plane
- Discard cameras where the point falls outside the image bounds or depth range
- Among the remaining cameras, select the one where the projected pixel is closest to the image’s principal point (optical center)
- Sample the pixel color at the projected location
Supported Configurations
| Configuration | Cameras | Typical Use |
|---|---|---|
| Single front camera | 1 | Basic forward-facing setups |
| Stereo pair | 2 | Stereo vision with depth estimation |
| Surround view | 4—6 | Autonomous vehicles with full coverage |
| Surround + fisheye | 6—10 | Mixed pinhole and fisheye rigs |
Verifying Calibration Quality
The camera projection overlay is one of the most effective tools for checking calibration:- Enable Image Projection mode in the point cloud panel
- Open the camera image panels alongside the 3D view
- Check that point cloud silhouettes align with object edges in the images
- Misalignment indicates calibration errors — check extrinsics (rotation/translation) first, then intrinsics (focal length, distortion)
Related
- Visualization Modes — Image projection and other coloring modes for point clouds
- Image Panel — Camera image display with LiDAR overlay
- Point Cloud Panel — 3D viewer with projection overlay controls