What Is Lidar-Camera Calibration?

Lidar-camera calibration establishes correspondences between 3-D lidar points and 2-D camera data to fuse the lidar and camera outputs together.

Lidar sensors and cameras are widely used together for 3-D scene reconstruction in applications such as autonomous driving, robotics, and navigation. While a lidar sensor captures the 3-D structural information of an environment, a camera captures the color, texture, and appearance information. The lidar sensor and camera each capture data with respect to their own coordinate system.

Lidar-camera校准包括转换the data from a lidar sensor and a camera into the same coordinate system. This enables you to fuse the data from both sensors and accurately identify objects in a scene. This figure shows the fused data.

Lidar and camera data fused together

Lidar-camera calibration consists of intrinsic calibration and extrinsic calibration.

Intrinsic calibration — Estimate the internal parameters of the lidar sensor and camera.
- Manufacturers calibrate the intrinsic parameters of their lidar sensors in advance.
- You can use theestimateCameraParametersfunction to estimate the intrinsic parameters of the camera, such as focal length, lens distortion, and skew. For more information, see the年代ingle Camera Calibrationexample.
- You can also interactively estimate camera parameters using theCamera Calibrator应用程序。
Extrinsic calibration — Estimate the external parameters of the lidar sensor and camera, such as location, orientation, to establish relative rotation and translation between the sensors.

Extrinsic Calibration of Lidar and Camera

The extrinsic calibration of a lidar sensor and camera estimates a rigid transformation between them that establishes a geometric relationship between their coordinate systems. This process uses standard calibration objects, such as planar boards with checkerboard patterns.

This diagram shows the extrinsic calibration process for a lidar sensor and camera using a checkerboard.

Lidar camera calibration process

The programmatic workflow for extrinsic calibration consists of these steps. Alternatively, you can use theLidar Camera Calibratorapp to interactively perform lidar-camera calibration.

Extract the 3-D information of the checkerboard from both the camera and lidar sensor.
1. To extract the 3-D checkerboard corners from the camera data, in world coordinates, use theestimateCheckerboardCorners3dfunction.
2. To extract the checkerboard plane from the lidar point cloud data, use thedetectRectangularPlanePointsfunction.
Use the checkerboard corners and planes to obtain the rigid transformation matrix, which consists of the rotationRand translationt. You can estimate the rigid transformation matrix by using theestimateLidarCameraTransformfunction. The function returns the transformation as arigid3dobject.

You can use the transformation matrix to:

Evaluate the accuracy of your calibration by calculating the error. You can do so either programmatically, usingestimateLidarCameraTransform、交互地使用Lidar Camera Calibrator应用程序。
Project lidar points onto an image by using theprojectLidarPointsOnImagefunction, as shown in this figure.
Fuse the lidar and camera outputs by using thefuseCameraToLidarfunction.
Estimate the 3-D bounding boxes in a point cloud based on the 2-D bounding boxes in the corresponding image. For more information, seeDetect Vehicles in Lidar Using Image Labels.

References

[1] Zhou, Lipu, Zimo Li, and Michael Kaess. “Automatic Extrinsic Calibration of a Camera and a 3D LiDAR Using Line and Plane Correspondences.” In2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 5562–69. Madrid: IEEE, 2018. https://doi.org/10.1109/IROS.2018.8593660.

年代ee Also