## Fisheye Calibration Basics

Camera calibration is the process of computing the extrinsic and intrinsic parameters of a camera. Once you calibrate a camera, you can use the image information to recover 3-D information from 2-D images. You can also undistort images taken with a fisheye camera. The Computer Vision Toolbox™ contains calibration algorithms for the pinhole camera model and the fisheye camera model. You can use the fisheye model with cameras up to a field of view (FOV) of 195 degrees.

Fisheye cameras are used in odometry and to solve the simultaneous localization and mapping (SLAM) problems visually. Other applications include, surveillance systems, GoPro, virtual reality (VR) to capture 360 degree field of view (fov), and stitching algorithms. These cameras use a complex series of lenses to enlarge the camera's field of view, enabling it to capture wide panoramic or hemispherical images. However, the lenses achieve this extremely wide angle view by distorting the lines of perspective in the images

Because of the extreme distortion a fisheye lens produces, the pinhole model cannot model a fisheye camera.

### Fisheye Camera Model

The Computer Vision Toolbox calibration algorithm uses the fisheye camera model proposed by Scaramuzza [1]. The model uses an omnidirectional camera model. The process treats the imaging system as a compact system. In order to relate a 3-D world point on to a 2-D image, you must obtain the camera extrinsic and intrinsic parameters. World points are transformed to camera coordinates using the extrinsic parameters. The camera coordinates are mapped into the image plane using the intrinsic parameters.

#### Extrinsic Parameters

The extrinsic parameters consist of a rotation, R, and a translation, t. The origin of the camera's coordinate system is at its optical center and its x- and y-axis define the image plane.

The transformation from world points to camera points is:

#### Intrinsic Parameters

For the fisheye camera model, the intrinsic parameters include the polynomial mapping coefficients of the projection function. The alignment coefficients are related to sensor alignment and the transformation from the sensor plane to a pixel location in the camera image plane.

The following equation maps an image point into its corresponding 3-D vector.

• are the ideal image projections of the real-world points.

• represents a scalar factor.

• are polynomial coefficients described by the Scaramuzza model, where .

• is a function of (u,v) and depends only on the distance of a point from the image center: .

The intrinsic parameters also account for stretching and distortion. The stretch matrix compensates for the sensor-to-lens misalignment, and the distortion vector adjusts the (0,0) location of the image plane.

The following equation relates the real distorted coordinates (u'',v'') to the ideal distorted coordinates (u,v).

### Fisheye Camera Calibration in MATLAB

To remove lens distortion from a fisheye image, you can detect a checkerboard calibration pattern and then calibrate the camera. You can find the checkerboard points using the `detectCheckerboardPoints` and `generateCheckerboardPoints` functions. The `estimateFisheyeParameters` function uses the detected points and returns the `fisheyeParameters` object that contains the intrinsic and extrinsic parameters of a fisheye camera. You can use the `fisheyeCalibrationErrors` object to check the accuracy of the calibration.

#### Correct Fisheye Image for Lens Distortion

Remove lens distortion from a fisheye image by detecting a checkboard calibration pattern and calibrating the camera. Then, display the results.

Gather a set of checkerboard calibration images.

`images = imageDatastore('calibrationImages');`

Detect the calibration pattern from the images. The 'PartialDetections' Name-Value argument is set to true by default allowing detection of partial checkerboards.

`[imagePoints,boardSize] = detectCheckerboardPoints(images.Files, 'HighDistortion', true);`

Generate world coordinates for the corners of the checkerboard squares.

```squareSize = 20; % millimeters worldPoints = generateCheckerboardPoints(boardSize,squareSize);```

Estimate the fisheye camera calibration parameters based on the image and world points. Use the first image to get the image size.

```I = readimage(images,10); imageSize = [size(I,1) size(I,2)]; params = estimateFisheyeParameters(imagePoints,worldPoints,imageSize);```

Remove lens distortion from the first image `I` and display the results.

```J1 = undistortFisheyeImage(I,params.Intrinsics); figure imshowpair(I,J1,'montage') title('Original Image (left) vs. Corrected Image (right)') J2 = undistortFisheyeImage(I,params.Intrinsics,'OutputView','same', 'ScaleFactor', 0.2); figure imshow(J2)```

`title('Output View with low Scale Factor')`

## References

[1] Scaramuzza, D., A. Martinelli, and R. Siegwart. "A Toolbox for Easy Calibrating Omnidirectional Cameras." Proceedings to IEEE International Conference on Intelligent Robots and Systems, (IROS). Beijing, China, October 7–15, 2006.