Simulation 3D Camera

Camera sensor model with lens in 3D simulation environment

  • Library:
  • Automated Driving Toolbox / Simulation 3D

Description

The Simulation 3D Camera block provides an interface to a camera with a lens in a 3D simulation environment. This environment is rendered using the Unreal Engine® from Epic Games®. The sensor is based on the ideal pinhole camera model, with a lens added to represent a full camera model, including lens distortion. For more details, see Algorithms.

If you set Sample time to -1, the block uses the sample time specified in the Simulation 3D Scene Configuration block. To use this sensor, you must include a Simulation 3D Scene Configuration block in your model.

The block outputs images captured by the camera during simulation. You can use these images to visualize and verify your driving algorithms. In addition, on the Ground Truth tab, you can select options to output the ground truth data for developing depth estimation and semantic segmentation algorithms. You can also output the location and orientation of the camera in the world coordinate system of the scene. The image shows the block with all ports enabled.

The table summarizes the ports and how to enable them.

PortDescriptionParameter for Enabling PortSample Visualization

Image

Outputs an RGB image captured by the cameran/a

Depth

Outputs a depth map with values from 0 m to 1000 meters

Output depth

Labels

Outputs a semantic segmentation map of label IDs that correspond to objects in the scene

Output semantic segmentation

Location

Outputs the location of the camera in the world coordinate system

Output location (m) and orientation (rad)

n/a

Orientation

Outputs the orientation of the camera in the world coordinate system

Output location (m) and orientation (rad)

n/a

Note

The Simulation 3D Scene Configuration block must execute before the Simulation 3D Camera block. That way, the Unreal Engine 3D visualization environment prepares the data before the Simulation 3D Camera block receives it. To check the block execution order, right-click the blocks and select Properties. On the General tab, confirm these Priority settings:

  • Simulation 3D Scene Configuration0

  • Simulation 3D Camera1

For more information about execution order, see How 3D Simulation for Automated Driving Works.

Ports

Output

expand all

3D output camera image, returned as an m-by-n-by-3 array of RGB triplet values. m is the vertical resolution of the image, and n is the horizontal resolution of the image.

Data Types: int8 | uint8

Object depth for each pixel in the image, output as an m-by-n array. m is the vertical resolution of the image, and n is the horizontal resolution of the image. Depth is in the range from 0 to 1000 meters.

Dependencies

To enable this port, on the Ground Truth tab, select Output depth.

Data Types: double

Label identifier for each pixel in the image, output as an m-by-n array. m is the vertical resolution of the image, and n is the horizontal resolution of the image.

The label identifiers have values that correspond to these object types. If the scene contains an object that does not belong to any of the object types shown, that object is assigned an ID of 0.

IDType
0

None/default

1

Building

2

Fence

3

Other

4

Pedestrian

5

Pole

6

Road line

7

Road

8

Sidewalk

9

Vegetation

10

Vehicle

11

Wall

12

Generic traffic sign

13

Stop sign

14

Yield sign

15

Speed limit sign

16

Weight limit sign

17

Right arrow warning sign

18

Left arrow warning sign

19

Left and right arrow warning sign

20

Left chevron warning sign

21

Right chevron warning sign

22

Left one-way sign

23

Right one-way sign

24

Wheelchair warning sign

25

School bus only sign

26

Right turn only arrow sign

27

Left turn only arrow sign

28

Straight only arrow sign

29

Right turn only sign

30

Left turn only sign

31

Straight only sign

32

No left turn sign

33

No right turn sign

34

No thru traffic sign

35

No U-turn symbol sign

36

No right turn symbol sign

37

No left turn symbol sign

38

No right turn on red sign

39

Crosswalk sign

40

Crosswalk signal

41

Traffic signal

42

Curve right warning sign

43

Curve left warning sign

44

Up right arrow warning sign

45

Up left arrow warning sign

46

Down right arrow warning sign

47

Down left arrow warning sign

48

Railroad crossing sign

49

Street sign

50

Roundabout warning sign

51

Fire hydrant

52

Exit sign

53

Bike lane sign

54

Keep right sign

55

Keep left sign

56

Disability sign

57

Sky

58

Curb

59

Flyover ramp

60

Road guard rail

61-63

Not used

64

Adult pedestrian

65

Young pedestrian

66

Generic animal

67

Deer

68

Kangaroo

69

Dog

70

Cat

71

Barricade

72

Motorcycle

73

Commercial vehicle

Dependencies

To enable this port, on the Ground Truth tab, select Output semantic segmentation.

Data Types: uint8

Sensor location along the X-axis, Y-axis, and Z-axis of the scene. The Location values are in the world coordinates of the scene. In this coordinate system, the Z-axis points up from the ground. Units are in meters.

Dependencies

To enable this port, on the Ground Truth tab, select Output location (m) and orientation (rad).

Data Types: double

Roll, pitch, and yaw sensor orientation about the X-axis, Y-axis, and Z-axis of the scene. The Orientation values are in the world coordinates of the scene. These values are positive in the clockwise direction when looking in the positive directions of these axes. Units are in radians.

Dependencies

To enable this port, on the Ground Truth tab, select Output location (m) and orientation (rad).

Data Types: double

Parameters

expand all

Mounting

Unique sensor identifier, specified as a positive integer. In a multisensor system, the sensor identifier distinguishes between sensors. When you add a new sensor block to your model, the Sensor identifier of that block is N + 1. N is the highest Sensor identifier value among existing sensor blocks in the model.

Example: 2

Name of the parent to which the sensor is mounted, specified as Scene Origin or as the name of a vehicle in your model. The vehicle names that you can select correspond to the Name parameters of the Simulation 3D Vehicle with Ground Following blocks in your model. If you select Scene Origin, the block places a sensor at the scene origin.

Example: SimulinkVehicle1

Sensor mounting location.

  • When Parent name is Scene Origin, the block mounts the sensor to the origin of the scene, and Mounting location can be set to Origin only. During simulation, the sensor remains stationary.

  • When Parent name is the name of a vehicle (for example, SimulinkVehicle1) the block mounts the sensor to one of the predefined mounting locations described in the table. During simulation, the sensor travels with the vehicle.

Vehicle Mounting LocationDescriptionOrientation Relative to Vehicle Origin [Roll, Pitch, Yaw] (deg)
Origin

Forward-facing sensor mounted to the vehicle origin, which is on the ground, at the geometric center of the vehicle (see Coordinate Systems for 3D Simulation in Automated Driving Toolbox)

[0, 0, 0]
Front bumper

Forward-facing sensor mounted to the front bumper

[0, 0, 0]
Rear bumper

Backward-facing sensor mounted to the rear bumper

[0, 0, 180]
Right mirror

Downward-facing sensor mounted to the right side-view mirror

[0, –90, 0]
Left mirror

Downward-facing sensor mounted to the left side-view mirror

[0, –90, 0]
Rearview mirror

Forward-facing sensor mounted to the rearview mirror, inside the vehicle

[0, 0, 0]
Hood center

Forward-facing sensor mounted to the center of the hood

[0, 0, 0]
Roof center

Forward-facing sensor mounted to the center of the roof

[0, 0, 0]

The (X, Y, Z) location of the sensor relative to the vehicle depends on the vehicle type. To specify the vehicle type, use the Type parameter of the Simulation 3D Vehicle with Ground Following block to which you are mounting. The tables show the X, Y, and Z locations of sensors in the vehicle coordinate system. In this coordinate system:

  • The X-axis points forward from the vehicle.

  • The Y-axis points to the left of the vehicle, as viewed when facing forward.

  • The Z-axis points up from the ground.

  • Roll, pitch, and yaw are clockwise-positive when looking in the positive direction of the X-axis, Y-axis, and Z-axis, respectively. When looking at a vehicle from the top down, then the yaw angle (that is, the orientation angle) is counterclockwise-positive, because you are looking in the negative direction of the axis.

Muscle Car — Sensor Locations Relative to Vehicle Origin

Mounting LocationX (m)Y (m)Z (m)
Front bumper2.4700.45
Rear bumper–2.4700.45

Right mirror

0.43–1.081.01

Left mirror

0.431.081.01

Rearview mirror

0.3201.20

Hood center

1.2801.14

Roof center

–0.2501.58

Sedan — Sensor Locations Relative to Vehicle Origin

Mounting LocationX (m)Y (m)Z (m)
Front bumper2.4200.51
Rear bumper–2.4200.51

Right mirror

0.59–0.941.09

Left mirror

0.590.941.09

Rearview mirror

0.4301.31

Hood center

1.4601.11

Roof center

–0.4501.69

Sport Utility Vehicle — Sensor Locations Relative to Vehicle Origin

Mounting LocationX (m)Y (m)Z (m)
Front bumper2.4200.51
Rear bumper–2.4200.51

Right mirror

0.60–11.35

Left mirror

0.6011.35

Rearview mirror

0.3901.55

Hood center

1.5801.39

Roof center

–0.5602

Small Pickup Truck — Sensor Locations Relative to Vehicle Origin

Mounting LocationX (m)Y (m)Z (m)
Front bumper3.0700.51
Rear bumper–3.0700.51

Right mirror

1.10–1.131.52

Left mirror

1.101.131.52

Rearview mirror

0.8501.77

Hood center

2.2201.59

Roof center

002.27

Hatchback — Sensor Locations Relative to Vehicle Origin

Mounting LocationX (m)Y (m)Z (m)
Front bumper1.9300.51
Rear bumper–1.9300.51

Right mirror

0.43–0.841.01

Left mirror

0.430.841.01

Rearview mirror

0.3201.27

Hood center

1.4401.01

Roof center

001.57

To determine the location of the sensor in world coordinates, open the sensor block. Then, on the Ground Truth tab, select Output location (m) and orientation (rad) and inspect the data from the Location output port.

Select this parameter to specify an offset from the mounting location by using the Relative translation [X, Y, Z] (m) and Relative rotation [Roll, Pitch, Yaw] (deg) parameters.

Translation offset relative to the mounting location of the sensor, specified as a real-valued 1-by-3 vector of the form [X, Y, Z]. Units are in meters.

If you mount the sensor to a vehicle by setting Parent name to the name of that vehicle, then X, Y, and Z are in the vehicle coordinate system, where:

  • The X-axis points forward from the vehicle.

  • The Y-axis points to the left of the vehicle, as viewed when facing forward .

  • The Z-axis points up.

The origin is the mounting location specified in the Mounting location parameter. This origin is different from the vehicle origin, which is the geometric center of the vehicle.

If you mount the sensor to the scene origin by setting Parent name to Scene Origin, then X, Y, and Z are in the world coordinates of the scene.

For more details about the vehicle and world coordinate systems, see Coordinate Systems for 3D Simulation in Automated Driving Toolbox.

Example: [0,0,0.01]

Dependencies

To enable this parameter, select Specify offset.

Rotational offset relative to the mounting location of the sensor, specified as a real-valued 1-by-3 vector of the form [Roll, Pitch, Yaw] . Roll, pitch, and yaw are the angles of rotation about the X-, Y-, and Z-axes, respectively. Units are in degrees.

If you mount the sensor to a vehicle by setting Parent name to the name of that vehicle, then X, Y, and Z are in the vehicle coordinate system, where:

  • The X-axis points forward from the vehicle.

  • The Y-axis points to the left of the vehicle, as viewed when facing forward .

  • The Z-axis points up.

  • Roll, pitch, and yaw are clockwise-positive when looking in the forward direction of the X-axis, Y-axis, and Z-axis, respectively. If you view a scene from a 2D top-down perspective, then the yaw angle (also called the orientation angle) is counterclockwise-positive, because you are viewing the scene in the negative direction of the Z-axis.

The origin is the mounting location specified in the Mounting location parameter. This origin is different from the vehicle origin, which is the geometric center of the vehicle.

If you mount the sensor to the scene origin by setting Parent name to Scene Origin, then X, Y, and Z are in the world coordinates of the scene.

For more details about the vehicle and world coordinate systems, see Coordinate Systems for 3D Simulation in Automated Driving Toolbox.

Example: [0,0,10]

Dependencies

To enable this parameter, select Specify offset.

Sample time of the block in seconds, specified as a positive scalar. The 3D simulation environment frame rate is the inverse of the sample time.

If you set the sample time to -1, the block inherits its sample time from the Simulation 3D Scene Configuration block.

Parameters

These intrinsic camera parameters are equivalent to the properties of a cameraIntrinsics object. To obtain the intrinsic parameters for your camera, use the Camera Calibrator app.

Focal length of the camera, specified as a 1-by-2 positive integer vector of the form [fx, fy]. Units are in pixels.

fx = F × sx

fy = F × sy

where:

  • F is the focal length in world units, typically millimeters.

  • [sx, sy] are the number of pixels per world unit in the x and y direction, respectively.

This parameter is equivalent to the FocalLength property of a cameraIntrinsics object.

Optical center of the camera, specified as a 1-by-2 positive integer vector of the form [cx,cy]. Units are in pixels.

This parameter is equivalent to the PrincipalPoint property of a cameraIntrinsics object.

Image size produced by the camera, specified as a 1-by-2 positive integer vector of the form [mrows,ncols]. Units are in pixels.

This parameter is equivalent to the ImageSize property of a cameraIntrinsics object.

Radial distortion coefficients, specified as a real-valued 1-by-2 or 1-by-3 nonnegative vector. Radial distortion occurs when light rays bend more than the edges of a lens than they do at its optical center. The distortion is greater when the lens is smaller. The block calculates the radial-distorted location of a point. Units are dimensionless.

This parameter is equivalent to the RadialDistortion property of a cameraIntrinsics object.

Tangential distortion coefficients, specified as a real-valued 1-by-2 nonnegative vector. Tangential distortion occurs when the lens and the image plane are not parallel. The coordinates are expressed in world units. Units are dimensionless.

This parameter is equivalent to the TangentialDistortion property of a cameraIntrinsics object.

Skew angle of the camera axes, specified as a nonnegative scalar. If the X-axis and Y-axis are exactly perpendicular, then the skew must be 0. Units are dimensionless.

This parameter is equivalent to the Skew property of a cameraIntrinsics object.

Ground Truth

Select this parameter to output a depth map at the Depth port.

Select this parameter to output a semantic segmentation map of label IDs at the Labels port.

Select this parameter to output the location and orientation of the sensor at the Location and Orientation ports, respectively.

Tips

Algorithms

The block uses the camera model proposed by Jean-Yves Bouguet [1]. The model includes:

  • The pinhole camera model [2]

  • Lens distortion [3]

The pinhole camera model does not account for lens distortion because an ideal pinhole camera does not have a lens. To accurately represent a real camera, the full camera model used by the block includes radial and tangential lens distortion.

For more details, see What Is Camera Calibration? (Computer Vision Toolbox)

References

[1] Bouguet, J. Y. Camera Calibration Toolbox for Matlab. http://www.vision.caltech.edu/bouguetj/calib_doc

[2] Zhang, Z. "A Flexible New Technique for Camera Calibration." IEEE Transactions on Pattern Analysis and Machine Intelligence. Vol. 22, No. 11, 2000, pp. 1330–1334.

[3] Heikkila, J., and O. Silven. “A Four-step Camera Calibration Procedure with Implicit Image Correction.” IEEE International Conference on Computer Vision and Pattern Recognition. 1997.

Introduced in R2019b