The Video Labeler app provides an easy way to mark rectangular region of interest (ROI) labels, polyline ROI labels, pixel ROI labels, and scene labels in a video or image sequence. This example gets you started using the app by showing you how to:
Manually label an image frame from a video.
Automatically label across image frames using an automation algorithm.
Export the labeled ground truth data.
Open the app and load a video of vehicles driving on a highway. Videos must be in a file
format readable by VideoReader
.
videoLabeler('visiontraffic.avi')
Alternatively, open the app from the Apps tab, under Image Processing and Computer Vision. Then, from the Load menu, load a video data source.
Explore the video. Click the Play button to play the entire video, or use the slider
to navigate between frames.
The app also enables you to load image sequences, with corresponding timestamps, by selecting Load > Image Sequence. The images must be readable by imread
.
To load a custom data source that is readable by VideoReader
or imread
, see Use Custom Data Source Reader for Ground Truth Labeling.
You can label the entire video or start with a portion of the video. In this example, you label a five-second time interval within the loaded video. In the text boxes below the video, enter these times in seconds:
In the Current Time box, type 5
and press Enter.
In the Start Time box, type 5
so
that the slider is at the start of the time interval.
In the End Time box, type
10
.
Optionally, to make adjustments to the time interval, click and drag the red interval flags.
The entire app is now set up to focus on this specific time interval. The video plays only within this interval, and labeling and automation algorithms apply only to this interval. You can change the interval at any time by moving the flags.
To expand the time interval to fill the entire playback section, click Zoom in Time Interval.
Define the labels you intend to draw. In this example, you define labels directly within the
app. To define labels from the MATLAB® command line instead, use the labelDefinitionCreator
.
An ROI label is a label that corresponds to a region of interest (ROI). You can define these types of ROI labels.
ROI Label | Description | Example: Driving Scene |
---|---|---|
Rectangle | Draw rectangular ROI labels (bounding boxes) around objects. | Vehicles, pedestrians, road signs |
Line | Draw linear ROI labels to represent lines. To draw a polyline ROI, use two or more points. | Lane boundaries, guard rails, road curbs |
Pixel label | Assign labels to pixels for semantic segmentation. You can label pixels manually using polygons, brushes, or flood fill. See Label Pixels for Semantic Segmentation. | Vehicles, road surface, trees, pavement |
In this example, you define a vehicle
group for labeling types of
vehicles, and then create a Rectangle
ROI label for a
Car
and a Truck
.
In the ROI Label Definition pane on the left, click Label.
Create a Rectangle
label named
Car
.
From the Group
drop-down menu, select
New Group
and name the group
Vehicle
Click OK.
The Vehicle group name appears in the ROI Label Definition pane with the label Car created. You can move a labels to a different position or group by left-clicking and dragging the label.
Add a second label. Click Label. Name the label Truck and make sure the Vehicle group is selected. Click OK.
In the first video frame within the time interval, use the mouse to draw rectangular Car ROIs around the two vehicles.
A sublabel is a type of ROI label that corresponds to a parent ROI label. Each sublabel must belong to, or be a child of, a specific label defined in the ROI Label Definition pane. For example, in a driving scene, a vehicle label might have sublabels for headlights, license plates, or wheels.
Define a sublabel for headlights.
In the ROI Label Definition pane on the left, click the Car label.
Click Sublabel.
Create a Rectangle
sublabel named headlight
and optionally write a description. Click OK.
The headlight sublabel appears in the ROI Label Definition pane. The sublabel is nested under the selected ROI label, Car, and has the same color as its parent label.
You can add multiple sublabels under a label. You can also drag-and-drop the sublabels to reorder them in the list. Right-click any label for additional edits.
In the ROI Label Definition pane, select the headlight sublabel.
In the video frame, select the Car label. The label turns yellow when selected. You must select the Car label (parent ROI) before you can add a sublabel to it.
Draw headlight sublabels for each of the cars.
Repeat the previous steps to label the headlights of the other car. To draw the labels more precisely, use the Pan, Zoom In, and Zoom Out options available from the toolstrip.
Sublabels can only be used with rectangular or polyline ROI labels and cannot have their own sublabels. For more details on working with sublabels, see Use Sublabels and Attributes to Label Ground Truth Data.
An attribute provides further categorization of an ROI label or sublabel. Attributes specify additional information about a drawable label. For example, in a driving scene, attributes might include the type or color of a vehicle.
You can define these types of attributes.
Attribute Type | Sample Attribute Definition | Sample Default Values |
---|---|---|
|
|
|
|
| |
|
| |
|
|
Add an attribute for the vehicle type.
In the ROI Label Definition pane on the left, select the Car label and click Attribute.
In the Attribute Name box, type
carType
. Set the attribute type to
List
.
In the List Items section, type different types of
cars, such as Sedan
,
Hatchback
, and Wagon
, each on
its own line. Optionally give the attribute a description, and click
OK.
In the first frame of the video, select a Car ROI label. In the Attributes and Sublabels pane, select the appropriate carType attribute value for that vehicle.
Repeat the previous step to assign a carType attribute to the other vehicle.
You can also add attributes to sublabels. Add an attribute for the headlight sublabel that tells whether the headlight is on.
In the ROI Label Definition pane on the left, select the headlight sublabel and click Attribute.
In the Attribute Name box, type
isOn
. Set the attribute type to
Logical
. Leave the Default
Value set to Empty
, optionally
write a description, and click OK.
Select a headlight in the video frame. Set the appropriate
isOn attribute value, or leave the attribute value
set to Empty
.
Repeat the previous step to set the isOn attribute for the other headlights.
To delete an attribute, right-click an ROI label or sublabel, and select the attribute to delete. Deleting the attribute removes attribute information from all previously created ROI label annotations.
A scene label defines additional information for the entire scene. Use scene labels to describe conditions, such as lighting and weather, or events, such as lane changes.
Create a scene label to use in the video.
In the Scene Label Definition pane on the left, click the
Define new scene label button, and create a scene label
named sunny
. Make sure Group is set
to None
. Click OK.
The Scene Label Definition pane shows the scene label definition. The scene labels that are applied to the current frame appear in the Scene Labels pane on the right. The sunny scene label is empty (white), because the scene label has not yet been applied to the frame.
The entire scene is sunny, so specify to apply the sunny scene label over the entire time interval. With the sunny scene label definition still selected in the Scene Label Definition pane, select Time Interval.
Click Add Label.
The sunny label now applies to all frames in the time interval.
So far, you have labeled only one frame in the video. To label the remaining frames, choose one of these options.
When you click the right arrow key to advance to the next frame, the ROI labels from the previous frame do not carry over. Only the sunny scene label applies to each frame, because this label was applied over the entire time interval.
Advance frame by frame and draw the label and sublabel ROIs manually. Also update the attribute information for these ROIs.
To speed up the labeling process, you can use an automation algorithm within the app. You can either define your own automation algorithm, see Create Automation Algorithm for Labeling and Temporal Automation Algorithms, or use a built-in automation algorithm. In this example, you label the ground truth using a built-in point tracking algorithm.
In this example, you automate the labeling of only the Car ROI labels. The built-in automation algorithms do not support sublabel and attribute automation.
Select the labels you want to automate. In the first frame of the video, press Ctrl and click to select the two Car label annotations. The labels are highlighted in yellow.
From the app toolstrip, select Select Algorithm > Point Tracker. This algorithm tracks one or more rectangle ROIs over short intervals using the Kanade-Lucas-Tomasi (KLT) algorithm.
(optional) Configure the automation settings. Click Configure Automation. By default, the automation algorithm applies labels from the start of the time interval to the end. To change the direction and start time of the algorithm, choose one of the options shown in this table.
Direction of automation | Run automation from | Example |
---|---|---|
|
|
|
|
| |
|
|
|
|
|
The Import selected ROIs must be selected so that the Car labels you selected are imported into the automation session.
Click Automate to open an automation session. The algorithm instructions appear in the right pane, and the selected labels are available to automate.
Click Run to track the selected ROIs over the interval.
Examine the results of running the algorithm.
The vehicles that enter the scene later are unlabeled. The unlabeled vehicles did not have an initial ROI label, so the algorithm did not track them. Click Undo Run. Use the slider to find the frames where each vehicle first appears. Draw vehicle ROIs around each vehicle, and then click Run again.
Advance frame by frame and manually move, resize, delete, or add ROIs to improve the results of the automation algorithm.
When you are satisfied with the algorithm results, click Accept. Alternatively, to discard labels generated during the session and label manually instead, click Cancel. The Cancel button cancels only the algorithm session, not the app session.
Optionally, you can now manually label the remaining frames with sublabel and attribute information.
To further evaluate your labels, you can view a visual summary of the labeled ground truth. From the app toolstrip, select View Label Summary. Use this summary to compare the frames, frequency of labels, and scene conditions. For more details, see View Summary of Ground Truth Labels. This summary does not support sublabels or attributes.
You can export the labeled ground truth to a MAT-file or to a variable in the MATLAB workspace. In both cases, the labeled ground truth is stored as a groundTruth
object. You can use this object to train a deep-learning-based computer
vision algorithm. For more details, see Training Data for Object Detection and Semantic Segmentation.
If you export pixel data, the pixel label data and ground truth data are saved in separate files but in the same folder. For considerations when working with exported pixel labels, see How Labeler Apps Store Exported Pixel Labels.
In this example, you export the labeled ground truth to the MATLAB workspace. From the app toolstrip, select Export Labels > To Workspace. The exported MATLAB variable, gTruth
, is a groundTruth
object.
Display the properties of the exported groundTruth
object. The information in
your exported object might differ from the information shown here.
gTruth
gTruth = groundTruth with properties: DataSource: [1×1 groundTruthDataSource] LabelDefinitions: [3×5 table] LabelData: [531×3 timetable]
DataSource
is a groundTruthDataSource
object containing the path
to the video and the video timestamps. Display the properties of this
object.
gTruth.DataSource
ans = groundTruthDataSource for a video file with properties Source: ...matlab\toolbox\vision\visiondata\visiontraffic.avi TimeStamps: [531×1 duration]
LabelDefinitions
is a table containing information about the label
definitions. This table does not contain information about the labels that are drawn on the
video frames. To save the label definitions in their own MAT-file, from the app toolstrip,
select Save > Label Definitions. You can then import these label definitions into another app session by
selecting Import Files.
Display the label definitions table. Each row contains information about an ROI label
definition or a scene label definition. If you exported pixel label data, the
LabelDefinitions
table also includes a PixelLabelID
column containing the ID numbers for each pixel label
definition.
gTruth.LabelDefinitions
ans = 3×5 table Name Type Group Description Hierarchy _______ _________ _________ ___________ ____________ 'Car' Rectangle 'Vehicle' '' [1×1 struct] 'Truck' Rectangle 'Vehicle' '' [] 'sunny' Scene 'None' '' []
Within LabelDefinitions
, the Hierarchy
column stores information about the sublabel and attribute definitions of a parent ROI label.
Display the sublabel and attribute information for the Car
label.
gTruth.LabelDefinitions.Hierarchy{1}
ans = struct with fields: carType: [1×1 struct] headlight: [1×1 struct] Type: Rectangle Description: ''
Display information about the headlight
sublabel.
gTruth.LabelDefinitions.Hierarchy{1}.headlight
ans = struct with fields: Type: Rectangle Description: '' isOn: [1×1 struct]
Display information about the carType
attribute.
gTruth.LabelDefinitions.Hierarchy{1}.carType
ans = struct with fields: ListItems: {3×1 cell} Description: ''
LabelData
is a timetable containing information about the ROI labels drawn at each timestamp, across the entire video. The timetable contains one column per label.
Display the first few rows of the timetable. The first few timestamps indicate that no
vehicles were detected and that the sunny
scene label is
false
. These results are because this portion of the video was not
labeled. Only the time interval of 5–10 seconds was
labeled.
labelData = gTruth.labelData; head(labelData)
ans = 8×3 timetable Time Car Truck sunny __________ ____________ ____________ _____ 5.005 sec [1×2 struct] [1×0 struct] true 5.0384 sec [1×2 struct] [1×0 struct] true 5.0717 sec [1×2 struct] [1×0 struct] true 5.1051 sec [1×2 struct] [1×0 struct] true 5.1385 sec [1×2 struct] [1×0 struct] true 5.1718 sec [1×2 struct] [1×0 struct] true 5.2052 sec [1×2 struct] [1×0 struct] true 5.2386 sec [1×2 struct] [1×0 struct] true
Display the first few timetable rows from the 5-10 second interval that contains labels.
gTruthInterval = labelData(timerange('00:00:05','00:00:10'),:); head(gTruthInterval)
ans = 8×3 timetable Time Car Truck sunny __________ ____________ ____________ _____ 5.005 sec [1×2 struct] [1×0 struct] true 5.0384 sec [1×2 struct] [1×0 struct] true 5.0717 sec [1×2 struct] [1×0 struct] true 5.1051 sec [1×2 struct] [1×0 struct] true 5.1385 sec [1×2 struct] [1×0 struct] true 5.1718 sec [1×2 struct] [1×0 struct] true 5.2052 sec [1×2 struct] [1×0 struct] true 5.2386 sec [1×2 struct] [1×0 struct] true
For each Car
label, the structure includes the position of the bounding box
and information about its sublabels and attributes.
Display the bounding box positions for the vehicles at the start of the time interval. Your bounding box positions might differ from the ones shown here.
gTruthInterval(1,:).Car{1}.Position % [x y width height], in pixels
ans = 1×4 single row vector 415.8962 82.4737 130.8474 129.3805 ans = 1×4 single row vector 235.2182 1.0000 117.0611 55.3500
From the app toolstrip, select Save and save a MAT-file of the app session. The saved session includes the data source, label definitions, and labeled ground truth. It also includes your session preferences, such as the layout of the app. To change layout options, select Layout.
The app session MAT-file is separate from the ground truth MAT-file that is exported when you select Export > From File. To share labeled ground truth data, as a best practice, share the ground truth MAT-file containing the groundTruth
object, not the app session MAT-file. For more details, see Share and Store Labeled Ground Truth Data.
groundTruth
| groundTruthDataSource
| labelDefinitionCreator
| vision.labeler.AutomationAlgorithm
| vision.labeler.mixin.Temporal