Main Content

Generate RoadRunner Scenario from Recorded Sensor Data

This example shows how to generate a RoadRunner scenario containing actor trajectories from recorded Global Positioning System (GPS) data and preprocessed actor track list.

RoadRunner Scenario is an interactive editor that enables you to design scenarios for simulating and testing automated driving systems. Generating scenarios from recorded vehicle data enables you to mimic real-world driving scenarios and improve the test coverage of automated driving systems.

This figure shows these steps.

  • Smooth GPS data and format actor track list

  • Reconstruct ego-vehicle trajectories

  • Extract non-ego actor properties from the track list

  • Export actor trajectories to CSV files

  • Import map in RoadRunner with same GeoReference

  • Import CSV actor trajectories in RoadRunner

  • Simulate RoadRunner Scenario

Load Sensor Data

Download a ZIP file containing a subset of sensor data from the PandaSet data set, and then unzip the file. This file contains GPS data, an actor track list, and camera information. In this example, you use the camera data for visual validation of the generated scenario.

dataFolder = tempdir;
dataFilename = "PandasetSensorData.zip";
url = "https://ssd.mathworks.com/supportfiles/driving/data/" + dataFilename;
filePath = fullfile(dataFolder,dataFilename);
if ~isfile(filePath)
    websave(filePath,url)
end
unzip(filePath,dataFolder)
dataset = fullfile(dataFolder,"PandasetSensorData");
data = load(fullfile(dataset,"sensorData.mat"));

Load the GPS data into the workspace.

gpsData = data.GPSData;

gpsData is a table with these columns:

  • timeStamp — Time, in seconds, at which the GPS data was collected.

  • latitude — Latitude coordinate value of the ego vehicle. Units are in degrees.

  • longitude — Longitude coordinate value of the ego vehicle. Units are in degrees.

  • altitude — Altitude coordinate value of the ego vehicle. Units are in meters.

Display the first five entries of gpsData.

gpsData(1:5,:)
ans=5×4 table
    timeStamp     latitude    longitude    altitude
    __________    ________    _________    ________

    1.5576e+09     37.374      -122.06      42.858 
    1.5576e+09     37.374      -122.06      42.858 
    1.5576e+09     37.374      -122.06      42.854 
    1.5576e+09     37.374      -122.06      42.849 
    1.5576e+09     37.374      -122.06      42.848 

Load the actor track list data. Alternatively, you can generate an actor track list by processing raw camera or lidar sensor data. For more information on how to generate actor track list from camera data, see the Extract Vehicle Track List from Recorded Camera Data for Scenario Generation example. For more information on generating a track list from lidar data, see the Extract Vehicle Track List from Recorded Lidar Data for Scenario Generation example.

tracklist = data.ActorTracklist;

tracklist is a table with these columns:

  • timeStamp — Time, in seconds, for each actor track.

  • TrackerInfo — Track information for the non-ego actors.

Each row of the TrackerInfo column is an M-by-1 structure with these fields, where M is the number of tracks per timestamp.

  • TrackID — Track ID of the actor.

  • ClassID — Classification ID of the actor.

  • Position — Position of the actor with respect to the ego vehicle. Units are in meters.

  • Dimension — Dimensions of the actor in the form [length width height]. Units are in meters. This field is optional.

  • Yaw — Yaw angle of the actor with respect to the ego vehicle. Units are in degrees. This field is optional.

Display the first five entries of tracklist.

tracklist(1:5,:)
ans=5×2 table
    timeStamp     TrackerInfo 
    __________    ____________

    1.5576e+09    {6×1 struct}
    1.5576e+09    {6×1 struct}
    1.5576e+09    {6×1 struct}
    1.5576e+09    {6×1 struct}
    1.5576e+09    {6×1 struct}

Display the TrackerInfo value for the first timestamp.

tracklist.TrackerInfo{1,1}
ans=6×1 struct array with fields:
    Position
    Yaw
    TrackID
    Dimension
    ClassID

Load the camera data recorded from a forward-facing monocular camera mounted on the ego vehicle.

cameraData = data.CameraData;

The camera data is a table with two columns:

  • timeStamp — Time, in seconds, at which the image data was captured.

  • fileName — Filenames of the images in the data set.

The images are located in the Camera folder in the dataset directory. Create a table that contains the file paths of these images for each timestamp by using the helperUpdateTable function.

imageFolder = "Camera";
cameraData  = helperUpdateTable(cameraData,dataset,imageFolder);

Display the first five entries of cameraData.

cameraData(1:5,:)

Remove data from workspace.

clear data

Crop and Preprocess Sensor Data

Crop the GPS, actor track list, and camera data relative to the GPS timestamp range by using the helperCropData function.

startTime = gpsData.timeStamp(1);
endTime = gpsData.timeStamp(end);

% Pack all the tables in a cell array. 
recordedData = {gpsData,tracklist,cameraData};

% Crop the data.
recordedData = helperCropData(recordedData,startTime,endTime);

The timestamp values of the recorded data set are in the POSIX® format, which Scenario Builder for Automated Driving Toolbox™ supports. Use the helperNormTimeInSecs function to normalize the timestamps using these arguments:

  • scale — Scale by which to convert the timestamp. Because the recorded timestamps are already in seconds, specify this argument as 1.

  • offset — Offset of the simulation start time. Specify the start time as the first timestamp in gpsData.

scale = 1;
offset = startTime;
recordedData =  helperNormTimeInSecs(recordedData,offset,scale);

Extract the GPS data, actor track list, and camera data with updated timestamp values from recordedData.

gpsData = recordedData{1,1}; 
tracklist = recordedData{1,2};
cameraData = recordedData{1,3};

Remove recordedData from the workspace.

clear recordedData

Create Ego Trajectory

Convert geographic GPS coordinates to local east-north-up (ENU) coordinates by using the latlon2local function. Specify an origin by using the latitude and longitude values of the first entry in the GPS data. The transformed coordinates define the trajectory waypoints of the ego vehicle. Units are in meters.

geoReference = [gpsData.latitude(1) gpsData.longitude(1) 0];    
[xEast,yNorth,zUp] = latlon2local(gpsData.latitude,gpsData.longitude,gpsData.altitude,geoReference);
waypoints = [xEast yNorth zUp];

Raw GPS data often contains noise. Smooth the GPS waypoints by using the smoothdata function.

window = round(size(waypoints,1)*0.2);
waypoints = smoothdata(waypoints,"rloess",window);

If your GPS data also suffers from drift in position and orientation, then you must improve your ego vehicle localization to generate an accurate ego trajectory. For more information, see the Improve Ego Vehicle Localization example.

Create the ego trajectory using the waypoints and their corresponding time of arrivals by using the waypointTrajectory (Sensor Fusion and Tracking Toolbox) System object™. For this example assume that no tracked vehicle leaves the ground at any of the waypoints, therefore set all the altitude values to 0. You must set the ReferenceFrame property of this System object to "ENU" because Scenario Builder for Automated Driving Toolbox supports only the ENU format for local coordinate data.

waypoints = double([waypoints(:,1:2) zeros(size(zUp))]);
egoTrajectory = waypointTrajectory(waypoints,gpsData.timeStamp,ReferenceFrame="ENU");

Extract Non-Ego Actor Trajectories

Visualize the actor track list and camera images by using the birdsEyePlot and helperPlotActors functions.

% Initialize the figure with bird's eye plot.
currentFigure = figure(Visible="on",Position=[0 0 1400 600]);
hPlot = axes(uipanel(currentFigure,Position=[0 0 0.5 1],Title="Non-Ego Actors"));
bep = birdsEyePlot(XLim=[0 70],YLim=[-35 35],Parent=hPlot);
camPlot = axes(uipanel(currentFigure,Position=[0.5 0 0.5 1],Title="Camera View"));
helperPlotActors(bep,camPlot,tracklist,cameraData)

Extract actor properties such as entry time, exit time, and dimension from the track list data by using the actorprops function. The function uses extracted ego trajectory information to return the non-ego actor properties in the world frame.

nonEgoActorInfo = actorprops(tracklist,egoTrajectory);

Display the first five entries of nonEgoActorInfo.

nonEgoActorInfo(1:5,:)
ans=5×10 table
    Age                    TrackID                     ClassID           Dimension           EntryTime    ExitTime             Mesh               Waypoints           Speed              Yaw      
    ___    ________________________________________    _______    _______________________    _________    ________    ______________________    ______________    ______________    ______________

     10    {'a6ebfc4a-1adb-4920-af12-f7ac56602785'}       1       2.037    5.273    1.825        0        0.89983     1×1 extendedObjectMesh    { 10×3 double}    { 10×1 double}    { 10×1 double}
    400    {'609aed9d-6681-4e61-83a4-d1e263723107'}       1       1.911    4.672    1.527        0           39.9     1×1 extendedObjectMesh    {400×3 double}    {400×1 double}    {400×1 double}
     10    {'1823b4a0-d5fa-41c0-9335-98ca77c933b6'}       1       2.043    4.537     1.87        0        0.89983     1×1 extendedObjectMesh    { 10×3 double}    { 10×1 double}    { 10×1 double}
    139    {'4cd79810-9938-47ba-a3a9-e81221024549'}       1       2.199    4.827    1.968        0         13.799     1×1 extendedObjectMesh    {139×3 double}    {139×1 double}    {139×1 double}
    400    {'64bf8e46-c5ba-47af-b6fe-40aef7c1aa71'}       1       1.981    4.974     1.58        0           39.9     1×1 extendedObjectMesh    {400×3 double}    {400×1 double}    {400×1 double}

Create CSV Trajectories for RoadRunner Scenario

RoadRunner Scenario can import actor trajectories using CSV files. Create a folder in the example working folder, and write CSV files for the ego and non-ego actors to that folder using the helperWriteRoadRunnerCSV function.

csvFolder = "CSVTrajectories";
if ~isfolder(csvFolder) 
    mkdir(csvFolder)
end
helperWriteRoadRunnerCSV(egoTrajectory,nonEgoActorInfo,csvFolder)

Import Scene from OpenStreetMap and GPS Data

Create a geographic player using a geoplayer object and display the full route using GPS data.

zoomLevel = 16;
center = mean([gpsData.latitude gpsData.longitude]);
player = geoplayer(center(1),center(2),zoomLevel);
plotRoute(player,gpsData.latitude,gpsData.longitude)

Obtain geographic bounding box coordinates from the GPS data by using the getMapROI function.

mapStruct = getMapROI(gpsData.latitude,gpsData.longitude);

The map file required for importing roads of the specified area is downloaded from the OpenStreetMap® (OSM) website. OpenStreetMap provides access to worldwide, crowd-sourced, map data. The data is licensed under the Open Data Commons Open Database License (ODbL). For more information on the ODbL, see the Open Data Commons Open Database License site.

url = mapStruct.osmUrl;
filename = "drive_map.osm";
websave(filename,url,weboptions(ContentType="xml"));

To open RoadRunner using MATLAB®, specify the path to your RoadRunner project. This code shows the path for a sample project folder location in Windows®.

rrProjectPath = "C:\RR\MyProject";

Specify the path to your local RoadRunner installation folder. This code shows the path for the default installation location in Windows.

rrAppPath = "C:\Program Files\RoadRunner R2022b\bin\win64";

Open RoadRunner using the specified path to your project.

rrApp = roadrunner(rrProjectPath,InstallationFolder=rrAppPath);

Interactively import the downloaded OpenStreetMap data into RoadRunner, and build roads for the imported data. For more information on how to import the OpenSteetMap data and build roads, see Build Roads Using OpenStreetMap Data (RoadRunner). To accurately map actors with extracted roads, use the same geoReference value while importing the OpenStreetMap. To set the geoReference value, use this process:

  • In the World Settings tool, specify the World Origin latitude and longitude values as the first and second value of geoReference respectively. Then, select Apply World Changes.

Create RoadRunner Scenario

Create a new scenario by using the newScenario function of the roadrunner object.

newScenario(rrApp)

Import the trajectories of the ego and non-ego actors from the corresponding CSV files by using the importScenario function.

format = "CSV Trajectory";
csvDir = dir(csvFolder+filesep+"*.csv");
for k=1:numel(csvDir)
    filename = fullfile(pwd,csvFolder,csvDir(k).name);
    importScenario(rrApp,filename,format)
end

Create a scenario simulation object for the current RoadRunner Scenario using the createSimulation function. The simulation object enables you to programatically interact with the RoadRunner scenario simulation using MATLAB.

rrSim = createSimulation(rrApp);
Connection status: 1
Connected to RoadRunner Scenario server on localhost:61632, with client id {52bb112b-31cb-4ffc-93a7-f152e7552bc6}

Define the simulation parameters of RoadRunner Scenario. Specify the maximum simulation time using the last timestamp of the GPS data. To plot the simulation results, enable data logging.

endTime = gpsData.timeStamp(end);
set(rrSim,MaxSimulationTime=endTime);
set(rrSim,Logging="on");

Run the simulation. Monitor the status of the simulation, and wait for the simulation to complete. Because RoadRunner Scenario cannot remove actors after their exit times, scenario simulation may can fail due to collision. To avoid stopping the scenario on collision, remove fail conditions using these steps:

1. In the Logic editor, click the condition node at the end of the scenario.

2. In the Attributes pane, click Remove Fail Condition.

set(rrSim,SimulationCommand="Start")
 while strcmp(get(rrSim,"SimulationStatus"),"Running")
     simstatus = get(rrSim,'SimulationStatus');
     pause(1)
 end

To view the scenario from the ego vehicle view or chase view, in the Simulation pane, in the Camera, section set Camera View to either Follow or Front. Then, set the Actor attribute to vehicle21, which is the ego vehicle for the scenario in this example.

Note: Since OpenStreetMap cannot provide accurate lane information, such as lane width, the actor can appear on or outside of their lane. To avoid such issues, you can either use HD Map or follow the Generate High Definition Scene from Lane Detections example to generate an high definition map from OpenStreetMap and lane detections.

This figure shows a scenario created in RoadRunner Scenario using OpenStreetMap and actor track list.

rrsosmTracklist.jpg

See Also

Functions

Related Topics