Main Content

rlSimulationOptions

Options for simulating a reinforcement learning agent within an environment

expand all in page

Description

Use an rlSimulationOptions object to specify simulation options for simulating a reinforcement learning agent within an environment. To perform the simulation, use sim.

For more information on agents training and simulation, see Train Reinforcement Learning Agents.

Creation

Syntax

simOpts = rlSimulationOptions
opt = rlSimulationOptions(Name=Value)

Description

simOpts = rlSimulationOptions returns the default options for simulating a reinforcement learning environment against an agent. Use simulation options to specify parameters about the simulation such as the maximum number of steps to run per simulation and the number of simulations to run. After configuring the options, use simOpts as an input argument for sim.

opt = rlSimulationOptions(Name=Value) creates a simulation options set with the specified properties using one or more name-value pair arguments.

example

Properties

expand all

Number of steps to run the simulation, specified as the comma-separated pair consisting of 'MaxSteps' and a positive integer. In general, you define episode termination conditions in the environment. This value is the maximum number of steps to run in the simulation if those termination conditions are not met.

Example: MaxSteps=1000

Number of simulations to run, specified as the comma-separated pair consisting of 'NumSimulations' and a positive integer. At the start of each simulation, sim resets the environment. You specify what happens on environment reset when you create the environment. For instance, resetting the environment at the start of each episode can include randomizing initial state values, if you configure your environment to do so. In that case, running multiple simulations allows you to validate performance of a trained agent over a range of initial conditions.

Example: NumSimulations=10

Stop simulation when an error occurs, specified as "off" or "on". When this option is "off", errors are captured and returned in the SimulationInfo output of sim, and simulation continues.

Storage type for environment data, specified as "memory", "file", or "none". This option specifies the type of storage used for data generated during training or simulation by a Simulink® environment. Specifically, the software saves anything that appears as the output of a sim (Simulink) command.

Note that this option does not affect (and is not affected by) any option to save agents during training specified within a training option object, or any data logged by a FileLogger or MonitorLogger object.

The default value is "memory", indicating that data is stored in an internal memory variable. When you set this option to "file", data is stored to disk, in MAT-files in the directory specified by the SaveSimulationDirectory property, and using the MAT-file version specified by the SaveFileVersion property. When you set this option to "none", simulation data is not stored.

You can use this option to prevent out-of-memory issues during training or simulation.

Example: "none"

Folder used to save environment data, specified as a string or character vector. The folder name can contain a full or relative path. When you set the SimulationStorageType property to "file", the software saves data generated during training or simulation by a Simulink environment in MAT-files in this folder, using the MAT-file version specified by the SaveFileVersion property. If the folder does not exist, the software creates it.

Example: "envSimData"

MAT-file version used to save environment data, specified as a string or character vector. When you set the SimulationStorageType property to "file", the software saves data generated by a Simulink environment in MAT-files in the version specified by SaveFileVersion, in the folder specified by the SaveSimulationDirectory property. For more information, see MAT-File Versions.

Example: Version="-v7.3"

Flag for using parallel simulation, specified as a logical. Setting this option to true configures the simulation to use parallel processing to simulate the environment, thereby enabling usage of multiple cores, processors, computer clusters or cloud resources to speed up simulation. To specify options for parallel simulation, use the ParallelizationOptions property.

Note that if you want to speed up deep neural network calculations (such as gradient computation, parameter update and prediction) using a local GPU you do not need to set UseParallel to true. Instead, when creating your actor or critic representation, use an rlRepresentationOptions object in which the UseDevice option is set to "gpu".

Using parallel computing or the GPU requires Parallel Computing Toolbox™ software. Using computer clusters or cloud resources additionally requires MATLAB® Parallel Server™.

For more information about training using multicore processors and GPUs, see Train Agents Using Parallel Computing and GPUs.

Example: UseParallel=true

Parallelization options to control parallel simulation, specified as a ParallelSimulation object. For more information about training using parallel computing, see Train Reinforcement Learning Agents.

The ParallelSimulation object has the following properties, which you can modify using dot notation after creating the rlSimulationOptions object.

Randomizer initialization for workers, specified as one the following:

  • –1 — Assign a unique random seed to each worker. The value of the seed is the worker ID.

  • –2 — Do not assign a random seed to the workers.

  • Vector — Manually specify the random seed for each work. The number of elements in the vector must match the number of workers.

Send model and workspace variables to parallel workers, specified as "on" or "off". When the option is "on", the host sends the variables used in models and defined in the base MATLAB workspace to the workers.

Additional files to attach to the parallel pool, specified as a string or string array.

Function to run on each worker before simulation starts, specified as a handle to a function having no input arguments. This function is run once per worker before simulation begins. Write this function to perform any processing that you need prior to simulation.

Function to run on each worker after simulation ends, specified as a handle to a function having no input arguments. You can write this function to clean up the workspace or perform other processing after simulation terminates.

Object Functions

simSimulate trained reinforcement learning agents within specified environment

Examples

collapse all

Open Live Script

Create an options set for simulating a reinforcement learning environment. Set the number of steps to simulate to 1000, and configure the options to run three simulations.

You can set the options using Name,Value pairs when you create the options set. Any options that you do not explicitly set have their default values.

simOpts = rlSimulationOptions(...
    MaxSteps=1000,...
    NumSimulations=3)
simOpts = 
  rlSimulationOptions with properties:

                   MaxSteps: 1000
             NumSimulations: 3
                StopOnError: "on"
      SimulationStorageType: "memory"
    SaveSimulationDirectory: "savedSims"
            SaveFileVersion: "-v7"
                UseParallel: 0
     ParallelizationOptions: [1x1 rl.option.ParallelSimulation]

Alternatively, create a default options set and use dot notation to change some of the values.

simOpts = rlSimulationOptions;
simOpts.MaxSteps = 1000;
simOpts.NumSimulations = 3;

simOpts
simOpts = 
  rlSimulationOptions with properties:

                   MaxSteps: 1000
             NumSimulations: 3
                StopOnError: "on"
      SimulationStorageType: "memory"
    SaveSimulationDirectory: "savedSims"
            SaveFileVersion: "-v7"
                UseParallel: 0
     ParallelizationOptions: [1x1 rl.option.ParallelSimulation]

Version History

Introduced in R2019a

See Also

Functions

Objects

Topics