rlTrainingOptions

R2026b

Options for training reinforcement learning agents

expand all in page

Description

Use an rlTrainingOptions object to specify options to train an agent within an environment. Training options include the maximum number of episodes to train, criteria for stopping training, criteria for saving agents, and options for using parallel computing. To train the agent using the specified options, pass this object to train.

For more information on training agents, see Train Reinforcement Learning Agents.

Creation

Syntax

trainOpts = rlTrainingOptions
trainOpts = rlTrainingOptions(PropertyName=Value)

Description

trainOpts = rlTrainingOptions returns the default options for training a reinforcement learning agent.

trainOpts = rlTrainingOptions(PropertyName=Value) creates the training option set trainOpts and sets its Properties using one or more name-value arguments.

example

Properties

expand all

Maximum number of episodes to train the agents, specified as a positive integer. Regardless of other criteria for termination, training terminates after MaxEpisodes.

Example: MaxEpisodes=1000

Maximum number of environment steps to run per episode, specified as a positive integer.

In general, you define the episode termination conditions in the environment. Specifically, when a termination condition is verified, the environment sets its isdone output to 1 or 2. Otherwise, the episodes terminates when, in that episode, the environment reaches a number of steps of at least MaxStepsPerEpisode.

Note

For Simulink® environments, the Stop Time value does not affect the duration of an episode, as it is temporarily set to Inf when the environment is called by the train, sim or runEpisode functions.

Example: MaxStepsPerEpisode=1000

Option to stop training when an error occurs during an episode, specified as "on" or "off". When this option is "off", errors are captured and returned in the SimulationInfo output of train, and training continues to the next episode.

Example: StopOnError="off"

Storage type for environment data, specified as "memory", "file", or "none". This option specifies the type of storage used for data generated during training or simulation by a Simulink environment. Specifically, the software saves anything that appears as the output of a sim (Simulink) command.

Note that this option does not affect (and is not affected by) any option to save agents during training specified within a training option object, or any data logged by a FileLogger or MonitorLogger object.

The default value is "memory", indicating that data is stored in an internal memory variable. When you set this option to "file", data is stored to disk, in MAT files in the directory specified by the SaveSimulationDirectory property, and using the MAT file version specified by the SaveFileVersion property. When you set this option to "none", simulation data is not stored.

You can use this option to prevent out-of-memory issues during training or simulation.

Example: SimulationStorageType="none"

Folder used to save environment data, specified as a string or character vector. The folder name can contain a full or relative path. When you set the SimulationStorageType property to "file", the software saves data generated during training or simulation by a Simulink environment in MAT files in this folder, using the MAT file version specified by the SaveFileVersion property. If the folder does not exist, the software creates it.

Example: SaveSimulationDirectory="envSimData"

MAT file version used to save environment data, specified as a string or character vector. When you set the SimulationStorageType property to "file", the software saves data generated by a Simulink environment in MAT files in the version specified by SaveFileVersion, in the folder specified by the SaveSimulationDirectory property. For more information, see MAT File Versions.

Example: Version="-v7.3"

Window length for averaging the scores, rewards, and number of steps for each agent, specified as a scalar or vector.

If the training environment contains a single agent, specify ScoreAveragingWindowLength as a scalar.

If the training environment is a multiagent environment, specify a scalar to apply the same window length to all agents.

To use a different window length for each agent, specify ScoreAveragingWindowLength as a vector. In this case, the order of the elements in the vector correspond to the order of the agents used during environment creation.

For options expressed in terms of averages, ScoreAveragingWindowLength is the number of episodes included in the average. For instance, if StopTrainingCriteria is "AverageReward", and StopTrainingValue is 500 for a given agent, then for that agent, training terminates when the average reward over the number of episodes specified in ScoreAveragingWindowLength equals or exceeds 500. For the other agents, training continues until:

  • All agents reach their stop criteria (the default is average steps over the last 5 episodes greater than 500).

  • The number of episodes reaches MaxEpisodes.

  • You stop training by clicking the Stop Training button in Reinforcement Learning Training Monitor or pressing Ctrl-C at the MATLAB® command line.

Example: ScoreAveragingWindowLength=10

Condition for terminating agents training, specified as one of the following strings:

  • "None" — Do not stop any agent training until the number of episodes reaches MaxEpisodes.

  • "AverageSteps" — Stop training an agent when its running average number of steps per episode equals or exceeds the critical value specified by the option StopTrainingValue. For each agent, the average number of steps per episode is computed using the window length specified in the "ScoreAveragingWindowLength" option.

  • "AverageReward" — Stop training for an agent when its running average reward equals or exceeds the critical value specified by the option StopTrainingValue. For each agent, the average number of steps per episode is computed using the window length specified in the "ScoreAveragingWindowLength" option.

  • "EpisodeReward" — Stop training for an agent when its reward in the current episode equals or exceeds the critical value specified by the option StopTrainingValue.

  • "GlobalStepCount" — Stop training for an agent when its total number of steps in all episodes (the total number of times the agent is invoked) equals or exceeds the critical value specified by the option StopTrainingValue.

  • "EpisodeCount" — Stop training for an agent when its number of training episodes equals or exceeds the critical value specified by the option StopTrainingValue.

  • "EvaluationStatistic" — Stop training for an agent when the statistic returned by the evaluator object used with train (if any) equals or exceeds the specified value specified by the option StopTrainingValue.

  • "Custom" — Stop training for an agent when its corresponding output flag returned by the custom function specified in StopTrainingValue is true.

Agents training termination behavior.

When not using a custom termination criteria, for a given agent, training ends when the termination condition specified by the StopTrainingCriteria option equals or exceeds the corresponding value specified in StopTrainingValue. When using a custom termination criteria, the training of a given agent stops when its corresponding flag element is true.

When an agent stops training, it stops updating its learning parameters but continues to simulate while the other agents keep training. For the other agents, the training continues until:

  • All agents reach their stop criteria (the default is average steps over the last 5 episodes greater than 500).

  • The number of episodes reaches maxEpisodes.

  • You stop training by clicking the Stop Training button in Reinforcement Learning Training Monitor or pressing Ctrl-C at the MATLAB command line.

Note

If you stop training by pressing Ctrl-C at the MATLAB command line, the training function does not return any value as output. Consequently, in this case, no object containing the training statistics will be available in the workspace.

For instance, if StopTrainingCriteria is "AverageReward" and StopTrainingValue is 100 for a given agent, then for that agent, training terminates when the average reward over the number of episodes specified in ScoreAveragingWindowLength equals or exceeds 100.

Example: StopTrainingCriteria="AverageReward"

This property is read-only.

Critical value of the training termination condition, specified as a scalar, vector or as a function name or handle.

Relationship to termination criteria.

The StopTrainingValue option is related to the StopTrainingCriteria option.

When not using a custom termination criteria, the following indications apply.

If the training environment contains a single agent, specify StopTrainingValue as a scalar. If the training environment is a multiagent environment, specify a scalar to apply the same termination value to all agents. To use a different termination value for each agent, specify StopTrainingValue as a vector. The order of the elements in the vector corresponds to the order of the agents used during environment creation.

When StopTrainingCriteria is set to "Custom", specify StopTrainingValue as a function name or handle. Your function must have one input and one output, as shown in the following signature.

flag = myTerminationFcn(trainingStats)

Here, trainingStats is a structure that contains the following fields, all described in the trainStats output argument of train.

  • EpisodeIndex

  • EpisodeReward

  • EpisodeSteps

  • AverageReward

  • TotalAgentSteps

  • EpisodeQ0

  • SimulationInfo

  • EvaluationStatistics

  • TrainingOptions

The function output flag is a vector with as many elements as the number of agents. The order of the elements in the vector corresponds to the order of the agents used during environment creation.

Agents training termination behavior.

When not using a custom termination criteria, for a given agent, training ends when the termination condition specified by the StopTrainingCriteria option equals or exceeds the corresponding value specified in StopTrainingValue. When using a custom termination criteria, the training of a given agent stops when its corresponding flag element is true.

When an agent stops training, it stops updating its learning parameters but continues to simulate while the other agents keep training. For the other agents, the training continues until:

  • All agents reach their stop criteria (the default is average steps over the last 5 episodes greater than 500).

  • The number of episodes reaches maxEpisodes.

  • You stop training by clicking the Stop Training button in Reinforcement Learning Training Monitor or pressing Ctrl-C at the MATLAB command line.

Note

If you stop training by pressing Ctrl-C at the MATLAB command line, the training function does not return any value as output. Consequently, in this case, no object containing the training statistics will be available in the workspace.

For instance, if StopTrainingCriteria is "AverageReward" and StopTrainingValue is 100 for a given agent, then for that agent, training terminates when the average reward over the number of episodes specified in ScoreAveragingWindowLength equals or exceeds 100.

Example: StopTrainingValue=100

Condition for saving agents during training, specified as one of the following strings:

  • "None" — Do not save any agents during training.

  • "EpisodeReward" — Save all the agents when an agent reward in the current episode equals or exceeds the critical value specified in SaveTrainingValue.

  • "AverageSteps" — Save the agents when the running average number of steps per episode equals or exceeds the critical value specified by the option SaveTrainingValue. The average is computed using the window specified in ScoreAveragingWindowLength.

  • "AverageReward" — Save the agents when the running average reward over all episodes equals or exceeds the critical value.

  • "GlobalStepCount" — Save the agents when the total number of steps in all episodes (the total number of times the agent is invoked) equals or exceeds the critical value.

  • "EpisodeCount" — Save the agents when the number of training episodes equals or exceeds the critical value.

  • "EpisodeFrequency" — Save the agents with a period specified in SaveAgentValue. For example, if SaveAgentCriteria is specified as "EpisodeFrequency" andSaveAgentValue is specified as 10, the agent is saved after every ten episodes.

  • "EvaluationStatistic" — Save the agents when the statistic returned by the evaluator object used with train (if any) equals or exceeds the specified value.

  • "Custom" — Save the agents when the custom function specified in SaveAgentValue returns true.

Set this option to store candidate agents that perform well according to the criteria you specify. When you set this option to a value other than "none", the software sets the SaveAgentValue option to 500. You can change that value to specify the condition for saving the agent.

For instance, suppose you want to store for further testing any agent that yields an episode reward that equals or exceeds 100. To do so, set SaveAgentCriteria to "EpisodeReward" and set the SaveAgentValue option to 100. When an episode reward equals or exceeds 100, train saves the corresponding agent (or all the corresponding agents for a multiagent environment) in a MAT file in the folder specified by the SaveAgentDirectory option. The MAT file is called AgentK.mat (or AgentsK.mat for a multiagent environment), where K is the number of the corresponding episode. The agents are stored within that MAT file as the saved_agent array. Note that the MAT file also includes the variable savedAgentResult which contains the training result information up to the corresponding episode.

Example: SaveAgentCriteria="EpisodeReward"

Critical value of the condition for saving agents, specified as a scalar, vector or as a function name or handle.

The SaveAgentValue option is related to the SaveAgentCriteria option.

When not using a custom save criteria, the following indications apply.

If the training environment contains a single agent, specify SaveAgentValue as a scalar. If the training environment is a multiagent environment, specify a scalar to apply the same saving value to each agent. To save the agents when one meets a particular criterion, specify SaveAgentValue as a vector. The order of the elements in the vector corresponds to the order of the agents used when creating the environment. When a criteria for saving an agent is met, all agents are saved in the same MAT file.

When SaveAgentCriteria is set to "Custom", specify SaveAgentValue as a function name or handle. Your function must have one input and one output, as shown in the following signature.

flag = mySaveFcn(trainingStats)

Here, trainingStats is a structure that contains the following fields, all described in the trainStats output argument of train.

  • EpisodeIndex

  • EpisodeReward

  • EpisodeSteps

  • AverageReward

  • TotalAgentSteps

  • EpisodeQ0

  • SimulationInfo

  • EvaluationStatistic

  • TrainingOptions

The function output flag is a vector with as many elements as the number of agents. The order of the elements in the vector corresponds to the order of the agents used during environment creation. The agents are saved when any element of flag is true.

When you specify a condition for saving candidate agents using SaveAgentCriteria, the software sets this value to 500. Change the value to specify the condition for saving the agent. See the SaveAgentCriteria option for more details.

Example: SaveAgentValue=100

Folder name for saved agents, specified as a string or character vector. The folder name can contain a full or relative path. When an episode occurs in which the conditions specified by the SaveAgentCriteria and SaveAgentValue options are satisfied, the software saves the current agent in a MAT file in this folder. If the folder does not exist, the training function creates it. When SaveAgentCriteria is "none", this option is ignored and no folder is created.

Example: SaveAgentDirectory=".\run1\Agents"

Option to display training progress at the command line, specified as the logical values false (0) or true (1). Set to true to write information from each training episode to the MATLAB command line during training.

Example: Verbose=true

Option to display training progress with Reinforcement Learning Training Monitor, specified as "training-progress" or "none". By default, calling train opens Reinforcement Learning Training Monitor, which graphically and numerically displays information about the training progress, such as the reward for each episode, average reward, number of episodes, and total number of steps. For more information, see train. To turn off this display, set this option to "none".

Example: Plots="none"

Option to use parallel training, specified as "off" (default), "on", or "auto".

Setting this option to "off" configures training to use a single process.

Setting this option to "on" configures training to use multiple processes (which can run on different cores, processors, computer clusters or cloud resources) to simulate the environment. This setting results in an error if a parallel pool cannot be resolved during training.

Setting this option to "auto" configures training to use multiple process if a parallel pool can be resolved. Otherwise, training runs on a single process.

Parallel training scales up the number of simulations with the environment, and can speed up the generation of data for learning. To specify options for parallel training, use the ParallelizationOptions property.

Using parallel computing or the GPU requires Parallel Computing Toolbox™ software. Using computer clusters or cloud resources additionally requires MATLAB Parallel Server™.

To speed up deep neural network calculations using a local GPU, you do not need to use this option. Instead, set the UseGPUForLearning agent option to "on" or "auto". For more information about training using multicore processors and GPUs, see Train Agents Using Parallel Computing and GPUs.

Example: UseParallel="off"

Options for parallel training, specified as a ParallelTraining object. For more information about training using parallel computing, see Train Agents Using Parallel Computing and GPUs.

The ParallelTraining object has the following properties, which you can modify using dot notation after creating the rlTrainingOptions object.

Parallel computing mode, specified as one of the following:

  • "sync" — Use parpool to run synchronous training on the available workers. In this case, each worker pauses execution until all workers are finished. The parallel pool client updates the actor and critic parameters based on the results from all the workers and sends the updated parameters to all workers. When training a PG agent using gradient-based parallelization Mode must be set to "sync".

  • "async" — Use parpool to run asynchronous training on the available workers. In this case, each worker sends its data back to the parallel pool client as soon as it finishes and then receives updated parameters from the client. The worker then continues with its task.

Example: Mode="async"

Randomizer initialization for workers, specified as one of the following:

  • –1 — Assign a unique random seed to each worker. The value of the seed is the worker ID.

  • –2 — Do not assign a random seed to the workers.

  • Vector — Manually specify the random seed for each worker. The number of elements in the vector must match the number of workers.

Example: WorkerRandomSeeds=[1 2 3 4]

Option to send model and workspace variables to parallel workers, specified as "on" or "off". When the option is "on", the client sends to the workers the variables defined in the base MATLAB workspace and used in the approximation models.

Example: TransferBaseWorkspaceVariables="off"

Additional files to attach to the parallel pool, specified as a string or string array.

Example: AttachedFiles="myInitFile.m"

Function to run before training starts, specified as a handle to a function having no input arguments. This function is run once per worker before training begins. Write this function to perform any processing that you need prior to training.

Example: SetupFcn=@mySetupFcn

Function to run after training ends, specified as a handle to a function having no input arguments. You can write this function to clean up the workspace or perform other processing after training terminates.

Example: CleanupFcn=@myCleanupFcn

Object Functions

trainTrain reinforcement learning agents within a specified environment

Examples

collapse all

Open Live Script

Create an options object for training a reinforcement learning agent. Set the maximum number of episodes and the maximum number of steps per episode to 1000. Configure the options to stop the training when the average reward equals or exceeds 480, and turn on both the command-line display and Reinforcement Learning Training Monitor for displaying training results. You can set the options using name-value pair arguments when you create the options object. Any options that you do not explicitly set have their default values.

trainOpts = rlTrainingOptions( ...
    MaxEpisodes=1000, ...
    MaxStepsPerEpisode=1000, ...
    StopTrainingCriteria="AverageReward", ...
    StopTrainingValue=480, ...
    Verbose=true, ...
    Plots="training-progress")
trainOpts = 
  rlTrainingOptions with properties:

                   MaxEpisodes: 1000
            MaxStepsPerEpisode: 1000
                   StopOnError: "on"
         SimulationStorageType: "memory"
       SaveSimulationDirectory: "savedSims"
               SaveFileVersion: "-v7"
    ScoreAveragingWindowLength: 5
          StopTrainingCriteria: "AverageReward"
             StopTrainingValue: 480
             SaveAgentCriteria: "none"
                SaveAgentValue: "none"
            SaveAgentDirectory: "savedAgents"
                       Verbose: 1
                         Plots: "training-progress"
                   UseParallel: "off"
        ParallelizationOptions: [1×1 rl.option.ParallelTraining]

Alternatively, create a default options object and use dot notation to change some of the values. 

trainOpts = rlTrainingOptions;
trainOpts.MaxEpisodes = 1000;
trainOpts.MaxStepsPerEpisode = 1000;
trainOpts.StopTrainingCriteria = "AverageReward";
trainOpts.StopTrainingValue = 480;
trainOpts.Verbose = true;
trainOpts.Plots = "training-progress";

trainOpts
trainOpts = 
  rlTrainingOptions with properties:

                   MaxEpisodes: 1000
            MaxStepsPerEpisode: 1000
                   StopOnError: "on"
         SimulationStorageType: "memory"
       SaveSimulationDirectory: "savedSims"
               SaveFileVersion: "-v7"
    ScoreAveragingWindowLength: 5
          StopTrainingCriteria: "AverageReward"
             StopTrainingValue: 480
             SaveAgentCriteria: "none"
                SaveAgentValue: "none"
            SaveAgentDirectory: "savedAgents"
                       Verbose: 1
                         Plots: "training-progress"
                   UseParallel: "off"
        ParallelizationOptions: [1×1 rl.option.ParallelTraining]

You can now use trainOpts as an input argument to the train command.

Open Live Script

To turn on parallel computing for training a reinforcement learning agent, set the UseParallel training option to "auto". This setting configures training (specifically the environment interaction) to use multiple process if a parallel pool is available, otherwise training runs on a single process. 

trainOpts = rlTrainingOptions(UseParallel="auto");

To configure your parallel training, configure the fields of the trainOpts.ParallelizationOptions. For example, specify to not assign a random seed to workers:

trainOpts.ParallelizationOptions.WorkerRandomSeeds = -2;
trainOpts.ParallelizationOptions
ans = 
  ParallelTraining with properties:

                              Mode: "sync"
                 WorkerRandomSeeds: -2
    TransferBaseWorkspaceVariables: "on"
                     AttachedFiles: []
                          SetupFcn: []
                        CleanupFcn: []

You can now use trainOpts as an input argument to the train command to perform training with parallel computing.

Open Live Script

To train an agent using the asynchronous advantage actor-critic (A3C) method, you must set the agent and parallel training options appropriately.

When creating the AC agent, set the NumStepsToLookAhead value to be greater than 1. Common values are 64 and 128.

agentOpts = rlACAgentOptions(NumStepsToLookAhead=64);

Use agentOpts when creating your agent. Alternatively, create your agent first and then modify its options, including the actor and critic options later using dot notation.

Configure the training algorithm to use asynchronous parallel training if a parallel pool is available. 

trainOpts = rlTrainingOptions(UseParallel="auto");
trainOpts.ParallelizationOptions.Mode = "async";

You can now use trainOpts to train your AC agent using the A3C method.

For an example on asynchronous advantage actor-critic agent training, see Train AC Agent to Balance Discrete Cart-Pole Using Parallel Computing.

Version History

Introduced in R2019a

expand all

See Also

Functions

Objects

Topics