PPO minibatch size for parallel training with variable number of steps
8 views (last 30 days)
Show older comments
I'm training a PPO Agent in sync parallelization mode.
Because of the nature of my environment, the number of steps is not the same for each episode, but can vary (sometimes wildly). Quoting from the reference for PPO Agent Options:
"When the agent is trained in parallel, ExperienceHorizon is ignored, and the whole episode is used to compute the gradients"
I don't fully understand the way in which the experiences collected during the episodes are divided into minibatches of the selected size, before the learning phase begins. Specifically, suppose that
- I have 2 parallel syncronous workers: the first one collects 30 experiences, the second one collects 70 experiences for a specifice couple of episodes
- The set of my minibatches has been set to 32
How are the experiences divided in minibatches?
As I understand it:
- Each worker sends its own experiences to the client --> So the client gaters 30 + 70 = 100 experiences
- These 100 experiences are divided into three groups of 32 (= minibatch size) each; 4 experiences are discarded, since 3 x 32 = 96
Is my reasoning correct? If so, I guess that if I want to limit the number of discarded experiences, the best way would be to decrease the minibatch size as much as possible (since I cannot know in advance the total number of experience available in every iteration)
0 Comments
Answers (1)
Emmanouil Tzorakoleftherakis
on 26 Feb 2024
No data will be discarded actually. As of R2023b, the 4 experiences that are left in your example form their own minibatch and are used that way. Note that this behavior may change in the future.
0 Comments
See Also
Categories
Find more on Training and Simulation in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!