Clear Filters
Clear Filters

运用MAPPO训练后​进行验证,发现保存的​所有Agent.ma​t文件的的验证rew​ard回合奖励都是一​个数,而且效果非常差

11 views (last 30 days)
郭欣
郭欣 on 15 Aug 2023
Commented: 郭欣 on 21 Aug 2023
我运用MAPPO强化学习做了500个episode的训练,训练结果从奖励函数来看是收敛了,训练结果如下:
训练加载的环境相对于验证环境复杂一点,但是我验证时发现所有保存的Agent输出的回合奖励非常差,而且所有episode保存的Agent都是一样的结果,感觉网络没有被训练收敛或者正常保存,我的文件写法如下:
trainOpts = rlMultiAgentTrainingOptions(...
"AgentGroups","auto",...%%{[1,2]}
"LearningStrategy","decentralized",...
"MaxEpisodes",500,...
"MaxStepsPerEpisode",Tf/Ts,...
"ScoreAveragingWindowLength",10,...
"StopTrainingCriteria","AverageReward",...
"StopTrainingValue",99999990,...
'Verbose',true,...%在命令行输出训练进度
SaveAgentCriteria="Averagereward",...
SaveAgentValue=-inf);

Answers (1)

Sarthak
Sarthak on 21 Aug 2023
Hi 郭欣,
As per my understanding, there could be a few potential reasons for the poor performance you’re observing:
  • As you mentioned that the training environment is slightly more complex than the validation environment, it is important to have similar complexities between the training and validation to ensure that the trained agents can be generalized.
  • Also, to improve the effectiveness of MAPPO, you may want to increase the amount of training data which covers a wide range of scenarios.
  • If not, already you may also have to normalize the values to stabilize value learning
  • Analysis and refinement of the reward function might also be necessary.
However, it is difficult to pinpoint the exact reason why you are experiencing poor results without looking at your implementation, you can implement a thorough debugging process to track the training progress and identify potential issues.
You can also refer to the following documentation for a better understanding of how to train your reinforcement learning agents
  1 Comment
郭欣
郭欣 on 21 Aug 2023
Thank you very much for your answer, in fact, the most confusing thing for me is that the 500 episodes of gent I trained and saved turned out to be exactly the same in the new verification environment when I was verifying, even if I use the trained data when I verified. What is the reason for this?

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!