Problem updating revolute joint trajectories during reinforcement learning in MATLAB Simscape

I am looking to use a simscape model as an environment to conduct reinfrocement learning (RL). The RL agent needs to change the kinematics of a robot once every 3 seconds. The trajectory being changed is a complicated spline that governs the motion of the revolute joint over those 3 seconds. The motion governs the trajectory of a swimming robot so its motion is cyclical and returns to the same state at which it started at the end of every cycle. The learning looks to adjust the shape, and magnitude of the signal by controlling command points of a spline at every learning step. Each learning step is one full cycle of the motor trajectory which in this case is 3 seconds.

I have a matlab function that given a set of actions (from the reinforcement learning agent) can output a trajectory of points in a timeseries or any form necessary. Unfortunately, you cannot define timeseries objects in simulink as you get this error:

" The 'timeseries' class does not support code generation. "

Additionally, I know using the "from workspace" block allows for the import of timeseries objects into simulink/simscape however, the agent's action selection is being done during the simulation itself so I am unable to know what actions the agent will take until the learning/simulation are well underway

Is there a way in which I can update the trajectory going to a revolute joint every learning step such that it can follow a unique trajectory each learning interval? Of course i'd also like a solution that is relatively computationally inexpensive as I will need to run the learning for ~100,000 episodes.

And just a little bit more information about my problem: I have built a full dynamic model of a swimming sea lion robot which functions very well. It is hooked into a learning loop very similar to the ones used in these two examples:

https://www.mathworks.com/help/reinforcement-learning/ug/quadruped-robot-locomotion-using-ddpg-gent.html

https://www.mathworks.com/help/reinforcement-learning/ug/train-biped-robot-to-walk-using-reinforcement-learning-agents.html

but unfortunately these do not use trajectories as their actions but rather joint torques. I do not have this ability as the robot that I am modeling can only be acctuated using position controled servomotors so it requires a trajectory rather than a scalar joint torque.

Thank you for any help in advance!

Answer 1

[Anthony]

For anyone curious I found a solution for my particular problem. I couldn't figure out the time series aspect so I made at matlab function that generates a vector of the trajectory that I am looking to follow. I then have a counter that counts each simulation time step and pulls the corresponding index of that vector and feeds it to the revolute joint. This solves the problem though I am unsure of how much it will impact simulation speed. Make sure to reset that the counter everytime the length of the vector is reached (which in this cases is the number of simulation steps per learning steps).