Definition of transitions and rewards in a MDP environment for a reinforcement learning problem.
2 views (last 30 days)
Show older comments
Good morning,
I have an environment with 50 states and 5 actions. From state 1, I transition to states 2 to 25 with action 1; from state 1, I transition to states 26 to 49 with action 2; from states 2 to 25, I transition to state 50 with action 3; from states 26 to 49, I transition to state 50 with action 4. I define the matrices MDP.T as follows:
MDP.T(1,2:25,1) = 1/24;
MDP.T(1,26:49,2) = 1/24;
MDP.T(2:25,50,3) = 1;
MDP.T(26:49,50,4) = 1;
MDP.T(50,50,5) = 0;
The code works only if I consider also these other transitions, which are not feasible in my example.
MDP.T(1,1,3) = 1;
MDP.T(1,1,4) = 1;
MDP.T(2:25,2:25,1) = 1/24;
MDP.T(2:25,2:25,2) = 1/24;
MDP.T(2:25,2:25,4) = 1/24;
MDP.T(2:25,2:25,5) = 1/24;
MDP.T(26:49,26:49,1) = 1/24;
MDP.T(26:49,26:49,2) = 1/24;
MDP.T(26:49,26:49,3) = 1/24;
MDP.T(26:49,26:49,5) = 1/24;
May you help me to understand how to define only feabile transitions?
Thank you in advance
3 Comments
Accepted Answer
arushi
on 18 Jan 2024
Hi Bianca,
As per my understanding, it seems that you want to define a transition matrix for an MDP with 50 states and 5 actions, where certain transitions are possible based on the current state and the action taken. The code does not work if you do not consider the other transitions. In some MDP implementations, especially when using certain toolboxes or libraries, it's necessary to define transitions for all state-action pairs, even if some actions are not applicable in certain states. This is typically done to avoid undefined behavior in the MDP solver or to comply with the requirements of the specific MDP framework being used.
To handle this in the code, you can assign a self-transition (transition back to the same state) with a probability of 1 for actions that are not applicable in each state. This effectively means that if you take an infeasible action in a state, you stay in the same state with certainty.
Hope this helps.
0 Comments
More Answers (0)
See Also
Categories
Find more on AI for Wireless in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!