What is the best activation function to get action between 0 and 1 in DDPG network?

I am using DDPG network to run a control algorithm which has inputs (actions of RL agent, 23 in total) varying between 0 and 1. I an defining this using rlNumericSpec

actInfo = rlNumericSpec([numAct 1],'LowerLimit',0,'UpperLimit', 1);

Then I am using tanhLayer in the actor network (similar to bipedal robot example) and then using

actorOptions = rlRepresentationOptions('Optimizer','adam','LearnRate',1e-4, 'GradientThreshold',1,'L2RegularizationFactor',1e-5);
actor = rlRepresentation(actorNetwork,env.getObservationInfo,env.getActionInfo, 'Observation',{'observation'},  'Action',{'ActorTanh1'},actorOptions);

But i feel that the model is only taking the extreme options ie mostly 0 and 1.

Will it be better to use a sigmoid function to get better action estimates?

Answer 1

[Emmanouil Tzorakoleftherakis]

Hello,

With DDPG, a common thing to do in the final 3 layers of the actor is to use a fully connected layer, a tanh layer and a scaling layer. Tanh will get the ouput of that layer between -1 and 1 and then you can use the scaling layer to scale/shift values as needed based on the specifications of the actuator in your problem.

It seems the problem here is due to noise that is being added during training with DDPG to allow sufficient exploration (for example see step 1 here). The default noise options have a pretty high variance, so when this is added to the output of the tanh layer, it ends up outside the [0, 1] range and is being clipped. This is why you are only getting the two extremes.

Try adjusting the DDPG noise options, and particularly the variance (make it smaller, e.g. <=0.1). Also, see here for some best practices when choosing noise parameters.

Hope that helps

[Sayak Mukherjee]

Thanks a lot for your reply. Is it feasible to use a sigmoid function in place of tanh layer?

Also my action outputs from tanh layer vary between -1 and 1 without using the scaling layer even though I have defined

actInfo = rlNumericSpec([numAct 1],'LowerLimit',0,'UpperLimit', 1);

does the LowerLimit and UpperLimit have no impact in the output of tanh layer?

Will using a scaling layer sort out this issue?

Also I was using a variance of 0.3

Thanks again for your help

[Emmanouil Tzorakoleftherakis]

As I mentioned, variance of 0.3 is very high for a [0 1] range. Reduce that to <=0.1 and that should help with the out of range issues. Also, I believe noise is added after applying uppr and lower limits, so during training you may still see some violations, but if you do inference after training stops, the upper and lower limits should be respected. The sigmoid layer would have the same problem unless you fix the noise variance - this is how DDPG works.

[Sayak Mukherjee]

So the final two layers of my actor network are

fullyConnectedLayer(numAct, 'Name', 'ActorFC3',  'Weights',2*5e-3*(rand(numAct,actorLayerSizes(2))-0.5), ... 
'Bias',2*5e-5*(rand(numAct,1)-0.5))                       
tanhLayer('Name','ActorTanh1')

And I am defining

actInfo = rlNumericSpec([numAct 1],'LowerLimit',0,'UpperLimit', 1);
actor = rlRepresentation(actorNetwork,env.getObservationInfo,env.getActionInfo,...
    'Observation',{'observation'}, 'Action',{'ActorTanh1'},actorOptions);

But when I see actions using

experience = sim(env,saved_agent,simOptions);

I get the bounds between -1 and 1

Is there something that I am doing wrong?

[Emmanouil Tzorakoleftherakis]

Hard to say without a reproduction model, but looks like the third argument in 'rlRepresentation' is env.getActionInfo and you define actInfo separately as well. Make sure the two match

[Sayak Mukherjee]

I should have been clearer

actInfo = rlNumericSpec([numAct 1],'LowerLimit',0,'UpperLimit', 1);
actInfo.Name = 'STIM'
env = rlSimulinkEnv(mdl,blk,obsInfo,actInfo);

And then I am defining the actornetwork

actorNetwork = [
    imageInputLayer([numObs 1 1],'Normalization','none','Name','observation')
    fullyConnectedLayer(actorLayerSizes(1), 'Name', 'ActorFC1', ...
            'Weights',2/sqrt(numObs)*(rand(actorLayerSizes(1),numObs)-0.5), ... 
            'Bias',2/sqrt(numObs)*(rand(actorLayerSizes(1),1)-0.5))
    reluLayer('Name', 'ActorRelu1')
    fullyConnectedLayer(actorLayerSizes(2), 'Name', 'ActorFC2', ... 
            'Weights',2/sqrt(actorLayerSizes(1))*(rand(actorLayerSizes(2),actorLayerSizes(1))-0.5), ... 
            'Bias',2/sqrt(actorLayerSizes(1))*(rand(actorLayerSizes(2),1)-0.5))
    reluLayer('Name', 'ActorRelu2')
    fullyConnectedLayer(numAct, 'Name', 'ActorFC3', ... 
            'Weights',2*5e-3*(rand(numAct,actorLayerSizes(2))-0.5), ... 
            'Bias',2*5e-5*(rand(numAct,1)-0.5))                       
    tanhLayer('Name','ActorTanh1')
    ];

% Create actor representation
actorOptions = rlRepresentationOptions('Optimizer','adam','LearnRate',1e-4, ...
                                       'GradientThreshold',1,'L2RegularizationFactor',1e-5);
actor = rlRepresentation(actorNetwork,env.getObservationInfo,env.getActionInfo, ... 
                         'Observation',{'observation'}, ...
                         'Action',{'ActorTanh1'},actorOptions);

So my question is do I need a separate scaling layer after tanh layer even though I have defined lowerlimit as 0 in actInfo. My actions fluctuated between -1 and 1 with this architecture. If I use sigmoid function then I get the action between 0 and 1.

[Emmanouil Tzorakoleftherakis]

If your desired range is [0 1], it is indeed better to use a sigmoid as a final layer (a scaling layer would work as well). That would solve your problem.

What's not clear to me is why the lower limit is not respected when you are using 'sim' with the tanh layer since it seems like you have it set up properly.

Can you verify which release you are using? I was able to reproduce this behavior in 20b, and it may be a bug. It seems you are using Simulink for the environment so a workaround is to use the 'Play' button instead of 'sim'.

[Sayak Mukherjee]

I am using 2020a right now since our servers are set up with 2020a, It looks like sigmoid function is not available for 2020a.

[Emmanouil Tzorakoleftherakis]

I would try the scaling layer. Were you able to get the expected actions with the 'Play' button?

[Sayak Mukherjee]

No

I am not getting expected results with play, still action is -ve in some cases.

Is the issue with version?

[Emmanouil Tzorakoleftherakis]

No, I think this is a bug. I would contact technical support at this point to put you in touch with the development team and see if they can provide a workaround. You will most likely need to provide a reproduction model as well.

[Sayak Mukherjee]

ok

thanks

[awcii]

@Sayak Mukherjee What about your problem ? Did you solve it ?