Is there a fix to the error "A regression layer must not be preceded by a softmax layer." ?

I am trying to import my keras network into MATLAB using the "importKerasNetwork" function, but there is an issue.

In my Keras network I use a softmax activation function before my output. When I am importing into MATLAB, I specify the 'OutputLayerType' as 'regression' since that most appropriately fits my network type.

The issue I am currently having is that when I try to import this network, I get the error "A regression layer must not be preceded by a softmax layer.". Is there any work around to this problem?

Answer 1

[Divya Gaddipati]

The softmax layer normalizes the input to the layer such that its elements sum up to 1. Therefore, it is useful when you want the network to compute a probability of a classification problem, since it ensures that the sum of the scores over the classes is 1, as requested for a probability measure.

Currently, softmaxLayer cannot directly precede a regressionLayer.

A workaround would be to define a custom softmax layer and a custom regression output layer and replace the softmax layer and output layer from the exported network.

For instructions and an example on how to define a custom regression output layer, please view the documentation page linked here:

https://www.mathworks.com/help/deeplearning/ug/define-custom-regression-output-layer.html

For instructions and an example on how to define a custom layer (for softmax) with learnable parameters, please refer to the documentation page linked here:

https://www.mathworks.com/help/deeplearning/ug/define-custom-deep-learning-layer.html

Note: Both regression and softmax layers should be custom layers.

Firstly, import the Keras network layers along with their weights using the importKerasLayers function.

Remove the last two layers, i.e, the softmax and output layer from the network using removeLayers.

Add your custom layers to the network using addLayers.

Connect the added layers in the required manner using connectLayers.

Here’s a rough sketch of the code for your reference:

modelfile = 'your_keras_model.h5'; 
net = importKerasLayers(modelfile, 'ImportWeights', true); 
 
custom_softmax_layer = mySoftmaxLayer('custom_softmax'); 
custom_reg_layer = myRegressionLayer('reg_out');  
 
net = removeLayers(net, 'output'); 
net = removeLayers(net, 'softmax_activation'); 
  
net = addLayers(net, custom_softmax_layer); 
net = addLayers(net, custom_reg_layer); 
 
net = connectLayers(net, 'layer_before_softmax', 'custom_softmax'); 
net = connectLayers(net, 'custom_softmax', 'reg_out'); 

For more information on importKerasLayers, removeLayers, addLayers, connectLayers functions, please refer to the following links:

importKerasLayers - https://www.mathworks.com/help/deeplearning/ref/importkeraslayers.html#mw_acb96ce0-2c37-49d1-8867-c7af005c8c7e

removeLayers - https://www.mathworks.com/help/deeplearning/ref/removelayers.html

addLayers - https://www.mathworks.com/help/deeplearning/ref/addlayers.html

connectLayers - https://www.mathworks.com/help/deeplearning/ref/connectlayers.html

Hope this helps!

[Cyrus WaChong]

Wow, I am blown away by your thoroughness, thank you so much Divya! You are a bona fide life saver