Alternative to softmax function for Neural Network predicting fractions of a whole

Hi, I created a feed forward Regression Neural Network to predict variables which are fractions of a whole (i.e. they sum up to 1). In order to have the network fullfil this criterion perfectly, I am using the softmax transfer function. Unfortunately, I realize that the network predicts smaller fractions very poorly, and I think this is due to the fact that the softmax transfer function normalizes my target fractions by dividing exponent of the fractions minus the largest fraction by its sum (exp(n-nmax)/sum(exp(n-nmax))), which results in much larger values for very small fractions. It wouldn't have to do that, since my fractions are already between 0 and 1. Can I change that somehow in the softmax transfer function, or is there an alternative to it that doesn't do this normailzation?

[Ajay Pattassery]

I understand that you are implementing a multinomial logistic regression by using Softmax layer.

I did not understand why having a larger value as output (still between 0 and 1) for a very small input value from a Softmax layer is a problem. Could you elaborate on that? If possible, provide some information on what you are trying to achieve with the neural network?

[F S]

Of course! Thank you for your comment. Firstly I have noticed that this is happening from simply observing the prediction of my data. Secondly I digged a bit deeper into the code that is doing the regularization in the softmax function, and I found this, where n is my input targets and a the normalized output (this is in the 'apply' function in the +softmax folder from 2017 Version. In the 2019 Version it's still the same, but looks somewhat more complicated because there was some paralell computing code built into that function, from what I can judge):

nmax = max(n,[],1);

n = bsxfun(@minus,n,nmax);

numer = exp(n);

denom = sum(numer,1);

denom(denom == 0) = 1;

a = bsxfun(@rdivide,numer,denom);

What I realize is happening, is that if one fraction is much larger than the others, say 80% and the others split the remaining 20% unequally, e.g. one fraction is exponetially smaller, say 0.1%, the neural network cannot really catch that, because after the normalization from this code, the 19.9% and 0.1% fraction would be much closer together between 0 and 1 than what they actually are because of the exponent (you can see by simply creating a matrix n of fractions and run it eg n=[80 18.9 1 0.1; 70 29 0.9 0.1; 0.01 0.989 0.001 99, 25 25 25 25]'), in respect to the larger fraction. I am not really sure that that is the problem, since I do not at all see the big picture of the neural netoworks toolbox. I would be very glad if somebody could clarify, and if my instinct is right, suggest me a better solution where that problem didn't arise.

[F S]

I also don't understand why it needs that exponet there, as since we are dealing with fractions, they are kind of between 0 and 1 already anyway, no?

[Ajay Pattassery]

Hello,

I did not understand why you want to have the sum of outputs of your network to be one. If your goal is to do Multinomial logistic regression as you have confirmed above (Where you will have more than two output classes and you are classifying your input to one of these classes) the highest probability value corresponds to the class where the input belongs to.

On the other hand, if you want to have the output that adds up to one, you could have custom layers like (say X = [Xi, Xj, Xk] is your input variable to the layer and Y the output)

Yi = Xi/(Xi + Xj + Xk)

Yj = Xj/(Xi + Xj + Xk)

Similarly Yk can be calculated

Please refer the following link for creating custom layers.

https://www.mathworks.com/help/deeplearning/ug/define-custom-deep-learning-layer.html

[F S]

It is not multinomial logistic regression in a classical sence, as there are no classes to be predicted, although it is mathematically equal to it as far as I can judge. It is fractions of constituents of a mass, of which I know the weight and weight distribution, and I now the density of the constituents. So now I want to predict how much of which constituent I have, knowing that their weights ALWAYS sum up to the total weight. It can't be physically solved unfortunately, because there is several combinations that can lead to the same total weight, but the weight distribution holds information that can help solve the problem.

If there is a neural network or other method that can solve such a problem other than softmax, I would be happy if somebody can point that out to me.

I am also happy to send you more information in a private email, containing some data that I cannot disclose publicly. My licence and support subscription are up to date.

[Ajay Pattassery]

Did you tried creating a custom layer which can force your output to one like the one I mentioned above instead of having a softmax layer.

Please refer the following link for creating custom layers.

https://www.mathworks.com/help/deeplearning/ug/define-custom-deep-learning-layer.html