Gradient clipping with custom feed-forward net
16 views (last 30 days)
Christoph Aistleitner on 28 Jul 2021
Everytime I am training my custom feed-forward net with 2 inputs and one output( timeseries) with the train(net,....) function:
after ~10 training epochs the value of the gradient reaches the prestet value and the training stops.
Changing the networks architecture is not an option in my case.
Is there a way to implement "gradient clipping" with a feed-forward net?
Or is there any other workaround for the "exploding gradient"?
More Answers (1)
Artem Lenskiy on 4 Dec 2022 at 13:29
Please check this link that illustrates several examples on how to implement training options that you would usually define via trainingOptions() and use with trainNetwork() but for customs loops. Here is an L2 clipping example given in the link above
function gradients = thresholdL2Norm(gradients,gradientThreshold)
gradientNorm = sqrt(sum(gradients(:).^2));
if gradientNorm > gradientThreshold
gradients = gradients * (gradientThreshold / gradientNorm);
You might also find this link useful https://au.mathworks.com/help/deeplearning/ug/detect-vanishing-gradients-in-deep-neural-networks.html that discuss detection of vanishing gradients in deep neural networks.