Issue with batch normalization layer of saved CNN

When loading a previously trained CNN, I got an issue with the batch normalization layers. When looking into the loaded CNN model the trainable mean and variance are empty.

Name: 'batchnorm_1'

TrainedMean: []

TrainedVariance: []

So the checkpoint doesn't seem to save these parameters. Are there any workarounds for this issue? I am using Matlab R2018b.

[Wes Baldwin]

Doesn't this mean using checkpoints on networks with a batchnorm layer is useless??? Kinda a big deal for long training!!! You could potentially lose days or weeks of training with no option but to start from the beginning.

Answer 1

[Vivek Singh]

We were able to reproduce the issue. We will inform you once the issue is fixed.

Since TrainedMean and TrainedVariance are calculated after the training is finished, therefore as a workaround you can use the below mentioned codes to explicitly save and load the Model.

%To save model with name "demoModel", assuming your network is in "net" 
save('demoModel','net') 

%To load model to variable net1
net1=load('demoModel.mat','net');
net1.net.Layers(n).TrainedMean %where n is the batch normalization layer

[Kjetil Kveim]

Perfect, thanks for the workaround :)

[ytzhak goussha]

Hello,

Was this issue fixed? I have encounterd the same problem when I stoped training and tried to predict with the saved checkpoint.

[ramin nateghi]

Hello,

I also faced this problem. When a model is saved by "save" function, it is ok and the model contains all information (TrainedMean and TrainedVariance) of the trained batch normalization layers. But, when the model is saved by the"checkpoint" during training, both of the TrainedMean and TrainedVariance params became empty. This is a bug for "checkpoint".

[Yi Wei]

Does it mean the saving process will cost too much time if a network contains normalization layers and the training data volume is large(e.g.8T)?

Answer 2

[Sam Leeney]

For anyone else stuck, there is a fix here; https://uk.mathworks.com/matlabcentral/answers/423588-how-to-classify-with-dag-network-from-checkpoint