CNN overfits when trained too long on low dataset

Hi!

As you can seen below I have an overfitting problem. I am facing this problem because I have a very small dataset: 3 classes of each 20 1D images. Therefore, I am using a very simple architecture so the model will be robust, and cannot be trained 'too well' to the training data. However, it seems if I train it for too long, the model will eventually still be specific to the training data, and not robust. However, at around 0.5 iterations the model works quite okay and robustly. There is literally only one convolutional layer containing 8 filters in my architecture.

At 0.5 iterations the model seems to be robust. A solution would be to stop the training here. However, this feels wrong, is there another way? Or is this the way to go...

Answer 1

[Antoni Woss]

As you mentioned, this looks like an overfitting problem. Terminating the training early is one way that you can avoid overfitting. There are also training options that you can specify to automate this.

For example, see the validation section on the following documentation page:

https://www.mathworks.com/help/deeplearning/ref/trainingoptions.html#d124e166764

In particular, setting the option OutputNetwork as best-validation-loss will return the network at the iteration with the minimal validation loss. From your plot, this looks to be around 0.5 * 10^4 (although there is a spike around 1.5 * 10^4 that looks sporadically lower). You might also want to set the ValidationPatience. This specifies the number of times that the loss on the validation set can be larger than or equal to the previously smallest loss before network training stops. This should help terminate the training before the upward drift in the validation loss. You may also want to increasing the spacing between validation loss evaluation to remove the oscillations and help isolate the upward drift. This can be done by setting the ValidationFrequency value, which is the number of iterations between evaluations of validation metrics.