Detect Issues When Training Model for Time Series Forecasting

R2026b

This topic describes various training options and techniques for fixing issues when training deep learning models for time series forecasting tasks.

Three common issues when training a deep learning model are:

• Overfitting — This occurs when the loss on the validation data is significantly higher than the loss on the training data.

• Vanishing Gradients — This occurs when the gradients become very small during training, causing the training and validation losses to decrease very slowly or plateau early.

• Exploding Gradients — This occurs when the gradients become very large during training, causing the training and validation losses to become unstable or increase suddenly.

It is difficult to detect these issues and to know how to fix them, but the Time Series Modeler app automatically detects these issues and then provides a list of general suggestions to help prevent these issues from arising when training a deep neural network. During training using the Time Series Modeler app, use the Training Diagnostics panel at the bottom of the app to see if your model has any issues.

During training, each diagnostic can be in one of these states:

• Not enough information available — This message appears during early training when the app is unable to determine if any issue is occurring. For overfitting, you will also see this message if you have not specified any validation data.

• No issues detected — This message means that the app found no signs of an issue.

• Issue detected — This message means the app has detected an issue.

Detect and Reduce Overfitting

Overfitting occurs when the loss on the validation data is significantly higher than the loss on the training data. The higher validation loss means that the model is overfitting to the training data and might not generalize well when you use it on unseen data. In this figure, you see that the training loss is still decreasing but the validation loss is increasing.

To detect overfitting, you can look at the training plot. You can also use the Time Series Modeler app to automatically detect overfitting. During training using the Time Series Modeler app, use the Training Diagnostics panel at the bottom of the app to see if your model is overfitting to the data. To check for overfitting, at each validation frequency, the app checks by what percentage the validation loss is higher than the average training loss. The software computes the average training loss across the last miniBatchSize iterations. If the validation loss is more than 10% higher than the average training loss for two or more validation checks in a row, then the model is overfitting.

Tip

By default, the app returns the model with the best validation loss. So, even if your network is overfitting at the end of training, the app can return a model at the point of no overfitting. The network returned by the app is equivalent to stopping training when the validation loss stops decreasing.

If your model is overfitting, the app suggests these fixes:

Fix	Where to Fix	Details
Increase the L2 regularization factor. For example, try increasing by a factor of 10.	In the Training Options section, select Show Advanced options. Then, expand Overfitting and change the L2Regularization training option.	Increasing L2 regularization penalizes large weights, encouraging simpler models that generalize better to new data. For more information, see L2Regularization.
Increase the dropout probability. For example, try increasing by 0.1.	Change the Dropout probability to a value greater than 0. This change is equivalent to adding dropout layers after each of the network blocks.	Adding dropout layers randomly deactivates neurons during training, which prevents the model from becoming overly reliant on specific pathways. For more information, see dropoutLayer
Use a decreasing learning rate schedule. For example, try using piecewise, polynomial, exponential, or cosine.	In the Training Options section, select Show Advanced options. Then, expand Learn Rate and change the LearnRateSchedule training option. The decreasing learn rate schedules are piecewise, polynomial, exponential, and cosine.	Using a decreasing learning rate schedule allows the model to make smaller, more precise weight updates as training progresses. This type of schedule helps the model to converge more smoothly and avoid overshooting optimal solutions. For more information, see LearnRateSchedule.
Decrease the number of hidden units. For example, try decreasing the number of hidden units by 10%.	Reduce the Hidden units value.	Reducing the number of learnable parameters limits the capacity of the model to memorize training data, encouraging the model to capture only the most essential patterns that generalize well to new data.
Use a smaller network architecture.	In the Model gallery, select a different network.	Changing the network architecture allows you to choose a model that can learn relevant patterns while avoiding excessive memorization of training data. For example, choose one of the Small networks, which have fewer learnable parameters.

Detect and Reduce Vanishing or Exploding Gradients

During neural network training, gradient-related issues can prevent a model from learning effectively and converging to a good solution. Two common problems are vanishing and exploding gradients, which affect how gradient information propagates through the network during backpropagation. Monitoring gradient behavior during training can help you to identify these issues early and make adjustments to improve training stability and performance.

During training using the Time Series Modeler app, use the Training Diagnostics panel at the bottom of the app to see if your model has vanishing or exploding gradients.

• Vanishing Gradients: The app detects vanishing gradients if the mean absolute gradient of the first learnable layer is two orders of magnitude smaller than that of the last learnable layer.

• Exploding Gradients: The app detects exploding gradients if the mean absolute gradients exceeded the max gradient threshold (10) for two consecutive or more steps or if the gradients contain NaN or Inf values.

If your model has exploding or vanishing gradients, then the app suggests these fixes:

Fix	Where to Fix	Details	Vanishing Gradients	Exploding Gradients
Normalize the input data.	In the Data Preprocessing section, select Normalize observations.	Unnormalized input data can cause large differences in feature scales, which can lead to unstable gradients during training.
Use a smaller network architecture.	In the Model gallery, select a different network.	Deep or wide networks can amplify gradient instability. Reduce the number of layers or the number of learnable parameters to make the network easier to train and improve gradient stability. For example, choose one of the Small networks, which have fewer learnable parameters.
Add normalization layers. For example, try adding layer normalization or batch normalization layers after each learnable layer.	To edit an existing network, in the Model Hyperparameters section, click Customize Network. You can also build a network from scratch by selecting Blank Network in the Model gallery.	Normalization layers help stabilize activations and gradients during training.
Use an adaptive solver. For example, try using Adam or RMSProp.	In the Training Options section, select Show Advanced options. Then, expand Solver and change the Solver training option to adam or rmsprop.	Adaptive solvers automatically adjust learning rates for individual parameters, which can help stabilize training when gradients vanish or explode. For more information, see solverName.
Add residual or skip connections to improve gradient flow. For example, connect earlier and later layer outputs using addition or concatenation layers.	To edit an existing network, in the Model Hyperparameters section, click Customize Network. You can also build a network from scratch by selecting Blank Network in the Model gallery.	Residual or skip connections improve gradient flow by providing shorter paths between earlier and later layers.
Check that the weight initialization matches the activation layer. For example, use Glorot initialization for tanh or sigmoid activation layers, and He initialization for ReLU activation layers.	To check the weight initialization, in the Model Hyperparameters section, click Customize Network. Select a learnable layer to see and change the weight initializer for that layer.	Inappropriate weight initialization can lead to unstable gradients. Use an initialization scheme that matches the activation function.
Decrease the learn rate. For example, try decreasing by a factor of 10.	In the Training options section, change the InitialLearnRate value.	A large learning rate can cause gradients to grow rapidly during training. Reduce the learning rate to make parameter updates more stable. For more information, see InitialLearnRate
Set the gradient threshold value to a smaller value. For example, try setting to a value of 1.	In the Training Options section, select Show Advanced options. Then, expand Overfitting and change the GradientThreshold training option.	Gradient clipping limits the magnitude of gradients during training and helps prevent instability. Set the gradient threshold to a smaller value to constrain large gradient updates. For more information, see GradientThreshold