How do I clear the Cache Storage when training a Neural Network with a Custom Training Loop?
7 views (last 30 days)
Show older comments
Hello everybody,
I was playing around with the code found on here:
First I trained the net without changing anything in the code but the number of epochs to 300 instead of 3000. I got this result:![](https://www.mathworks.com/matlabcentral/answers/uploaded_files/1411219/image.png)
![](https://www.mathworks.com/matlabcentral/answers/uploaded_files/1411219/image.png)
Everything like you would expect.
After that I modified the loss-function in the function "modelLoss":
I changed "loss = lossF + lossU;" to "loss = lossU;".
So I basically trained the net only with the Initial (U = sin(-pi*x)) and Boundary (U = 0) Conditions and got the following result:
![](https://www.mathworks.com/matlabcentral/answers/uploaded_files/1411224/image.png)
That made me realize MATLAB has to store the results from the training before and used it here, otherwise the net could not produce a output that close to the true solution wihtout even knowing the PDE.
I tried to delete as much as possible between the two different trainings, so I tried "clear all" instead of "clear" and added "clearCache(accfun)" after the line "accfun = dlaccelerate(@modelLoss);". It had an inpact, but the output was still depending on the trainings I did beforehand.
This is the output after training only with Initial and Boundary Conditions when I freshly started MATLAB:![](https://www.mathworks.com/matlabcentral/answers/uploaded_files/1411229/image.png)
![](https://www.mathworks.com/matlabcentral/answers/uploaded_files/1411229/image.png)
As you would expect just nonsense.
And this is the output after training only with Initial and Boundary Conditions when I trained with the "default"-loss-function before. The code included "clear all" and "clearCache(accfun)".
![](https://www.mathworks.com/matlabcentral/answers/uploaded_files/1411234/image.png)
Still not a good solution but much better than the one after just starting MATLAB.
This made me conclude "clear all" and "clearCache(accfun)" is not enough to reset everything.
I researched a lot for a solution, but I wasn't successful so I would be very thankful if someone here has an idea how to approach this!
Thanks!
Julian
2 Comments
Answers (1)
Himanshu
on 28 Aug 2023
Hello Julian,
I understand that you are facing an issue where the output of your neural network is influenced by previous training sessions, even after attempting to clear variables and cache.
The error likely occurred due to the internal states and cached information from the underlying framework affecting the network's behaviour.
You can resolve the issue by using the following steps:
- Set the random seed before training each time to ensure consistent random initialization of network parameters.
- After each training session, explicitly reinitialize the model's parameters before starting the next training session.
- If you are using an optimizer like Adam, consider reinitializing the optimizer's internal states before each training session.
- Instead of "clear all," consider using "clearvars" to clear variables while retaining necessary configurations and paths.
- In cases where the internal state of the framework might be influencing training, consider restarting MATLAB itself.
You can refer to the below documentation to learn more about Random Number Generation and the "clearvars" function in MATLAB.
0 Comments
See Also
Categories
Find more on Image Data Workflows in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!