Custom deep learning loop take more memory than using trainNetwork()?

1 view (last 30 days)
Hi,
I followed the instructions from the link below to create a custom training loop by using a U-Net architecture.
By the same network architecture and with same "multi-gpu" setting (I have 2 RTX 2060 GPU), I found that I can only take 4 minibatch size at best in the custom training loop, while 16 minibarch size at best by using the built-in trainNetwork() function.
Is this a normal phenomenon that custom loop training will take more gpu memory than trainNetwork()?
Thanks!

Accepted Answer

Shashank Gupta
Shashank Gupta on 28 Oct 2020
Yes, it is an expected behaviour, the custom loop does take some extra amount of memory while the existing function trainNetwork is very optimised. More custom loop more inefficiency and thus more GPU memory usage. Neverthless, you can optimise the custom training loop but even then we can't be fully sure that it is as much optimised as trainNetwork.
I hope this clear some of your confusion.
  3 Comments
Shashank Gupta
Shashank Gupta on 30 Oct 2020
Hey Qiao,
Have a look at this Link, this might enable you to use parallel capabilities in the custom training loop.
Currently, there is no specific reference that talks about the optimisation of custom loop specifically because it is hard to generalise anything and come up with a documented reference. Generally these jobs are really subjective, depends on what sort of things you want to implement, Nevertheless, some suggestions, look for dlarray capable function for quick computing, Try using more MATLAB function rather than implementing your own. try to use as less code as necessary.

Sign in to comment.

More Answers (0)

Products


Release

R2020b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!