Loading and saving variables when using parallel toolbox

3 views (last 30 days)
I am solving a systme of equations using fsolve, where in order to compute those equations I need to solve a recursive problem. The recursive problem depends on the inputs and takes a long time to compute. However, one way to speed it up is to use a better guess. Unfortunately, it is not easy to come up with this guess. So what I am doing now is to come up with some "reasonable" guess the first time and then each time the solver calls the function I save the final result of the recursive problem and load it as the guess for the next time the function is called by fsolve. Basicaly, the code I am using has the following structure
output = my_fun(param)
if exist('guess_Amat.mat','file')==2
load guess_Amat
Amat = my_fun_2(param,guess_Amat) % Compute Amat (a recursive problem where guess_Amat is the initial guess)
% Compute other things using Amat including output variable
% saving Amat
guess_Amat = Amat;
save guess_Amat guess_Amat
This works well when I am using a single core. However, when I set in the options
the code breaks since (as I understand) different workers are trying to save/load the Amat.mat file at the same time.
My question: Is there a way to adjust the code so that I can "save" A_mat each time and then "load" it for the next evaluation of my_fun to speed up the recursive protion of the code. When using the above code with a single worker loading/saving Amat saves me about 2/3 of the time so I am wondering if there is a way to adapt this appraoch when using the optimoptions('fsolve','UseParallel',true).

Accepted Answer

Peter O
Peter O on 1 Sep 2020
Edited: Peter O on 1 Sep 2020
File access with parallel workers can be tricky, and in the past I've been successful by creating a separate temp directory per worker. If you don't need file access explicitly, and it sounds like you don't, consider using a persistent variable? If the function runs locally on a worker, I believe they'll each hold their own value here. To clear the persistent variable on workers, you need to call pctRunOnAll clear persistent
Since fsolve uses parallel workers to compute the gradient, I'm not sure if there are restrictions on when/how the parallel workers are used (don't have the toolbox, so can't test). Potential code could look like:
function output = my_fun(param)
persistent A_guess
if isempty(A_guess)
A_guess = zeros(1000,1); %Initialize
Amat = my_fun_2(param,A_guess) % Compute Amat (a recursive problem where guess_Amat is the initial guess)
% Compute other things using Amat including output variable
A_guess = Amat; % Update persistent
  1 Comment
Michal Szkup
Michal Szkup on 1 Sep 2020
That works just fine! The only issue I had was clearing the persistent variable; somehow pctRunOnAll clear persistent did not work. In the end, I ended up just clearing the function my_fun at that dealt with the problem.

Sign in to comment.

More Answers (1)

Matt J
Matt J on 1 Sep 2020
Edited: Matt J on 1 Sep 2020
This might work. Basically, I am saying that you must repeatedly call fsolve in a loop having it do 1 iteration at a time.You generate guess_Amat once before the loop and at the end of each loop iteration.
function main()
guess_Amat=___________; %supply initial guess_Amat
for i=1:N
[x0,fval]=fsolve( @(param) my_fun(param, guess_Amat) , x0,options);
guess_Amat = my_fun_2(x0,guess_Amat);
function fval = my_fun(param, Amat)
Michal Szkup
Michal Szkup on 1 Sep 2020
Thanks Matt! Peter's answer was just what I was looking for, but your trick is really neat and surely helpful for many other problems.

Sign in to comment.


Find more on Problem-Based Optimization Setup in Help Center and File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!