cplex object params not transferred to parfor loop

1 view (last 30 days)
I am trying to solve the batch of CPLEX problems in parallel using parfor loop. I'm not very fluent in Matlab data types, but CPLEX problems seem to be stored as handle class objects with two structs: Model and Param.
Now, say I have an array with the paths to my .lp problems and I am reading them into cell array:
subp_array = {}; % array to store problems
nSubp = 1; % for indexing problems
% read models into subp_array
for file = filepaths
prob = Cplex('uc');
prob.readModel(file{1});
prob.Param.threads.Cur = 1;
subp_array{nSubp} = prob;
nSubp = nSubp+1;
end
The param threads.Cur is particularly important, because it indicates how many threads can be utilized for a single CPLEX problem (and with anything other than 1, it does not bring any time benefit). Now when I run the loop to solve the problems and I print the param value:
parfor subp_index = 1:length(subp_array)
prob = subp_array{subp_index};
disp(prob.Param.threads.Cur);
% solve the problem
prob.solve();
end
They all happen to be 0 (default value). If I run the for loop instead, it keeps the value of 1.
What happens here? I'm actually trying to catch more differences in my original code as parfor returns completely different results than for - but this one could be an indication for me where the differences come from.
Thanks

Accepted Answer

Edric Ellis
Edric Ellis on 31 Jul 2020
The most likely cause of the problem here is that the output of Cplex cannot be saved to a file and then loaded again. Transfer of data to the workers in a parallel pool is equivalent to calling save and then load (but the data isn't stored on disk at any point), and some data types in MATLAB do not correctly support save and load.
If you determine that the Cplex data types cannot be correctly saved and loaded, then the simplest workaround is to build the Cplex data directly on the worker, operate on it there, and return to the client only results (which are hopefully of a simple data type that can be correctly transferred). So, you might do something like this:
parfor idx = 1:numel(filepaths)
prob = Cplex('uc');
prob.readModel(filepaths{idx});
prob.Param.threads.Cur = 1;
prob.solve();
out{idx} = doSomething(prob); % Not sure what you need here.
end
This code assumes that the workers can see the same files as the client. This will be the case if you are using the 'local' cluster type (i.e. the default). If the workers can't see the same files, then you might need to use addAttachedFiles to make the files available on the workers.
  3 Comments
Edric Ellis
Edric Ellis on 31 Jul 2020
One thing that you need to bear in mind is that when you use a handle object inside a parfor loop, each worker gets a copy of that object. Updates that you make on the worker are not automatically reflected back at the client. So perhaps that's what's going on. I'd still be looking to try and build the handle objects directly on the workers inside the parfor loop, and sending back to the client only the results - presuming they can safely be copied. More here in the doc.
Katarzyna Furmanska
Katarzyna Furmanska on 31 Jul 2020
Yeah, I've read the doc and I've already fixed one issue by adding appropriate line reflecting what is done in 'maps{ii} = mymap' line. It definitely helped, but there are still some differences.
I'm afraid that building the objects directly in the parfor loop would not work here, as there's lot happening before parfor and also it is wrapped in an outside while loop; perhaps it would bring too much overhead than benefit of using. Although, I managed to read the A(i).cpx.Solution.x values into the cell array and assign them to A(i).x outside parfor loop, so I think there is no other way than just read all the others in the same manner.
Thank you Edric for quick response!

Sign in to comment.

More Answers (0)

Categories

Find more on Parallel for-Loops (parfor) in Help Center and File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!