Passing big matrix to workers
1 view (last 30 days)
Show older comments
I'm trying to pass very large matrix (65000x460x150) to workers with Worker Object Wrapper, with no luck.
The code:
Imat_worker = WorkerObjWrapper(Imat);
parfor kk = 1:size(num_iterations,2)
Imat_par = Imat_worker(:,:,1:num_iterations(kk));
some code...
end
I get the following message while running the code:
Error using WorkerObjWrapper/workerInit (line 156)
The parallel pool that SPMD was using has been shut down.
Error in WorkerObjWrapper (line 97)
WorkerObjWrapper.workerInit( tmpId, ctor, args, dtor );
A write error occurred while sending to worker 2.
Any ideas what am I doing wrong?
It is worth noting that I'm running the code on a server with enough RAM for the matrix.
Thanks in advance!
8 Comments
Accepted Answer
Matt J
on 5 Jul 2018
Edited: Matt J
on 5 Jul 2018
In that case, since I need only a slice of the matrix for each worker, shouldn't I clone only the relevant data somehow before the parfor ?
Yes you should, and that's something parfor will do for you automatically as long as the operations you do on Imat inside the loop satisfy certain restrictions.
2 Comments
Matt J
on 5 Jul 2018
Edited: Matt J
on 5 Jul 2018
Yes, making Imat a sliced variable is much more sensible if it is possible to do so.
If you can afford the loss in precision, you could also consider storing the original Imat as single floats. You could still do the calculations in double precision on the workers by converting the current slice being processed to double in the course of the loop. The total amount of double data being held anywhere simultaneously will then be much smaller.
More Answers (0)
See Also
Categories
Find more on Parallel for-Loops (parfor) in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!