How do I specify the number of workers in parfeval without deleting and recreating my pool

I have a large, highly parallelized, and pretty efficient process I run for a while using parfor with all my cores (physical and logical) and a few gpus.

Once that process is finished, I need to perform a different process that is memory intensive, and I cannot use all my cores without hitting memory problems. But I can do the process just fine using half my cores.

Is there a way to take an existing pool (about 40 workers) and say "hey for this bit to code (pareval, not parfor) I only want to use 20 of my 40 workers that I have available in my pool without resetting my pool". Resetting the pool takes time that I'd like to avoid if possible. Sort of like parfor (ii = start:stop, nWorkers), but for parfeval.

To clarify, I know I can create a new pool with specified number of works, and the process in question is suited for parfeval, but not parfor. Lets assume for now that my problem is not memory efficiency and reducing the amount of memory, Ill deal with that seperately. This question is just about specifying number of workers in parfeval without deleting my pool and creating a new one.

Suggestions?

Answer 1

[Edric Ellis]

Unfortunately, there's no API to constrain the number of workers from the pool eligible to be used by parfeval. What you could consider is the following hack: basically, put half the workers into an infinite pause until you're done doing the rest of your work. I.e.

% Block half the workers
numToBlock = 20;
for idx = numToBlock:-1:1
    blockers(idx) = parfeval(@pause, 0, Inf);
end
stopBlocking = onCleanup(@() cancel(blockers));

% Do stuff
f = parfeval(@myMemoryIntensiveThing,<args>);
... % more stuff
... % more stuff

% Eventually, clean up.
stopBlocking = []; % releases the blocked workers.

You could confine the ugliness by writing a simple class, something like

classdef Blocker < handle
    properties (SetAccess = immutable)
        Futures
    end
    methods
        function obj = Blocker(pool, numToBlock)
            assert(numToBlock <= pool.NumWorkers && numToBlock > 0);
            for idx = numToBlock:-1:1
                fut(idx) = parfeval(pool, @pause, 0, Inf);
            end
            obj.Futures = fut;
        end
        function delete(obj)
            cancel(obj.Futures);
        end
    end
end

[David Saidman]

Thatll do it. Not prettiest solution but pretty enough and will do what I needed. Thanks. Quick answer.