MATLAB Answers

Simon
0

Parfor waitbar : How to do this more cleanly?

Asked by Simon
on 6 Jun 2019
Latest activity Edited by Edric Ellis
on 11 Jun 2019
I'm a frequent casual user of parfor loops. Getting a waitbar (progress meter) from parallel code used to involve having to have each thread append to a file to record progress, and there are a number of contributions on the file exchange that do this. It was kinda silly, but it was necessary.
Now that the DataQueue system exists, it should be possible to do this more sensibly. So I went looking for a good waitbar on the File Exchange. And I couldn't find one, so I tried to make my own.
The code below works nicely, but it depends on using three globals - which isn't great, because if those variable names are being used by something else, it's gonna break stuff. Can anybody suggest a better way to do this, preferably not involving globals? Persistent variables should work within update.m, but it's getting the info (waitbar handle and expected number of interations) from create.m to update.m that's stumped me.
There are three functions, all part of the ParWaitbar package. Usage is explained in the first one:
create.m:
function [ q ] = create( n, txt )
%FNPARWAITBAR Produces a waitbar object that works with parfor.
% Relies on globals. Not sure how to avoid this.
% Input:
% n: Number of iterations expected
% txt: Text for the waitbar dialog.
% Output: q: handle for the dataqueue
%
% Usage: [ q ] = ParWaitbar.create( 10, "Predicting earthquakes..." );
% parfor i=1:10
% pause(rand*5); (or do useful stuff)
% send( q, i ); % doesn't matter what's sent (i here). We're just
% counting the items rcvd.
% end
% ParWaitbar.cleanup(q);
global wb; %waitbar handle.
global wb_N; % number of iterations completed
global wb_max; %number of iterations expected
q = parallel.pool.DataQueue;
wb = waitbar( 0, txt );
wb_max = n;
wb_N = 0;
afterEach( q, @ParWaitbar.update );
end
update.m:
function update(~)
global wb; %shoudl have waitbar handle.
global wb_N; % number of iterations completed
global wb_max; %number of iterations expected
wb_N = wb_N + 1;
waitbar( wb_N / wb_max, wb );
end
cleanup.m
function cleanup( q )
% FIXME we're left with q afterwards. Presumably a fn can't delete
% something outside its own scope
global wb; %shoudl have waitbar handle.
global wb_N; % number of iterations completed
global wb_max; %number of iterations expected
close(wb);
delete(q);
clear wb wb_N wb_max q
end
(as an aside, the fact that we're left with q leftover as a "handle to deleted DataQueue" isn't ideal. But that's far less of a problem than the use of globals with fixed names.)

  0 Comments

Sign in to comment.

1 Answer

Answer by Edric Ellis
on 10 Jun 2019
 Accepted Answer

Hm, I've been meaning to tidy up my work-in-progress parallel.pool.DataQueue pool waitbar for quite a while. Here's roughly what it would look like. This uses a handle object that gets copied to the workers, so there's no need for global state.
classdef PoolWaitbar < handle
properties (SetAccess = immutable, GetAccess = private)
Queue
N
end
properties (Access = private, Transient)
ClientHandle = []
Count = 0
end
properties (SetAccess = immutable, GetAccess = private, Transient)
Listener = []
end
methods (Access = private)
function localIncrement(obj)
obj.Count = 1 + obj.Count;
waitbar(obj.Count / obj.N, obj.ClientHandle);
end
end
methods
function obj = PoolWaitbar(N, message)
if nargin < 2
message = 'PoolWaitbar';
end
obj.N = N;
obj.ClientHandle = waitbar(0, message);
obj.Queue = parallel.pool.DataQueue;
obj.Listener = afterEach(obj.Queue, @(~) localIncrement(obj));
end
function increment(obj)
send(obj.Queue, true);
end
function delete(obj)
delete(obj.ClientHandle);
delete(obj.Queue);
end
end
end
This works with parfor, spmd, and parfeval, like this:
pw = PoolWaitbar(100, 'Example');
parfor ii = 1:20
increment(pw)
end
spmd
for ii = 21:40
if labindex == 1
increment(pw);
end
end
end
for ii = 41:100
parfeval(@() increment(pw), 0);
end

  2 Comments

Fabulous, thank you! I don't fully understand Matlab's OO facilities, but this seems to do nicely.
Is there any reason to do increment(pw) instead of pw.increment? The latter feels more intuitive to me for using an object's method, but I don't know what normal is in Matlab for this
Are you planning to put this on File Exchange?
This PoolWaitbar does use a slightly subtle trick to make things work - the use of Transient properties stops the client-side handle being sent to the workers.
There's no real difference between increment(pw) and pw.increment - but the former is the usual function-call style that tends to be used by MathWorks documentation and example code.
I would like to put this on File Exchange - but it's part of a larger suite of similar utilities, and there's some polishing required...

Sign in to comment.