Parallel Matrix Multiplication on a Distributed Computing System

Hi,

Some functions in MATLAB like matrix multiplication are parallelized by default. I was wondering if matrix multiplication would still be parallelized by default when such operations are performed on a cluster using the Distributed computing toolbox. Will MATLAB automatically recognize how many workers are available and parallelize matrix operations accordingly?

The reason I ask is because I am trying to speed up my code which has a lot of matrix operations being performed. Switching to computers with a larger number of cores has dramatically reduced the runtime. However, I have never tested if such an effect persists in a cluster.

I use the batch command to send a job to the cluster and within the batch command I open several workers using {'matlabpool', numberofWorkers}. Will matrix multiplication be parallelized automatically using this method?

Thanks for your help!

Answer 1

[Konrad Malkowski]

There are two levels of parallelism present in MATLAB:

1. Implicit Multi-threaded parallelism for certain built-in MATLAB commands, such as Matrix-Matrix Multiplication or Matrix Factorization.
2. Explicit parallelism present in Parallel Computing Toolbox

Let's focus on the implicit multi-threaded parallelism first.

If you have a multi-core or multi-processor machine then the implicit multi-threaded parallelism is on by default in the client MATLAB. When I use the term client MATLAB, I mean the interactive MATLAB that you are running on your Windows/Linux/Mac desktop.

The number of threads used is set automatically by MATLAB at run time. You can type

>>maxNumCompThreads

to find out how many threads MATLAB is using for computation.

Keep in mind that MATLAB ignores hyperthreading. So for example if you have a hyperthreaded processor, your operating system might report 8 cores, but MATLAB will only see the 4 physical cores and report 4 as result of maxNumCompThreads.

If you need to you can disable implicit MATLAB multi-threading using one of the following:

1. Start Start MATLAB with -singleCompThread startup option
2. Type maxNumCompThreads(1) in your program

Note that maxNumCompThreads is currently deprecated and could be discontinued in a future release of MATLAB.

http://www.mathworks.com/help/releases/R2011b/techdoc/ref/maxnumcompthreads.html http://www.mathworks.com/help/releases/R2011b/techdoc/ref/matlabwindows.html

In relation to the earlier answer, if you start MATLAB on a machine that has 192 cores, MATLAB will report maxNumCompThreads of 192. On such a large machine the client MATLAB will have implicit parallelism of 192 threads. However, you as a user will not be able to control on which cores the threads are run. That will be handled by the operating system.

Now let's discuss the explicit parallelism provided by the Parallel Computing Toolbox and MDCS.

When you type MATLABPOOL open, MATLAB starts instances of headless MATLAB workers. These workers run either on your local machine (local scheduler) or on a MDCS cluster. These workers are by default single-threaded.

You can test that using the following code snippet:

matlabpool open
spmd
   maxNumCompThreads
end
matlabpool close

However, if you believe that your application would benefit from using a hybrid of explicit and implicit parallelism, for example your tasks are performing many matrix multiplications or matrix factorizations, you can re-enable the implicit parallelism by placing

maxNumCompThreads(N)

at the top of the function you are trying to run on workers, or inside spmd block.

spmd
   maxNumCompThreads(N);
end

In my example N is the number of threads that you want to use and should be a reasonable value, for example 2, 4, 8.

You should only re-enable implicit parallelism on workers in situations where it is really warranted, for example, if you have a single MATLAB worker on a multi-core node.

If you have multiple MATLAB workers running on a single node, I would not recommend enabling multi-threaded support as this will most likely result in performance degradation. See Cleve’s article for benchmarks related to mixing multi-threading and workers: http://www.mathworks.com/company/newsletters/news_notes/june07/clevescorner.html

At the moment it is still possible to enable multi-threading on the workers using the maxNumCompThreads command. However, this functionality is deprecated and could be removed in a future release of MATLAB.

[Nicholas]

Thanks Konrad! This clears it up. So theoretically I can take advantage of the implicit multi-threaded parallelism in matrix multiplication by initializing a single worker on a cluster with many cores and setting maxNumCompThreads(NumberOfCores). I will try this out. Thanks!

Answer 2

[Walter Roberson]

I have not seen a clear answer on this, but perhaps I am just being stubborn in not understanding this previous Question which seems to say Yes.

My answer to the above was a link to an earlier discussion which I would have always interpreted as seeming to say No.

Somewhere around here I wrote an Answer that discussed using multiple levels of matlabpool calls, but I am having difficulty tracking down that earlier response.

[Nicholas]

Hey thanks for your reply! So if I am understanding this correctly:

All of the workers I initialize on the cluster are single-threaded by default. Therefore it will not be able to take advantage of multi-threaded operations like matrix multiplication. Thus, I would need to increase the number of threads per worker in order to take advantage of parallel matrix multiplication.

Is this correct?

[Walter Roberson]

Unfortunately I have not been able to tell by reading the documentation. The answer I linked to above seems to say that the automatic thread use for matrix multiplication and so on, will happen even in single-threaded mode, but that you will not have any control over it. That is the only place I have seen that possibility described, and in my mind it seems contradictory to what I have understood of how threading works.

I will send a note to Konrad pointing him to this Question.