Speeding up a load file workflow

I'm working with a program that outputs simulation to results to many .mat files. To analyze these results, I need to load each of these mat files. Currently, I use the following lines to do this.

    for i = 1:numFiles
        cases{i} = load(files(i).name);
    end

after preallocating the cases cell array, of course. The problem is, repeatedly using the load function like this can drastically increase the time I need to analyze my results, depending on how many mat files this program outputs.

My question, then, is this: Is there any way to load multiple files at once?

Thanks!

Answer 1

[Swastik Sarkar]

Hi @Paul,

I know of 2 options to load the MAT-files faster, both of which will require the Parallel Computing Toolbox.

One approach is to utilize the parfor loop to load MAT files in parallel on separate workers:

    parfor i = 1:numFiles
        cases{i} = load(files(i).name);
    end

Another approach is to use the parfeval function to create futures and wait for them asynchronously:

for k = 1:numFiles
    futures(k) = parfeval(@load, 1, files(k).name);
end

for k = 1:numFiles
    
    [idx, loadedData] = fetchNext(futures);
    
    cases{idx} = loadedData;
end

Hope this helps load MAT-files faster.

[Paul]

I accepted this because you have answered my question. If I don't want to use the PCT, do you know of any ways to do that?

[Walter Roberson]

load() is compatible with using backgroundPool and parfeval

Other than Parallel Computing Toolbox, and Background Pools, there is no way to load multiple files simultaneously.