Merging contents of multiple MAT files

Hello,

I got a collection of MAT files (generated by External Mode data archiving) that have names such as below and their count can go into 1000's: - MyMatFile_0.mat - MyMatFile_1.mat (...) - MyMatFile_2999.mat - MyMatFile_3000.mat

What i want to achieve is to merge all the arrays from inside the MAT files vertically together (like [MyMatFile_0.mat ; MyMatFile_1.mat ... ] and save that new single array to a spearate MAT file or simply leave it in workspace.

So for, example, with the files I'm attaching, each of them contains a 20x2 double array and I want to merge them together to make one MAT file with 60x2 double array.

Also, this batch of files is generated per each experiment i do and for each experiment the total number of MAT files may vary and the amount of elements in array in each MAT file also may vary (but there will be always 2 columns).

Any help is appreciated.

Answer 1

[Stephen23]

To read the files in the correct order download my FEX submission natsortfiles:

https://www.mathworks.com/matlabcentral/fileexchange/47434-natural-order-filename-sort

Then all you need is this (the input and output files are attached):

D = 'path to the folder where the files are saved'
D = '.';
S = dir(fullfile(D,'TestLog_*.mat'));
S = natsortfiles(S); % alphanumeric sort by filename
for k = 1:numel(S)
    T = load(fullfile(D,S(k).name));
    S(k).data = struct2cell(T);
end
M = vertcat(S.data);
save('TestLog.mat','M')

This will work for any number of .mat files, with a few conditions:

• each .mat file contains only one variable.
• the output file is in a different folder or does not match the dir filename string (otherwise you could end up accumulating the same data repeatedly).

[Multiplexer]

Thank you very much, works perfectly.

[Stephen23]

@Multiplexer: my pleasure!

The trick is to always load into an output variable... everything else follows on from that :)

[Multiplexer]

Hello,

Sorry for asking for more, but would it be difficult to modify the script to work in suituation where MAT fail contains two variables each with two columns so the in the result, instead of one merged MAT file there would be two MAT files, first containing first variables and second containing second variables

[Stephen23]

"... would it be difficult to modify the script to work in suituation where MAT fail contains two variables each ..."

Your life would be much simpler if the variable names in each .mat file were exactly the same, then this task would be trivial: if you can change the code that writes the .mat files, then that is the best option:

C = cell(2,...)
for k = ...
    T = load(...);
    C{1,k} = T.hello;
    C{2,k} = T.world;
end

As it is you have a few options:

• load the entire .mat file contents into a structure, use fieldnames, match the fieldnames (e.g. strncmp or strncmpi for your fieldnames) and use the indices to allocate to two cell arrays.
• load one variable at-a-time using load's -regexp option, and use the same code as in my answer, once for each cell array.
• load the entire .mat file contents into a structure, sort the fields using sortfields, convert to cell using struct2cell, use indexing to assign to two cell arrays.

So the answer to your question is "it is not difficult, but it is fiddly... and can easily be avoided by naming the data better".

[Multiplexer]

Is this the way you meant to modify data? Each MAT file now contains two variables and the variable name is exactly same between each MAT file.

I started modifying the code but i messed something up big time. Instead of doing merge it started copying same file over and over again.

D = 'path to the folder where the files are saved'
S = dir(fullfile(D,'TestLog_*.mat'));
N = natsortfiles({S.name}); % download from FEX.
C = cell(2,numel(N));

   for k = 1:numel(N)
       T = load(fullfile(D,N{k}));
       C{1,k} = T.Excite;
       C{2,k} = T.Response;
   end
   M = vertcat(C{:});
   save('TestLog.mat','M')

[Stephen23]

"Is this the way you meant to modify data? Each MAT file now contains two variables and the variable name is exactly same between each MAT file."

Yes, that is perfect. It will make your code much simpler. Here is some code that reads your new sample files, and follows the requirements that you gave in your earlier comment:

D = 'path to the folder where the files are saved';
S = dir(fullfile(D,'TestLog_*.mat'));
S = natsortfiles(S); % sort the files into alphanumeric order.
for k = 1:numel(S)
    T = load(fullfile(D,S(k).name));
    S(k).Excite = T.Excite;
    S(k).Response = T.Response;
end
Excite   = vertcat(S.Excite);
Response = vertcat(S.Response);
save('TestExcite.mat',  'Excite')
save('TestResponse.mat','Response')

[Multiplexer]

Thank you very much!

[Hanah Aemilia Choice]

When I run this script (the first posted solution), I get the error message:

Reference to non-existent field 'data'.

Error in displayscript (line 23)

M = vertcat(S.data);

I'm wondering what the cause of this would be.

[Stephen23]

@Hanah Aemilia Choice

"I'm wondering what the cause of this would be."

No files match the DIR name pattern so S is sempty, so the loop never iterates, so the data field is not created.

Take a look at numel(S)