Assign values of .mat files into matrix

Dear all,

I have multiple .mat files (see attachment). I want to organize the MFR_mod variables within the those .mat files into one matrix. I feel like there is a smarter way then what I tried below. This will take me forever, since MFR_mod goes from 1 until 81.

for ii = 1:20
    load(['silomodresults',num2str(ii),'.mat'])
end

MFR_tot = [MFR_mod1; MFR_mod2; MFR_mod3; MFR_mod4; MFR_mod5; MFR_mod6; MFR_mod7; MFR_mod8; MFR_mod9; MFR_mod10; MFR_mod11; MFR_mod12; MFR_mod13; MFR_mod14; MFR_mod15; MFR_mod16; MFR_mod17; MFR_mod18; MFR_mod19; MFR_mod20];

Answer 1

[Stephen23]

D = 'path to the folder where the files are saved';
S = dir(fullfile(D,'silomodresults*.mat'));
C = {};
for k = 1:numel(S)
    F = fullfile(D,S(k).name);
    T = load(F,'-regexp','^MFR_mod\d+$');
    N = fieldnames(T);
    V = str2double(regexp(N,'\d+','once','match'));
    C(V) = struct2cell(T); %#ok<SAGROW>
end
M = vertcat(C{:}); % optional

Giving:

>> size(M)
ans =
    81     1
>> plot(M)

[Tessa Kol]

Dear Stephen,

Thank you for your answer. Is something like that also possible when I don't have to manually assign the names MFR_mod1 MFR_mod2 ... etc. to the calculated MFR_mod(i) for every silomoddata.mat file? Like I try to explain in the comment to Ameer Hamza.

[Stephen23]

Numbering variables like that is a very bad approach to writing code, and makes processing the data difficult.

You would be much better off storing the data in two arrays:

1. a cell array containing the data matrices
2. a numeric vector of the indices.

For example, your fifth .mat file would look like this:

dtc = {[..], [..], [..], [..], [..]};
idx = [   5,   11,   20,   38,  81];
save('whatever.mat','dtc','idx')

Obviously C and V can be created very easily using any for/while loop in your code, and do NOT require any magical numbered variable names. Not only that, they can be loaded reliably, e.g. in the loop shown in my answer:

S = load(.'whatever.mat');
C(S.idx) = S.dtc;

No messing about with strings and regular expressions and the like :)

[Tessa Kol]

I take silomoddata5.mat as an example.

I calculated the MFR_mod. Which will give me:

Then I save this into a new .mat file

idx = [   5,   11,   20,   38,  81];
save('silomodresults5.mat','MFR_mod','idx')

S = load('silomodresults5.mat');
C(S.idx) = S.MFR_mod;

But still I have to run to code manually for every silomoddata.mat file to obtain de MFR_mod. And also change the idx manually each time.

[Stephen23]

"But still I have to run to code manually for every silomoddata.mat file to obtain de MFR_mod."

Why not use a loop?

"And also change the idx manually each time."

Presumably that there is some way of generating this index from the input parameters. Or are they pseudo-random?

[Tessa Kol]

I tried using a for/while loop as you suggested. I used the silomoddata1.mat, silomoddata2.mat, silomoddata3.mat, silomoddata4.mat and silomoddata5.mat. But something is wrong. Running the code below is taking a very long time and I stopped the simulation eventually. Changing idx manually each time doesn't bother me that much.

TimeStepstable = [3.04, 6.5];

for ii = 1:5
    load(['silomoddata',num2str(ii),'.mat']);
    N_run = numel(runData_struc);
% Pre-allocate one cell for each loop variable
particlesInSilo = cell(max(cellfun(@numel,runData_struc)), N_run);
siloMass = particlesInSilo;
time = particlesInSilo;

k =1;
for i = 1:N_run
    for j = 1:numel(runData_struc{i})
        %Count the number of particles in the silo (silo outlet is located at z = 0.3 m)
        particlesInSilo{j,i} = find(expData_struc{1,i}{1,j}(:,3)>=0.3);
        % Multiply the number of particles with the volume of the particle
        siloMass{j,i} = sum(expData_struc{1,i}{1,j}(cell2mat(particlesInSilo(j,i)),6));
        % Corresponding time step
        time{j,i} = runData_struc{1,i}{1,j}(:,2);
        k = k + 1;
    end
end

% Index the time step values closest to the desired time step where the mass flow is stable
for i = 1:N_run
    for t = 1:numel(TimeStepstable)
        [~, idxtime(t,i)] = min(abs(cell2mat(time(:,i))-TimeStepstable(t)));
    end
end

% Calculate the mass flow rate (MFR) for the stable region
for i = 1:N_run
    siloMass_stable{i} = cell2mat(siloMass(idxtime(1,i):idxtime(2,i),i));
    time_stable{i} = cell2mat(time(idxtime(1,i):idxtime(2,i),i));
    opts = fitoptions('Method', 'LinearLeastSquares', 'Robust', 'off');
    [MFR_fit{i}, GoodnessOfFit] = fit(time_stable{i}(:,1),siloMass_stable{i}(:,1),'poly1',opts);
    MFR_mod(i) = abs(MFR_fit{i}.p1);
end
while ii == 1
    idx = [   1];
save(['silomodresultsv2',num2str(ii),'.mat'],'MFR_mod','idx')
end
while ii == 2
    idx = [   2];
save(['silomodresultsv2',num2str(ii),'.mat'],'MFR_mod','idx')
end
while ii == 3
    idx = [   3];
save(['silomodresultsv2',num2str(ii),'.mat'],'MFR_mod','idx')
end
while ii == 4
    idx = [   4];
save(['silomodresultsv2',num2str(ii),'.mat'],'MFR_mod','idx')
end
while ii == 5
    idx = [   5,   11,   20,   38,  81];
save(['silomodresultsv2',num2str(ii),'.mat'],'MFR_mod','idx')
end
end

[Tessa Kol]

I changed it to a if/else loop and it seems to work. I will let you know tomorrow if everything went well from here and if the intended goal is reached.

TimeStepstable = [3.04, 6.5];

for ii = 1:5
    load(['silomoddata',num2str(ii),'.mat']);
    N_run = numel(runData_struc);
% Pre-allocate one cell for each loop variable
particlesInSilo = cell(max(cellfun(@numel,runData_struc)), N_run);
siloMass = particlesInSilo;
time = particlesInSilo;

k =1;
for i = 1:N_run
    for j = 1:numel(runData_struc{i})
        %Count the number of particles in the silo (silo outlet is located at z = 0.3 m)
        particlesInSilo{j,i} = find(expData_struc{1,i}{1,j}(:,3)>=0.3);
        % Multiply the number of particles with the volume of the particle
        siloMass{j,i} = sum(expData_struc{1,i}{1,j}(cell2mat(particlesInSilo(j,i)),6));
        % Corresponding time step
        time{j,i} = runData_struc{1,i}{1,j}(:,2);
        k = k + 1;
    end
end

% Index the time step values closest to the desired time step where the mass flow is stable
for i = 1:N_run
    for t = 1:numel(TimeStepstable)
        [~, idxtime(t,i)] = min(abs(cell2mat(time(:,i))-TimeStepstable(t)));
    end
end

% Calculate the mass flow rate (MFR) for the stable region
for i = 1:N_run
    siloMass_stable{i} = cell2mat(siloMass(idxtime(1,i):idxtime(2,i),i));
    time_stable{i} = cell2mat(time(idxtime(1,i):idxtime(2,i),i));
    opts = fitoptions('Method', 'LinearLeastSquares', 'Robust', 'off');
    [MFR_fit{i}, GoodnessOfFit] = fit(time_stable{i}(:,1),siloMass_stable{i}(:,1),'poly1',opts);
    MFR_mod(i) = abs(MFR_fit{i}.p1);
end
if ii == 1
    idx = [   1];
save(['silomodresultsv2',num2str(ii),'.mat'],'MFR_mod','idx')

elseif ii == 2
    idx = [   2];
save(['silomodresultsv2',num2str(ii),'.mat'],'MFR_mod','idx')


elseif ii == 3
    idx = [   3];
save(['silomodresultsv2',num2str(ii),'.mat'],'MFR_mod','idx')

elseif ii == 4
    idx = [   4];
save(['silomodresultsv2',num2str(ii),'.mat'],'MFR_mod','idx')

elseif ii == 5
    idx = [   5,   11,   20,   38,  81];
save(['silomodresultsv2',num2str(ii),'.mat'],'MFR_mod','idx')
end
end

[Tessa Kol]

Based upon your code and hints I managed to organize the data into a single matrix (see code below). Maybe it can be even shorter/nicer, but for now this works perfectly.

TimeStepstable = [3.04, 6.5];
TimeStepsettle = [1.56];
V_silo = 0.66*0.60*0.053-0.25*0.053*(0.25*tand(45));

%% Objective function input data

for ii = 19:20
    load(['silomoddata',num2str(ii),'.mat']);
    N_run(ii) = numel(runData_struc);
    
    
    % Pre-allocate one cell for each loop variable
    particlesInSilo = cell(max(cellfun(@numel,runData_struc)), N_run(ii));
    siloMass = particlesInSilo;
    time = particlesInSilo;

    k =1;
    for i = 1:N_run(ii)
        for j = 1:numel(runData_struc{i})
            %Count the number of particles in the silo (silo outlet is located at z = 0.3 m)
            particlesInSilo{j,i} = find(expData_struc{1,i}{1,j}(:,3)>=0.3);
            % Multiply the number of particles with the volume of the particle
            siloMass{j,i} = sum(expData_struc{1,i}{1,j}(cell2mat(particlesInSilo(j,i)),6));
            % Corresponding time step
            time{j,i} = runData_struc{1,i}{1,j}(:,2);
            k = k + 1;
        end
    end

    % Index the time step values closest to the desired time step where the mass flow is stable
    for i = 1:N_run(ii)
        for t = 1:numel(TimeStepstable)
            [~, idxtime(t,i)] = min(abs(cell2mat(time(:,i))-TimeStepstable(t)));
        end
    end

    % Calculate the mass flow rate (MFR) for the stable region
    for i = 1:N_run(ii)
        siloMass_stable{i} = cell2mat(siloMass(idxtime(1,i):idxtime(2,i),i));
        time_stable{i} = cell2mat(time(idxtime(1,i):idxtime(2,i),i));
        opts = fitoptions('Method', 'LinearLeastSquares', 'Robust', 'off');
        [MFR_fit{i}, GoodnessOfFit] = fit(time_stable{i}(:,1),siloMass_stable{i}(:,1),'poly1',opts);
        MFR_mod(i) = abs(MFR_fit{i}.p1);
    end
    
    % Bulk density
    for i = 1:N_run(ii)
        for s = 1:numel(TimeStepsettle)
            [~, idxtimesettle(s,i)] = min(abs(cell2mat(time(:,i))-TimeStepsettle(s)));
        end
    end

    for i = 1:N_run(ii)
        siloMass_settle(i) = cell2mat(siloMass(idxtimesettle(1,i),i));
        rho_bmod(i) = siloMass_settle(i)/V_silo;
    end
    
    if ii == 1
        idx = [1];
        save(['silomodresults',num2str(ii),'.mat'],'MFR_mod','rho_bmod','idx')

    elseif ii == 2
        idx = [2];
        save(['silomodresults',num2str(ii),'.mat'],'MFR_mod','rho_bmod','idx')

    elseif ii == 3
        idx = [3];
        save(['silomodresults',num2str(ii),'.mat'],'MFR_mod','rho_bmod','idx')

    elseif ii == 4
        idx = [4];
        save(['silomodresults',num2str(ii),'.mat'],'MFR_mod','rho_bmod','idx')

    elseif ii == 5
        idx = [5,11,20,38,81];
        save(['silomodresults',num2str(ii),'.mat'],'MFR_mod','rho_bmod','idx')
    elseif ii == 6
        idx = [12,15,31,55,65];
        save(['silomodresults',num2str(ii),'.mat'],'MFR_mod','rho_bmod','idx')
    elseif ii == 7
        idx = [21,22,23,24,25];
        save(['silomodresults',num2str(ii),'.mat'],'MFR_mod','rho_bmod','idx')
    elseif ii == 8
        idx = [30,32,33,34,35];
        save(['silomodresults',num2str(ii),'.mat'],'MFR_mod','rho_bmod','idx')
    elseif ii == 9
        idx = [40,41,42,43,44];
        save(['silomodresults',num2str(ii),'.mat'],'MFR_mod','rho_bmod','idx')
    elseif ii == 10
        idx = [13,14,16,17,18];
        save(['silomodresults',num2str(ii),'.mat'],'MFR_mod','rho_bmod','idx')
    elseif ii == 11
        idx = [50,51,52,53,54];
        save(['silomodresults',num2str(ii),'.mat'],'MFR_mod','rho_bmod','idx')
    elseif ii == 12
        idx = [60,61,62,63,64];
        save(['silomodresults',num2str(ii),'.mat'],'MFR_mod','rho_bmod','idx')
    elseif ii == 13
        idx = [70,71,72,73,74];
        save(['silomodresults',num2str(ii),'.mat'],'MFR_mod','rho_bmod','idx')
    elseif ii == 14
        idx = [19,26,27,28,29];
        save(['silomodresults',num2str(ii),'.mat'],'MFR_mod','rho_bmod','idx')
    elseif ii == 15
        idx = [6,7,8,9,10];
        save(['silomodresults',num2str(ii),'.mat'],'MFR_mod','rho_bmod','idx')
    elseif ii == 16
        idx = [36,37,38,39,45];
        save(['silomodresults',num2str(ii),'.mat'],'MFR_mod','rho_bmod','idx')
    elseif ii == 17
        idx = [46,47,48,49,56];
        save(['silomodresults',num2str(ii),'.mat'],'MFR_mod','rho_bmod','idx')
    elseif ii == 18
        idx = [57,58,59,66,67];
        save(['silomodresults',num2str(ii),'.mat'],'MFR_mod','rho_bmod','idx')
    elseif ii == 19
        idx = [68,69,75,76];
        save(['silomodresults',num2str(ii),'.mat'],'MFR_mod','rho_bmod','idx')
    elseif ii == 20
        idx = [77,78,79,80];
        save(['silomodresults',num2str(ii),'.mat'],'MFR_mod','rho_bmod','idx')
    end
end

%% Combining input data

MFR_modtot = zeros(1,81);
rho_bmodtot = zeros(1,81);

for ii = 1:20
    S(ii) = load(['silomodresults',num2str(ii),'.mat']);
    MFR_modtot(S(ii).idx) = S(ii).MFR_mod;
    rho_bmodtot(S(ii).idx)= S(ii).rho_bmod;
end

[Stephen23]

You could define all of the indices before the loop:

idc = {1,2,3,4,[5,11,20,38,81],[12,15,31,55,65],...};

and then inside the loop you can replace that entire huge lbock of code

if i== ..
    ..
elseif i== ..
    ..
    ..
end

with just this:

idx = idc{ii};
fnm = sprintf('silomodresults%d.mat',ii);
save(fnm,'MFR_mod','rho_bmod','idx')

It is still a mystery to me, how those indices are defined.

[Tessa Kol]

Thank you for the tip!

The indices are based on poor folder structure management from my side.

Answer 2

[Ameer Hamza]

One of the major problem is naming the variables like this x1, x2, ...: https://www.mathworks.com/matlabcentral/answers/304528-tutorial-why-variables-should-not-be-named-dynamically-eval. You can try something like this

MFR_tot = zeros(1, 20);
for ii = 1:20
    data = load(['silomodresults',num2str(ii),'.mat']);
    MFR_tot(ii) = data.(sprintf('MFR_mod%d', ii))
end

[Tessa Kol]

I came across the issue with eval many times in different posts. You solution didn't work unfortunatly, because the .mat files have unorganized MFR_mod. Maybe if I start from the beginning there is a better solution to this.

So first I have created 20 .mat files. Each .mat file is about 1.5 GB. So it is not possible to send it in this post.

For each .mat file I run the same code manually. The below code is an example of running the code for only the silomoddata20.mat file.

%% Loading the Data
load([pwd,'\silomoddata20.mat'])

%% Parameters
% Number of simulation runs
N_run = numel(runData_struc);
TimeStepstable = [3.04, 6.5];
TimeStepsettle = [1.56];

%% Mass flow Rate
% Pre-allocate one cell for each loop variable
particlesInSilo = cell(max(cellfun(@numel,runData_struc)), N_run);
siloMass = particlesInSilo;
time = particlesInSilo;

k =1;
for i = 1:N_run
    for j = 1:numel(runData_struc{i})
        %Count the number of particles in the silo (silo outlet is located at z = 0.3 m)
        particlesInSilo{j,i} = find(expData_struc{1,i}{1,j}(:,3)>=0.3);
        % Multiply the number of particles with the volume of the particle
        siloMass{j,i} = sum(expData_struc{1,i}{1,j}(cell2mat(particlesInSilo(j,i)),6));
        % Corresponding time step
        time{j,i} = runData_struc{1,i}{1,j}(:,2);
        k = k + 1;
    end
end

% Index the time step values closest to the desired time step where the mass flow is stable
for i = 1:N_run
    for t = 1:numel(TimeStepstable)
        [~, idxtime(t,i)] = min(abs(cell2mat(time(:,i))-TimeStepstable(t)));
    end
end

% Calculate the mass flow rate (MFR) for the stable region
for i = 1:N_run
    siloMass_stable{i} = cell2mat(siloMass(idxtime(1,i):idxtime(2,i),i));
    time_stable{i} = cell2mat(time(idxtime(1,i):idxtime(2,i),i));
    opts = fitoptions('Method', 'LinearLeastSquares', 'Robust', 'off');
    [MFR_fit{i}, GoodnessOfFit] = fit(time_stable{i}(:,1),siloMass_stable{i}(:,1),'poly1',opts);
    MFR_mod(i) = abs(MFR_fit{i}.p1);
end

%% save results

save silomodresults20.mat MFR_mod rho_bmod

I run the above code manually for every silomoddata*.mat file. Because I didn't know how to put this in a for loop.

If the whole code is in for loop I hope to get a 1x81 matrix of MFR_mod. The problem with that is the order.

silomoddata1.mat contains simulationrun 1

silomoddata2.mat contains simulationrun 2

silomoddata3.mat contains simulationrun 3

silomoddata4.mat contains simulationrun 4

silomoddata5.mat contains simulationrun 5, simulationrun 11, simulationrun 20, simulationrun 38 and simulationrun 81

silomoddata6.mat contains simulationrun 12, simulationrun 15, simulationrun 31, simulationrun 55 and simulationrun 65

silomoddata20.mat contains simulationrun 77, simulationrun 78, simulationrun 79, simulationrun 80

... etc.

When I calculated the MFR_mod of every silomoddata.mat file I can't just simply combine them into a matrix. The orde isn't right. That was what I am trying to achieve here, a matrix of MFR_mod in the correct order of 1 until 81.

Answer 3

[Mathieu NOE]

hello

seems all your mat file contains the same size of data

so the modification of you r code is fairly simple - if I understand what you want :

MFR_tot = [];

for ii = 1:20
    data = load(['silomodresults',num2str(ii),'.mat']);
    MFR_tot = [MFR_tot; data];  % concatenation
end

    

[Tessa Kol]

Dear Mathieu,

The mat file do not have the same size of data. For example,

silomodresults5.mat contains: MFR_mod5, MFR_mod11, MFR_mod20, MFR_38, MFR_81 (5 values)

silomodresults6.mat contains: MFR_mod12, MFR_mod15, MFR_mod31, MFR_mod55, MFR_mod65 (5 values)

silomodresults20.mat contains: MFR_mod77, MFR_mod78, MFR_mod79, MFR_mod80 (4 values)

Also, I want them also in acending order. I tried to do this manually as explained in my post.

[Mathieu NOE]

ok , now I understand

is there any possibility for you to generate dummy mat files , but much smaller. I'd like to try to solve it

I think the key thing is to extract the field (variables) names and then re order the whole thing based on the numeric content of the variable name

That shoudn't be too difficult for someone fluent in string / structure processing (I don't mean me only !)

I'll give it a try tomorrow if you can send me some dummy mat files in between

[Tessa Kol]

Dear Mathieu,

I compressed silomoddata5.mat silomoddata6.mat and silomoddata20.mat into a zip file. You can download this using the link below (about 4 GB).

https://sendgb.com/xPkYbUa902H

Is this oke?