How to create a loop that runs a function and save output through subfolders in a directory?

I am able to run my code only from here C:\Users\39218\Desktop\PROJECT\TD\P2\S6.

Then I have to move in P2S#....then P13S# (there is no a sequential order in the name)

My code computes 4 matrices (indeed there is also a plot where there are all subplots together) and save the output in the current folder.

What I want is to run my code from C:\Users\39218\Desktop\PROJECT. Furthermore, the code has to save those 4 matrices and the big plot for each one S# in a new folder (empty) with the associate name P#S#

FolderStructure= dir ('*.c3d');  %start from folder PM
con= struct2cell(FolderStructure);
myFiles =con(1,:)'
Phases_I=[];
Phases_O=[];
GR_I=[];
GR_O=[];
for z =1:length(myFiles)
    name = string(myFiles(z))
%     do something
    Phases_I=[Phases_I; Times_Tot_I]
    Phases_O=[Phases_O; Times_Tot_O]
    GR_I=[GR_I;GR_Tot_I]
    GR_O=[GR_O;GR_Tot_O]
end
% subplot
save('Phases_I.mat','Phases_I')
save('Phases_O.mat','Phases_O')
save('GR_I.mat','GR_I')
save('GR_O.mat','GR_O')

[Stephen23]

Do NOT follow any advice to use cd. Using cd is slow and pointless because all MATLAB functions that read/write data files accept absolute/relative filenames. Using relative/absolute filenames is more efficient and easier to debug.

https://www.mathworks.com/help/matlab/matlab_env/specify-file-names.html

The MATLAB documentation clearly states "Avoid programmatic use of cd, addpath, and rmpath, when possible. Changing the MATLAB path during run time results in code recompilation."

source: https://www.mathworks.com/help/matlab/matlab_prog/techniques-for-improving-performance.html

Answer 1

[Stephen23]

I replicated your file structure as I understood your explanation, it looks like this:

Obviously I do not have your data files, so I just filled the S# folders with a few CSV files to test this code with. You will need to adapt the code to suit your exact needs and file structure! Also note that if you want the files imported in alphanumeric order then one solution is to download my FEX submission natsortfiles and replace all of the instances of

setdiff(...)

with

natsortfiles(setdiff(...))

Note that the data Phases_I, etc. are collected into the non-scalar structure S3:

https://www.mathworks.com/help/matlab/matlab_prog/access-multiple-elements-of-a-nonscalar-struct-array.html

All folders with names starting with P or S located in the relevant subdirectories are processed.

D = '.'; % path to the parent directory
S1 = dir(fullfile(D,'P*'));
C1 = setdiff({S1([S1.isdir]).name},{'.','..'});
for k1 = 1:numel(C1) % loop over P# directories
    S2 = dir(fullfile(D,C1{k1},'S*'));
    C2 = setdiff({S2([S2.isdir]).name},{'.','..'});
    for k2 = 1:numel(C2) % loop over S# directories
        S3 = dir(fullfile(D,C1{k1},C2{k2},'*.c3d'));
        for k3 = 1:numel(S3) % loop over files
            F = fullfile(D,C1{k1},C2{k2},S3(k3).name);
            M = csvread(F);
            ... do whatever with your data
%           S3(k3).Phases_I = Times_Tot_I;
%           S3(k3).Phases_O = Times_Tot_O;
%           S3(k3).GR_I = GR_Tot_I;
%           S3(k3).GR_O = GR_Tot_O;
        end
%       Phases_I = vertcat(S3.Phases_I); % See note about NATSORTFILES and file order!
%       Phases_I = vertcat(S3.Phases_O);
%       GR_I = vertcat(S3.GR_I);
%       GR_I = vertcat(S3.GR_O);
%       do whatver with those variables...
        % plot, save, etc.
        % Make new subdirectory:
        new = fullfile(D,[C1{k1},C2{k2}]);
        mkdir(new)
%       saveas(fgh,fullfile(new,'image.png'))
    end
end

[matteo bottoni]

• The path is like DATA P2 S3 LOAD file.ext
• S7 LOAD file.ext
• P11 S4 LOAD file.ext
• P7 ..... ...... .....
• ..... ...... ........

cd gives me this problem (but I wrongly am not considering folder "LOAD") :

Error using cd

Cannot CD to C:\Users\39338\Desktop\PROJECTMATLAB\DATA\P2S9_17112017_US_W-_3.c3d (Name is nonexistent or not

a directory).

Indeed the code computes data only for the first P# and it can't switch.

Thank u all so much however.

clc;
clear all;

parentDir=pwd;
FolderStructure= dir ('C:\Users\39338\Desktop\PROJECTMATLAB\DATA');
FolderCell = struct2cell(FolderStructure);
myFiles = FolderCell(1,:)';
Phases_I=[];
Phases_O=[];
GR_I=[];
GR_O=[];
for z =3:length(myFiles)
    name = string(myFiles(z));
    dr=['C:\Users\39338\Desktop\PROJECTMATLAB\DATA\',char(name)];
    fs=dir(dr);
    cd(dr);
    fs=struct2cell(fs);
    subFl= fs(1,:)';

    for k =3:length(subFl)
        subd=pwd;
        cd(subFl{k})
        FolderStructure= dir ('*.c3d');
        con= struct2cell(FolderStructure);
        myFiles =con(1,:)'
        for zz =1:length(myFiles)
            name = string(myFiles(zz))
            %load file
            
            Phases_I=[Phases_I; Times_Tot_I]
            Phases_O=[Phases_O; Times_Tot_O]
            GR_I=[GR_I;GR_Tot_I]
            GR_O=[GR_O;GR_Tot_O]
        end
        
        save('Phases_I.mat','Phases_I') % and others
        cd(subd);
    end
end

[Stephen23]

@matteo bottoni: you appear to have copied some inefficient and unreliable code from somewhere else, it certainly has nothing to do with my answer, so I am not really sure why you are quoting it in a comment to my answer.

If your actual path includes an extra subdirectory named LOAD this is trivial to include in my code:

        S3 = dir(fullfile(D,C1{k1},C2{k2},'LOAD','*.c3d'));
        ...
            F = fullfile(D,C1{k1},C2{k2},'LOAD',S3(k3).name);

And the rest will remain unchanged. I just added the LOAD subdirectory to my test file structure:

and my code worked without any error, reading all of the files (exactly as expected).

[matteo bottoni]

I replied you to let you know that there is no a sequential order in my path. I showed the code because I don't know if the error about cd is the same which you told (Only to understand it)

However into load file I have built a new a difficult code for my experience, I am only a beginner unfortunately.

[Stephen23]

"I replied you to let you know that there is no a sequential order in my path."

My code does not assume anything about the name order of the files or folders.

My code uses dir to get the folder names and filenames, (just as your inefficient and fragile code does), and dir does not care about what order the filenames or folder names have. If you want them in a particular order, then you have to sort them yourself (which is why I added the note about natsortfiles to my answer).

"...if the error about cd is the same which you told"

The error appears to be caused by the fact that you have not taken the LOAD subdirectory into account.

Answer 2

[Guru Mohanty]

Hi, I understand you are trying to access subfolders and write data into it. You can access the content of subfolder by these following commands.

To move into a different directory use cd, To create new directory use mkdir, and to get parent directory pwd can be used.

Here is a sample code for it.

clc;clear all;
parentDir=pwd;
FolderStructure= dir ('TD');  %start from folder PM
con= struct2cell(FolderStructure);
myFiles =con(1,:)';
Phases_I=[];
Phases_O=[];
GR_I=[];
GR_O=[];
for z =3:length(myFiles)
    name = string(myFiles(z));
    dr=['TD\',char(name)];
    fs=dir(dr);
    cd(dr);
    fs=struct2cell(fs);
    subFl= fs(1,:)';
    for k =3:length(subFl)
        subd=pwd;
        cd(subFl{k})
        save('Phases_I.mat','Phases_I')
        save('Phases_O.mat','Phases_O')
        save('GR_I.mat','GR_I')
        save('GR_O.mat','GR_O')
        cd(subd);
    end
    mkdir 'SOUT'
    cd(parentDir);
end

[Stephen23]

Bugs and other features of this code:

1- Incorrectly assuming that the first two elements of the structure returned by dir are '.' and '..'. In fact it is easy to demonstrate many file/folder names (with quite common characters in them) can be returned before those, as has been discussed many times before on this forum:

https://www.mathworks.com/matlabcentral/answers/484430-using-struct-dir-selpath-what-do-and-mean-as-struct-name#answer_395754

https://www.mathworks.com/matlabcentral/answers/40949-omitting-pointers-when-listing-folder-contents#answer_50568

https://www.mathworks.com/matlabcentral/answers/484747-index-for-subfolders-without-pdf-files#comment_755117

https://www.mathworks.com/matlabcentral/answers/13978-list-of-files-sorting#answer_19278

As Walter Roberson wrote in that last link: "In short: if your code assumes that '.' and '..' are the first two entries in a directory, your code has a bug (even in MS Windows). If your code assumes that directory entries are returned in any sorted order, your code has a bug (in all OS.)"

2- Absolutely no checking if the names returned by dir are files or folders.

3- Inefficient cd to access data files, rather than absolute/relative filenames:

https://www.mathworks.com/matlabcentral/answers/484604-what-should-i-use-instead-of-cd#comment_754766

https://www.mathworks.com/matlabcentral/answers/471968-error-with-changing-directory-in-for-loop#answer_383457

https://www.mathworks.com/matlabcentral/answers/112086-matlab-slow-when-too-many-files-in-directory#answer_141542

4- String concatenation rather than fullfile.

5- Superfluous variables defined, never used.

6- Superfluous type conversions, e.g.

    name = string(myFiles(z));
    dr=['TD\',char(name)];

In order to get the character vector out of a cell array of character vectors the author first converts one cell of the cell array to string type and then converts that string back to character vector.

https://www.mathworks.com/help/matlab/matlab_prog/access-data-in-a-cell-array.html

Other superfluous data type conversion: turning all of the structures returned by dir into cell arrays. Why? What possible purpose does that serve? Do TMW employees no longer learn how to access data in structures?

7- Processes the sub-directories P2 - P13 in ASCIIbetical order (or whatever order is returned by the OS), not numeric order. Not critical for the code as given, but when the OP decides to merge/concatenate/... the data of multiple files, then this will be easily overlooked.