I have a bunch of named test data with some missing specimens. Want to perform mathematical operations on said data automatically.

Hello,

I have 16 plates of material, each of which is cut into 10 specimens. I collected the data from an Instron machine, and as such labeled each test folder as the material name followed by the plate number and then the specimen name followed by the specimen number I.E. Gold1_S1, Gold1_S2, Gold2_S3, etc.

I readtabled this entire 160 folder directory into my matlab as a 160ish row struct of many thousands of row cells, and I made sure that each test is named according to its folder name I gave it when setting up the test. I also sort_nat them so they are in order from Gold1_S1 to S10, and then Gold2_S1 to S10 and so on until Gold16_S10.

Problem is, some of the specimens were missing when I was testing them, so it means that I cannot use a nice simple for loop for processing each plate of data separately because some plates have only 7 specimens. Also, some of the specimens on the plates are different from the other specimens, so it further complicates things.

I am trying to group and process these specimens based on the name I gave them when I was testing them, but I can't figure out how to do it efficiently.

My best case scenario would be to base the calculations on the names of the plates and specimens. For instance if I could figure out how to tell which plate the specimen belong to it would be easy, as each data set is assigned to the name of the plate and specimen. I was thinking an if statement where if specimen belongs to plate 1, then take all those specimens and average their shear stress, and so on for each plate. I just cannot figure out how to make matlab read which plate the specimen belongs to.

TL;DR Have many plates which specimens are cut out of. Have data assigned to variables that are named according to plate and specimen. Want to automatically determine which plate and specimen the data comes from, and perform math based on that determination.

Thank you, and please be gentle,

Luck

Answer 1

[Mathieu NOE]

hello

this is my suggestion ... I created a list of files with missing files (could be a different scenario for each group)

this is a screenshot of the excel file I used to create a list with missing files (the excel file is attached for convenience)

then the first task is to read all the files and store their data in a cell array indexed with group and sample

then in a second loop you can perform the data analysis group by group , whatever the number of files available for each group.

we could also use structures instead of cell arrays if you prefer.

the plot shows (dummy) data for each group.

you can see that the data plotted do correspond to the number of valid files per group (legend accordingly)

from there you can easily expand and build your own data post processing

hope it helps

clc
clearvars

%% some dummy file list created ad - hoc
datafiles = readcell('liste.xlsx');
datafiles = (datafiles(:)); % put list in column 
datafiles = datafiles(cellfun(@ischar,datafiles)); % remove empty cells (missing file)


%% step 1 : read all files and store them in a cell array (one cell = data from one file)
for ck = 1:numel(datafiles)
    thisFile = string(datafiles(ck));
    numArray = extractNumFromStr(thisFile); % extract the two digits (first is the group, second is the sample)
    group = numArray(1);
    sample = numArray(2);
    my_data_array{group,sample} = group + sample*rand(10,1); % my dummy data generator ; import your data here and store 
        
end

%% step 2 : do post-processing by groups 
for ci = 1:group  % first loop on "group"
    my_group_data = my_data_array(ci,1:sample);
    lgk = 0;
    legendStr = [];
    
    for cj = 1:sample  % inner loop on sample files for same group
        
            my_sample_data = my_group_data{cj};
            
            % some code here to produce data to be plotted 
            if isempty(my_sample_data);
                data2plot(:,cj) = NaN*zeros(10,1); % for the plot in each "group" window
            else
                data2plot(:,cj) = my_sample_data; % for the plot in each "group" window
                lgk = lgk +1;
                legendStr{lgk} = ['Sample #' num2str(cj)];
            end

    end
    
    figure(ci)
    plot(data2plot);
    legend(legendStr);
    title(['Group : ' num2str(ci)])
    
end

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
function numArray = extractNumFromStr(str)
    str1 = regexprep(str,'[,;=_]', ' ');
    str2 = regexprep(regexprep(str1,'[^- 0-9.eE(,)/]',''), ' \D* ',' ');
    str3 = regexprep(str2, {'\.\s','\E\s','\e\s','\s\E','\s\e'},' ');
    numArray = str2num(str3);
end

[Mathieu NOE]

hello

problem solved ?

[Luck Haviland]

Hi Matthieu. Thank you for putting in the time to try and solve this problem. I ended up using some form of omitnan so that way I would not have to remove the missing spaces, the for loops would instead just ignore them.

I am uncertain if your method works but I will assume that it does.

Thanks again,

Luck

[Mathieu NOE]

hi Luck

ok - good luck for the future

bye