Average data from multiple csv files

Hello Guys,

I have a bunch of CSV files, attached just 4 files.

I need to read all CSV files and then average data in the columns 3 and 4 in each file and then create a new CSV files with only average values from all files.

Can someone suggest what is the best way to get average data from multiple CSV files in one file?

I would need to run a loop in my script because there are more than 100files that I need to process.

Thanks for the help in advance.

Answer 1

[millercommamatt]

Here's an example of how to approach this. You'll have to make this work with your specific data and functions. You can do this using a parfor loop is speed is an issue and you design things correctly.

%% pseudo code below

% get a list of your files
list_of_files = dir('/path/to/files/file_prefix*file_extension'); % * = wildcard

% create an array of NaNs with a number of rows matching the number of files
% and a number of columns for however many variables you want to store information on 
storage_array = NaN(length(list_of_files),num_vars);

% you'll want to keep track of which file you're working on so we can just
% count then and we'll start at 1
active_file_num = 1;

% loop through each file
for file = list_of_files % beware syntax here, it's datatype dependent
    
    % read your files
    file_contents = read_file(file); %probably readtable
    
    % get whatever stats you want; write a function to do this for a single
    % file's data
    stats = function_that_gets_stats(file_contents);
    
    % put those stats into the correct row of your storage array
    storage_array(active_file_num, :) = stats;
    
    % increment up your file counter so that the next time through the loop
    % you'll work on the next file
    active_file_num = active_file_num +1;
end

% write out the contents of your storage array to a shiny new CSV
writetable(storage_array, '/path/to/saved/data/your_stats.csv');

Please read in the documentation on how to work with tables and the details of specifying for loops. This syntax isn't exact. I'm assuming you're aware of how to deal with the output of function like dir, that you can start a loop using something like a cell or string array insted of a numeric array, and that you can work with tables well enough to pass just the data you want to whatever stat function you want to work with.