Extracting Data from Over 600 .nc Files...

Hi all!

Here's a tough one I'm hoping to receive some guidance on: I'm trying to average a specific variable over six hundred days. That is, I have 600 .nc files from different dates which I have to look into, extract data from, and ultimately work with.

The variable itself is specific humidity (qv), and it is of size 1 x 4, where each column represents: longitude, latitude, pressure level, and time of day (timestep). I have figured out how to read qv for all pressure levels at a specific location and a specific timestep using ONE .nc file, but I would like to automate the process to do this (i.e. read the variable) for all 600 days using all 600 .nc files. Any pointers or suggestions? Thanks in advance!!!

Here's what I have so far:

merra = ('MERRA2_100.inst6_3d_ana_Np.19800115.SUB.nc')
ncdisp(merra)
qv = []
latitude = ncread(merra, 'lat')
longitude = ncread(merra, 'lon')
time = ncread(merra, 'time')
level = ncread(merra, 'lev')
for i = 1:size(level)
    i
    z = ncread(merra, 'QV', [130, 38, i, 1], [1, 1, 1, 1])    %the 130 is longitude, the 38 is latitude...while 1 is timestep
    qv = [qv; z]
end
qv 

%the variable 'qv' is an array of size 21 x 1, representing values of
%specific humidity at 21 different pressure levels. 

Answer 1

[Jan]

See: FAQ: How to process a sequence of files

Folder = 'C:\Your\Data\Folder';
FileList = dir(fullfile(Folder, '*.nc'));
for iFile = 1:numel(FileList)
   merra = fullfile(FileList(iFile).folder, FileList(iFile).name);
   ... your code comes here
end

Do not reset qv to the empty matrix, but pre-allocate the output and use 2 indices:

Folder = 'C:\Your\Data\Folder';
FileList = dir(fullfile(Folder, '*.nc'));
qv = zeros(21, numel(FileList));
for iFile = 1:numel(FileList)
   ...
   for i = ...
       qv(i, iFile) = ncread(merra, 'QV', [130, 38, i, 1], [1, 1, 1, 1]);
   end
end

This is tricky:

for i = 1:size(level)

size() replies a vector. Then 1:size(level) uses the 1st element of the size vector and ignores the rest. If this is really wanted, prefer to do this explicitly:

for i = 1:size(level, 1)

If you mean the number of elements, use

for i = 1:numel(level)

[Michelle De Luna]

@Jan: Thank you for taking the time to look into this! Your response was very helpful! It makes sense now, the only issue then is that I receive an error message reading the following: "Unable to perform assignment because the size of the left side is 1x1 and the size of the right side is 1x21." I know this is a follow-up question and understand if you don't have time to respond, but any suggestions would be greatly appreciated. (Please note, I changed the range of my latitude to be [130, 38, i, 1] [1, 21, 1, 1]...) Thanks!

folderData = 'C:\Your\Data\Folder'
fileList = dir(fullfile(folderData, '*.nc'))
qv = zeros(21, numel(fileList))

for iFile = 1:numel(fileList)
    merra = fullfile(fileList(iFile).folder, fileList(iFile).name)
    ncdisp(merra);
    lon = ncread(merra, 'lon');
    lat = ncread(merra, 'lat');
    time = ncread(merra, 'time');
    level = ncread(merra, 'lev')
    for i = 1:numel(level)
        i
        qv(i, iFile) = ncread(merra, 'QV', [130, 38, i, 1], [1, 21, 1, 1])
    end
end

[Jan]

Then expand qv:

qv = zeros(21, numel(fileList), 21);
...
    qv(i, iFile, :) = ncread(merra, 'QV', [130, 38, i, 1], [1, 21, 1, 1])

[Michelle De Luna]

@Jan: Thank you! You've been very helpful. Wishing you a great week!