How can I read many netcdf files with complex names by using loop in MATLAB?

Hello Guys,

I would like to read many Netcdf files with this type of names:

3B-HHR.MS.MRG.3IMERG.20140401-S000000-E002959.0000.V03D.HDF5.nc?precipitationCal[960:969][986:992],lat[986:992],lon[960:969]

3B-HHR.MS.MRG.3IMERG.20140401-S003000-E005959.0030.V03D.HDF5.nc?precipitationCal[960:969][986:992],lat[986:992],lon[960:969]

3B-HHR.MS.MRG.3IMERG.20140401-S010000-E012959.0060.V03D.HDF5.nc?precipitationCal[960:969][986:992],lat[986:992],lon[960:969]

As you see the first number shows the date 20140401 and then time in order to hour, minute and second.

I need to use a loop to read them in order to sorting time!!

these three files as a sample are attached.

Thank you in advance for considering this question.

Regards,

Answer 1

[Kelly Kearney]

Are you reading these files locally? Or is it via OPeNDAP? If the files are local, it may be easiest to query all the filenames using dir, and then sort by the date portion of the files.

If you can't do that, then do you know how many files to expect? Start date, end date, and time interval between files? Do you know the pattern to the -E portion of the file name?

-Kelly

[ehsan]

Dear Kelly,

I am reading the files locally. There are 32208 file with time interval 30 minutes (29 minutes and 59 seconds). Started from 01.04.2014 at 00:00:00 to 31.01.2016 at 23:59:59.

[ehsan]

Actually I need to read precipitation parameter (the name of the parameter in the file is "precipitationCal") in these files.

pre = ncread('3B-HHR.MS.MRG.3IMERG.20140401-S000000-E002959.0000.V03D.HDF5.nc?precipitationCal[960:969][986:992],lat[986:992],lon[960:969]','precipitationCal');

but not only for one file but also for all range of the files which I have in order to date.

[Kelly Kearney]

Assuming the files are all in the same folder, and that the date is the only thing changing in the file name pattern, then they should be returned in sorted order by dir. You can pull out the date portion of the string via regexp if you want to make sure of that, though:

% Get file list

pth = '~/Downloads/'; % Replace with your path
Files = dir(fullfile(pth, '3B-HHR*'));
files = arrayfun(@(X) fullfile(pth, X.name), Files, 'uni', 0);

% Extract date portion of file name

tstr = regexp(files, '3IMERG\.(\d*-S\d*)-E', 'tokens', 'once');
tstr = cat(1, tstr{:});

% Sort by date

date = datetime(tstr, 'InputFormat', 'yyyyMMdd-''S''HHmmss');
[date,isrt] = sort(date);
files = files(isrt);

% Figure out size of precipitationCal variable

Info = ncinfo(files{1});
isprecip = strcmp({Info.Variables.Name}, 'precipitationCal');
sz = Info.Variables(isprecip).Size;

% Read data from files 

data = zeros([sz length(files)]);
for ii = 1:length(files)
    data(:,:,ii) = ncread(files{ii}, 'precipitationCal');
end

[ehsan]

Dear Kelly,

Thank you for your consideration.

Your assumptions are right. all files are in the same folder and only date-time are changing.

Unfortunately my MATLAB version is R2014a and doesn't have "datetime" function. Therefore, I give the error message.

Do you have any other idea?

[Kelly Kearney]

You can replace the datetime call with

date = datenum(regexprep(tstr, '-S', ''), 'yyyymmddHHMMSS');

(datenum is a little less flexible than datetime in parsing formats with extra characters, hence the regexprep to get rid of the non-date characters).