MATLAB Answers

Issue with ncread when _FillValue, add_offset, and scale_factor are present

98 views (last 30 days)
I have run into an issue when trying to concatenate multiple netcdf files with the same variables into one file, combining these variables.
First, I create a new netcdf file using the header information from one of the files I wish to concatenate using ncinfo and ncwriteschema, changing the length of the variable along which I wish to concatenate to unlimited.
When I read a variable that does not have the attributes _FillValue, add_offset, and scale_factor using ncread, the variable reads into my workspace as I would expect, and writes as expected into the new file.
However, when I read a variable that does have those attributes, the variable reads into my workspace seemingly without applying those attributes, contrary to ncread's documentation. I checked this using netcdf.getVar, and the results are the same between the two methods, which seems to indicate that ncread is failing to apply the attributes.
Strangely, however, when I use ncwrite to store the variable data in the new netcdf file, and then read it back in with either ncread or netcdf.getVar, I get something different! The variable data now oscillates between the _FillValue and -1*_FillValue.
Hypothesis: To me, this seems to indicate that ncwrite is trying to apply the attributes, but since ncread or netcdf.getVar did not apply them inversely, it's now applying them a second time! This causes the values to be larger in magnitude than the _FillValue, and confuses the netCDF file.
An easy workaround would be to apply the attributes myself using ncreadatt and ncwriteatt, but I'm not sure my hypothesis is correct, so I'm hesitant to brute force the process in that way.
Working on attaching code and small enough netCDF files, will update when those are ready.
UPDATE: netCDF files are uploaded as "netcdf_files.zip", matlab code uploaded as "forumpost_ncread.m".

  0 Comments

Sign in to comment.

Accepted Answer

meghannmarie
meghannmarie on 2 Dec 2020
Edited: meghannmarie on 2 Dec 2020
Ncread is working, put a breakpoint on line 108. When you use ncread and have a scale factor/add offset, it returns a double. In your code, you are taking your double variable and recasting it an int16 which is turning everything into integers. You need to delete lines 108 and 109.
vname = info.Variables(kn).Name;
var = ncread(fget,vname); %variable that is double precision when there is a scale factor/add offset
dtype = info.Variables(kn).Datatype;
var = cast(var,dtype); %You are now recasting double to int16, delete this

  4 Comments

Show 1 older comment
meghannmarie
meghannmarie on 2 Dec 2020
It looks like it made my files correctly. Only thing I see wrong is it puts February before January. To fix that, you need to sort your flist variable.
What problem are you still having?
When I try this the numbers seem to be correct:
files = {'ECMWF_1hr_2018_Jan.nc';'ECMWF_1hr_2018_Feb.nc'};
for f = 1:numel(files)
d = ncread(file,'u10',[1 1 1],[1 1 1])
ncid = netcdf.open(file);
varid = netcdf.inqVarID(ncid,'u10');
i = netcdf.getVar(ncid,varid,[0 0 0],[1 1 1]);
add_offset = netcdf.getAtt(ncid,varid,'add_offset');
scale_factor = netcdf.getAtt(ncid,varid,'scale_factor');
d == double(i) * scale_factor + add_offset
netcdf.close(ncid)
end
Sam Kastner
Sam Kastner on 2 Dec 2020
Sorry for the vagueness! I'm still having issues with ncread, but using netcdf.getVar and its accomplices seems to work just fine. I'll just have to do the offset/scale factor stuff manually as you do above. Thanks for the help!
meghannmarie
meghannmarie on 3 Dec 2020
If you add the lines below to your code, you will see ncread is working just fine. Apply the offset and scale factor and then look at your comparison pictures. Without applying those, you are comparing apples to oranges (int16 numbers to doubles).
ncid = netcdf.open(flist(1).name); % open original netcdf file for reading
varid = netcdf.inqVarID(ncid,'u10'); % get wind speed variable id, should be 3
u_longway = netcdf.getVar(ncid,varid); % get wind speed variable data
add_offset = netcdf.getAtt(ncid,varid,'add_offset');
scale_factor = netcdf.getAtt(ncid,varid,'scale_factor');
u_longway = double(u_longway) * scale_factor + add_offset;
netcdf.close(ncid);
ncid = netcdf.open(fname); % open new netcdf file for reading
varid = netcdf.inqVarID(ncid,'u10'); % get wind speed variable id, should be 3
u_longway_new = netcdf.getVar(ncid,varid); % get wind speed variable data
add_offset = netcdf.getAtt(ncid,varid,'add_offset');
scale_factor = netcdf.getAtt(ncid,varid,'scale_factor');
u_longway_new = double(u_longway_new) * scale_factor + add_offset;
netcdf.close(ncid);

Sign in to comment.

More Answers (0)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!