How can I merge different .nc files related to different time steps?

Data source: HYCOM

Context: Listed below are the attributes of data for a day at time step 12 am. Let's say I have data of Jan at different timesteps. And I have to merge them all into a single file to get variables velocity u and velocity_v sizes as [176,239,40,248] (different files data of a single month merged) as variables are captured for every 3 hours/temporal resolution is 3 hours. Please help with the code?

[Mathieu NOE]

have you already started something ?

[bhakti]

Thanks for your response. I tried something using Xarray in Python. I am not quite sure how to go about it in Matlab?

[Mathieu NOE]

I could probably try to start something but I need some nc files

[bhakti]

Hello Mathieu,

Thank you for your willingness to help and I am sorry for not pinning the files here.

Data pertaining to a month are attached here.

https://drive.google.com/drive/folders/1m_KAx2iKswTjW0xwc4ni6lh-txFLUK2k?usp=sharing

In case the file size seems larger to work with, here are files for a week : https://drive.google.com/drive/folders/1qWymw9kpCSC73KuWMtiOy9SVDbxat8th?usp=sharing

Thanks in advance!

[Peter Perkins]

Do you ultimately want two 176-by-239-by-40-by-248 arrays (or maybe one 176-by-239-by-40-by-248-by-2 array), or do you want one array that is (176*239*40*248)-by-6, where columns 1:4 of the latter are values for time, depth, lat, and lon, and columns 5:6 are values of velocity at the corresponding time/depth/lat/lon combination for each row?

I.e., in fewer dimensions,

>> x
x =
      0.81472      0.91338       0.2785
      0.90579      0.63236      0.54688
      0.12699      0.09754      0.95751

or

>> y
y =
            1            1      0.81472
            2            1      0.90579
            3            1      0.12699
            1            2      0.91338
            2            2      0.63236
            3            2      0.09754
            1            3       0.2785
            2            3      0.54688
            3            3      0.95751

Your answer almost certainly depends on what you need to do with your result.

[bhakti]

Hello Peter,

Thanks for your response.

I will use the results to make hdf5 files.

I need the velocities at their corresponding lon/lat/depth/time combination for each row. That is 2 arrays namely, u_velocity[176,239,40,248] and v_velocity[176,239,40,248].

Thanks in advance!

Answer 1

[Mathieu NOE]

hello @bhakti

here you have your u and v velcocity data stored in 4D arrays u_out and v_out as requested

I tried my code with the 7 days data nc files and it seems to work fine (size is in this case [176,239,40,56] )

also I often use this Fex submission to be sure the files are sorted correctly (what the regular dir does not do well)

Natural-Order Filename Sort - File Exchange - MATLAB Central (mathworks.com)

                     
fileDir = pwd; % current directory (or specify which one is the working directory)

S = dir(fullfile(fileDir,'*.nc')); % get list of data files in directory
S = natsortfiles(S); % sort file names into natural order , see : 
           %(https://fr.mathworks.com/matlabcentral/fileexchange/47434-natural-order-filename-sort)

for k = 1:numel(S)
    filename = S(k).name; 
    % Load in a few selected variables
    u = ncread(filename,'water_u');
    v = ncread(filename,'water_v');
    % concatenate data
    u_out(:,:,:,k) = u;
    v_out(:,:,:,k) = v;
end

[bhakti]

Hello Mathieu,

Thank you for your swift response. I tried and it worked but the sizes of u_out and v_out seem to be coming as [1 5].

[Mathieu NOE]

hello again

no, look here (arrows)

u_out and v_out are 4D arrays with the correct dimensions

what you are talking about is something else

how did you generated size_u and size_v ?

[bhakti]

Hello Mathieu,

Thanks again.

I did size(u_out) and size(v_out).

But the code achieved the mandate. Thank you for it.

[Mathieu NOE]

hello again

according to your screenshot the [1,5] is the dimension of the vector size(u_out) not size(u_out) itself

but on my side size(u_out) has only dimension 1 by 4 and not 1 by five

size_u = size(u_out)

size_u = 176 239 40 56

whos

Name Size Bytes Class Attributes

N 1x1 8 double

S 56x1 45344 struct

ans 1x4 32 double

fileDir 1x45 90 char

filename 1x14 28 char

k 1x1 8 double

size_u 1x4 32 double

u 176x239x40 13460480 double

u_out 176x239x40x56 753786880 double

v 176x239x40 13460480 double

v_out 176x239x40x56 753786880 double