Using nanmean in retime when sampling intervals don't perfectly line up

I have a number of short time-period high resolution timetables that I want to retime to match a longer, lower resolution timeseries. I have done this using the following (contained within a for loop that also does some other stuff):

% Smoothed Obs
    a_Obs = Obs_Timetables.(fnm)(:,[2:3,14:15,27]);
    a_SmoothedObs = retime(a_Obs,time(min_time_idx:max_time_idx),@(x)mean(x,'omitnan')); % still showing final row as nan
    Obs_SmoothedObs.(fnm) = a_SmoothedObs; 

where Obs_Timetables is a structure containing the short, high resolution timetables, time is the datetime variable used to construct the low resolution timetable, and min_ (max_) time_idx is the index in time of the first (last) datetime value in the high resolution timetable. Having used both the format shown above and @nanmean in place of @x ... here, the retime step sets all values in the final row of my new short, low resolution timetables (a_SmoothedObs) to NaN. I think this is because the final time in the a_Obs and the time at time(max_time_idx) are not the same (with a_Obs.time(end) typically smaller than time(max_time_idx), although this is happenstance). I would like it to give me the nanmean as if a_Obs.time did extend to the time at time(max_idx), but all the values between that and the current final value were nan. Is there a way to do this that doesn't involve tacking an additional nan line onto the end of a_Obs and then removing it afterwards?

[Mathieu NOE]

hello

could you share the table (as mat file) or an extract of it ?

tx

Answer 1

[SAI SRUJAN]

Hi Victoria,

I understand that you are facing an issue with retiming high-resolution time series data to align with a lower-resolution timeline using the 'retime' MATLAB function.

When we retime high-resolution time series to match a lower-resolution, the last interval in the high-resolution series seems not to align perfectly with the lower-resolution series causing the 'retime' function to produce 'NaN' values for the final row because it doesn't find any corresponding data points within that final time period.

You can create a custom function that handles the aggregation and treats missing intervals, please go through the following code sample to proceed further,

% Define your custom aggregation function
function result = customNanMean(x)
    if isempty(x)
        result = NaN;
    else
        result = mean(x, 'omitnan');
    end
end

a_SmoothedObs = retime(a_Obs, time(min_time_idx:max_time_idx), @(x) customNanMean(x));

For a comprehensive understanding of the 'retime' MATLAB function, please go through the following documentation.

https://www.mathworks.com/help/matlab/ref/timetable.retime.html

I hope this helps!

[Victoria Dutch]

Thanks! I've copied the function you've written into my script and get the following error:

Error: File: CompareOutputs_Timeseries_Method2.m Line: 677

Column: 1

Function definitions in a script must appear at the end of the file. Move all statements after the "customNanMean" function definition to before the first local function definition.

Will this be resolved if I just save the function separately, or is there something else I need to do? If this is how to proceed, do I need to give the new file a specific name?

EDIT: Code runs when I save the function in a separate file, but the final line of a_Obs is still entirely nan.