What is the best smoothing procedure to calculate differentials for a large number of data points (x,y)?

7 views (last 30 days)
There is a set of data (X,Y) independently obtained from an experiment. Plotting Y vs X looks like this. There are ~114000 data points.
Upon zooming in, you can see the scatter in the data. Because of this when the slope (instantaneous differential) is calcultaed using diff(Y)./diff(X), the scatter is magnified and the trend I'm looking for is completely lost.
For now, I used an arbitratry smoothing procedure using moving average to somewhat acquire the trend I'm looking for, which looks like this:
Could someone please suggest a more standard procedure that I could employ that would be statistically accurate? I have attached the dataset (X,Y).

Answers (1)

Bruno Luong
Bruno Luong on 6 Aug 2022
Edited: Bruno Luong on 6 Aug 2022
The Savitzky-Golay filter (moving polynomial fit) is a good filter that preserves decendly the slope.
There is a similar thread discussed not long ago, in short you fit data with appropriate model/tools 'depending on the characteristic of you data); then take the derivative of the model.
If I use my own free knots spline tool on your data I get this
load('example.mat')
pp=BSFK(X,Y); %FEX file
pp1=ppder(pp);
xq=linspace(min(X),max(X),513);
ydq=ppval(pp1,x)
plot(xq,ydq)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!