Detrending still leaves a slope
9 views (last 30 days)
I am attaching two line profile examples for my case. After using detrend, I obtain the profile given in 'detrend_lineprofile' photo for the original data shown in 'lineprofile' photo. Although detrending seems it is working, still there is a slope towards the end of the profile after the detrend.
I want to get rid of this slope and bring the base of the profile to an even, flat condition. Is there a way to get rid of this slope after detrending?
Thanks in advance for all the help.
Star Strider on 7 Sep 2023
Edited: Star Strider on 7 Sep 2023
This is the best I can do with your data.
A reasonably robust approach is —
LD = load('profile_data.mat')
Z = LD.topo_z_data
y = Z(:,5);
L = size(Z,1);
x = linspace(0, L-1, L).'; % Assume Column Vectors
Lva = islocalmin(y, 'FlatSelection','center'); % All Points Meeting Criteria
NrLv = nnz(Lva)
Lvp = islocalmin(y, 'FlatSelection','center', 'MinProminence',0.5); % Define 'Outliers'
NrLvp = nnz(Lvp)
Lv = Lva & ~Lvp; % Delete 'Outliers'
NrLv = nnz(Lv)
xsel = x(Lv);
ysel = y(Lv);
% y = sum(x*0.02 + rand(1,5)*100.*exp(-(x-randi([100 900],1,5)).^2/1E+3) + randn(size(x))/10, 2); % Create Data
% plot(xsel, ysel, '.r')
% plot(x(Lvp), y(Lvp), '.g')
B = [xsel(1) 1; xsel(end) 1] \ ysel([1 end]); % Linear Regression Of End Points (Or Suitably Representative Points)
Baseline = [x ones(size(x))] * B; % Calculate Baseline Cirrection
ycorr = y - Baseline; % Subtract Correction
EDIT — (7 Sep 2023 at 15:56)
Changed code to use provided data.
More Answers (1)
John D'Errico on 7 Sep 2023
Edited: John D'Errico on 7 Sep 2023
It depends on how you define detrend. Clearly, you have a different definition from me, at least, as well as the author of the function detrend.
On average, there is no slope. No overall trend. Yes, if you look at the baseline of that curve, there is an overall trend. But that is not how detrend is defined to work. Of course you can always find some subset of points on any scattered curve that will appear to have a trend.
My guess would be, IF you passed a straight line fit (using, for example, polyfit) through the detrended curve, the slope of that fit would be exactly zero.
So, called with only one argument, detrend will remove a linear trend from the overall curve. For example, since you give no data...
t = linspace(0,1,100);
y0 = exp(t) + randn(size(t))/10;
y1 = detrend(y0);
So a linear trend has been removed. I might have done this instead:
P1 = polyfit(t,y0,1);
y1b = y0 - polyval(P1,t);
As you can see, the two curves are identical (since the difference is zero.) As well, if you read the help for detrend, it says nothing at all about finding, AND removing a baseline trend.
Finally, let me test my claim that the slope of the detrended curve will be zero. The first coefficient out of polyfit should be zero, to within floating point trash.
As well, the second coefficient is also zero, since detrending also removed any constant, DC component in the curve.
Anyway, feel free to define detrending in any way you wish. But don't expect software to read your mind. And if that day ever comes when software can read my mind, I'm heading for the hills anyway. But then, it probably knows that already.