Detrending still leaves a slope

9 views (last 30 days)
I am attaching two line profile examples for my case. After using detrend, I obtain the profile given in 'detrend_lineprofile' photo for the original data shown in 'lineprofile' photo. Although detrending seems it is working, still there is a slope towards the end of the profile after the detrend.
I want to get rid of this slope and bring the base of the profile to an even, flat condition. Is there a way to get rid of this slope after detrending?
Thanks in advance for all the help.
Bugrahan Guner
Bugrahan Guner on 7 Sep 2023
I just attached a data set for the case

Sign in to comment.

Accepted Answer

Star Strider
Star Strider on 7 Sep 2023
Edited: Star Strider on 7 Sep 2023
This is the best I can do with your data.
A reasonably robust approach is —
% imshow(imread('lineprofile.jpg'))
% imshow(imread('detrend_lineprofile.jpg'))
LD = load('profile_data.mat')
LD = struct with fields:
topo_z_data: [1000×10 double]
Z = LD.topo_z_data
Z = 1000×10
0.0501 0.0765 0 0 0 0.2352 0 0.1216 0 0 0 0 0.4114 0.5229 0.0092 0 0.5022 0.2099 0.3771 0.0217 0.4932 0.4469 0.4332 0.3709 0.2149 0.2916 0.3531 0.1412 0.2723 0.0516 0.3805 0.6880 0.4912 0.1714 0.5189 0.2788 0.2863 0 0.5265 0.0088 0.5829 0.5417 0.6285 0.6627 0.0938 0.3920 0.4706 0.1596 0.5727 0.4251 0.5005 0.6379 0.5792 0.4328 0.2421 0.5059 0.5528 0.2265 0.8142 0.4656 0.5559 0.8270 0.6569 0.8655 0.2935 0.2508 0.8058 0.2850 0.8548 0.6962 0.6962 1.0147 0.7112 0.6039 0.3564 0.3766 4.3116 0.4459 0.7278 0.7808 0.5026 0.9304 1.0840 0.8096 0.6392 0.9144 3.8141 0.8225 0.7197 0.7080 0.9621 0.6400 1.3384 1.0655 0.7736 0.6823 3.2333 0.7367 0.6273 0.8611
y = Z(:,5);
L = size(Z,1);
x = linspace(0, L-1, L).'; % Assume Column Vectors
Lva = islocalmin(y, 'FlatSelection','center'); % All Points Meeting Criteria
NrLv = nnz(Lva)
NrLv = 188
Lvp = islocalmin(y, 'FlatSelection','center', 'MinProminence',0.5); % Define 'Outliers'
NrLvp = nnz(Lvp)
NrLvp = 34
Lv = Lva & ~Lvp; % Delete 'Outliers'
NrLv = nnz(Lv)
NrLv = 154
xsel = x(Lv);
ysel = y(Lv);
% y = sum(x*0.02 + rand(1,5)*100.*exp(-(x-randi([100 900],1,5)).^2/1E+3) + randn(size(x))/10, 2); % Create Data
plot(x, y)
hold on
% plot(xsel, ysel, '.r')
% plot(x(Lvp), y(Lvp), '.g')
hold off
B = [xsel(1) 1; xsel(end) 1] \ ysel([1 end]); % Linear Regression Of End Points (Or Suitably Representative Points)
Baseline = [x ones(size(x))] * B; % Calculate Baseline Cirrection
ycorr = y - Baseline; % Subtract Correction
plot(x, ycorr)
title('Baseline Detrended')
EDIT — (7 Sep 2023 at 15:56)
Changed code to use provided data.
  1 Comment
Bugrahan Guner
Bugrahan Guner on 7 Sep 2023
Thank you so much, I will try it on the data as well. In the meantime I am attaching the data set but what you did is exactly what I am looking for. Thanks.

Sign in to comment.

More Answers (1)

John D'Errico
John D'Errico on 7 Sep 2023
Edited: John D'Errico on 7 Sep 2023
It depends on how you define detrend. Clearly, you have a different definition from me, at least, as well as the author of the function detrend.
On average, there is no slope. No overall trend. Yes, if you look at the baseline of that curve, there is an overall trend. But that is not how detrend is defined to work. Of course you can always find some subset of points on any scattered curve that will appear to have a trend.
My guess would be, IF you passed a straight line fit (using, for example, polyfit) through the detrended curve, the slope of that fit would be exactly zero.
help detrend
DETREND Remove a polynomial trend. Y = DETREND(X) removes the best straight-line fit linear trend from the data in vector X and returns the residual in vector Y. If X is a matrix, DETREND removes the trend from each column of the matrix. If X is a table or timetable, then DETREND operates on each variable separately. Y = DETREND(X,n) removes a polynomial trend with degree n. n = 1 by default. Setting n = 0 is equivalent to using the 'constant' option and setting n = 1 is equivalent to using the 'linear' option. Y = DETREND(X,n,BP) removes a continuous, piecewise polynomial trend according to the break points specified in BP. BP can be a vector of sample point values or a logical vector of length equal to the number of sample points, where true indicates the position of a break point. Y = DETREND(___,NANFLAG) specifies how NaN values are treated: "includemissing" / "includenan" - (default) NaNs are included when calculating the polynomial trend. If NaNs are present, then the trend and the corresponding output will be vectors of NaNs. "omitmissing" / "omitnan" - NaN values are omitted when calculating the trend. NaN values in the input produce corresponding NaN values in the output. Y = DETREND(___,'Continuous',TF) specifies whether the piecewise polynomial trend is required to be continuous. TF must be one of the following: true - (default) the fitted trend is continuous everywhere false - the fitted trend is not required to be continuous Y = DETREND(___,'SamplePoints',S) also specifies the sample points S associated with the data in X. S must be a floating-point, duration or datetime vector. S must be sorted and contain unique points. If the first input is a table, S can also specify a table variable. You can use S to specify time stamps for the data. By default, DETREND assumes the data is sampled uniformly at points S = [1 2 3 ... ]. Arguments supported only when first input is table or timetable: Y = DETREND(...,'DataVariables',DV) removes a polynomial trend from the data only in the table variables specified by DV. Other variables in the table not specified by DV pass through to the output without being operated on. The default is all table variables in X. DV must be a table variable name, a cell array or string array of table variable names, a vector of table variable indices, a logical vector, a function handle that returns a logical scalar (such as @isnumeric), or a table vartype subscript. The output table Y has the same size as the input table X. Y = DETREND(...,'ReplaceValues',TF) specifies how the detrended data is returned. TF must be one of the following: true - (default) replace table variables with the detrended data false - append the detrended data as additional table variables If the value of ReplaceValues is false, then the width of Y is the sum of the input data width and the number of data variables specified. Examples: % Remove a continuous, piecewise linear trend sig = [0 1 -2 1 0 1 -2 1 0]; % signal with no linear trend trend = [0 1 2 3 4 3 2 1 0]; % two-segment linear trend x = sig + trend; % signal with added trend y = detrend(x,1,5) % break point at 5th element % Remove a nonlinear trend sig = repmat(sig,1,2); trend = ((1:18)/18).^3; x = sig + trend; y = detrend(x,3); plot(sig); hold on plot(y) % Remove discontinuous linear trend with sample points t = -10:10; x = t.^3 + 6*t.^2 + 4*t + 3; bp = 0; y = detrend(x,1,bp,'SamplePoints',t,'Continuous',false); plot(t,x,t,y,t,x-y,':k') legend('Input Data','Detrended Data','Trend','Location','northwest') % Remove quadratic trend from the selected variable in a table t = (-4:4)'; trend = (t.^2 + 4*t + 3); sig = [0 1 -2 1 0 1 -2 1 0]'; x = sig + trend; T = table(t,trend,sig,x); T = detrend(T,2,"DataVariables","x","SamplePoints","t","ReplaceValues",false); stackedplot(T,"XVariable","t") See also TRENDDECOMP, MEAN, ISCHANGE, POLYFIT, POLYVAL Documentation for detrend doc detrend Other uses of detrend gpuArray/detrend iddata/detrend timeseries/detrend
So, called with only one argument, detrend will remove a linear trend from the overall curve. For example, since you give no data...
t = linspace(0,1,100);
y0 = exp(t) + randn(size(t))/10;
y1 = detrend(y0);
So a linear trend has been removed. I might have done this instead:
P1 = polyfit(t,y0,1);
y1b = y0 - polyval(P1,t);
ans = 0
As you can see, the two curves are identical (since the difference is zero.) As well, if you read the help for detrend, it says nothing at all about finding, AND removing a baseline trend.
Finally, let me test my claim that the slope of the detrended curve will be zero. The first coefficient out of polyfit should be zero, to within floating point trash.
ans = 1×2
1.0e-15 * 0.0381 -0.1313
As well, the second coefficient is also zero, since detrending also removed any constant, DC component in the curve.
Anyway, feel free to define detrending in any way you wish. But don't expect software to read your mind. And if that day ever comes when software can read my mind, I'm heading for the hills anyway. But then, it probably knows that already.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!