Problem having isoutlier detecting anything

I have what looks like a very easy problem but I cannot seem to solve it. I have a dataset (attached) that has some obvious (to the human eye) outliers.

I cannot get isoutlier to detect it in any way. My attempt is essentially this:

idx = isoutlier(x(:,2),'movmedian',w);

I have put the code in a for loop, spanning all possible values of w, I get at most 3 outliers detected when the window size is 3 and those detected are not actually outliers.

Using movmean instead movmedian detects no outliers for any value of w. I have also played with the threshold factor, but without luck. This seemed to me like a straightforward application for the outlier detection. What am I missing?

Answer 1

[Akira Agata]

One possible way to detect this type of outlier would be like this:

load('out1.mat');

% Assuming that data has 2nd order polynomial curve trend
p = polyfit(x(:,1),x(:,2),2);
y = polyval(p,x(:,1));

% Detect outlier in de-trend data
idx = isoutlier(x(:,2)-y);

% Show the result
plot(x(:,1),x(:,2))
hold on
plot(x(idx,1),x(idx,2),'ro')
legend({'Original data','Detected outlier'},'FontSize',14)

[Vittorio]

Nice idea! The data I attached are only part of a larger data set. I'll play with your suggestion to see what I get.

[Chris Turnes]

You can also generalize this approach a little bit if your data doesn't globally fit a polynomial, but does over large local regions by replacing the polyfit portion with a call to smoothdata using the loess, lowess, or sgolay methods. For your data, you can get similar results doing:

 % Local weighted quadratic fit on a window of span 0.3
 y = smoothdata(x(:,2), 'loess', 0.3, 'SamplePoints', x(:,1));
 % Find outliers in the difference between the smoothed and original.
 tf = isoutlier(y-x(:,2));
 % Visualize the difference.
 plot(x(:,1), x(:,2), x(tf,1), x(tf,2), 'o')
 legend({'Original data','Detected outlier'},'FontSize',14)

[BAPPA MUKHERJEE]

Currently I am working on this topic. Can you please help me to load smoothdata function in directory, because in 2012 version its shows undifined function.

Answer 2

[Ryan Takatsuka]

The outlier detection generally works best on single data point outliers (not multiple in a row). Your data has a large amount of outliers in a row that have a significant effect on the trend, or moving average, of the data.

In order to detect these outliers, you will need to use a very large moving average window to minimize the effect that the outlier have on it. Additionally, the threshold will need to be modified a bit. I used a window size of w=50 and a threshold of 0.5. This detects the outliers, but also falsely identifies points and the beginning and end of the dataset because the moving average has such a large window.

a = isoutlier(x(:,2), 'movmean', 50, 'ThresholdFactor', 0.5);

It also helps to plot the moving average that is used to calculate the outliers. This is shown in the image:

[BAPPA MUKHERJEE]

I am unable to plot the last figure. could you please elaborate this code upto the plotting stage.

Answer 3

[BAPPA MUKHERJEE]

Currently I am working on this topic. Can any one help me to load smoothdata function in directory, because in 2012 version its shows undifined function.