Filtering outlier in 2D coordinate data

Hi experts,

I need your help to remove some outliers in my 2D coordinate data shown below.

At, at least, four locations, there are some outliers that need to be removed from my data. Please see some spikes in the plots (in red circles).

Does anyone have an algoritm, or can suggest one, that can be used to detect and move (or replace) those values?

Note that I may have several problems with slightly different cases, e.g. the number of points of the outliers may be more or less than the one shown here.

PS.

I have attached a txt file containing the 2D coordinate in this post.

[Image Analyst]

Please explain the data format, or how you read it in. When I use

xy = dlmread('data.txt', ' ');

I get a a 2687 by 14 array. And the x and y coordinates seem to wander from one column to the next instead of being, like, all of column 1 are the x and all of column 2 are the y. What's going on?

[BeeTiaw]

The first column is the x-coordinate and second column is the y-coordinate. The size of the data is [2687 x 2].

I'd just used the following command to load the data.

data = load('Data.txt');
x = data(:,1);
y = data(:,2);

[Image Analyst]

What are the protruding shapes called? Are these ears or tabs not always in the same indexes for some reason? If not, why not? If not the same indexes, then are in they the same x,y location, like so you could use a template or mask to mask them out?

[BeeTiaw]

The shape is computed from another algorithm.

I intend to detect if there are any spikes in the plot and just connect the points after removing the spikes.

The way I connect the points will depend on the curvature of the shape around the spikes, e.g. linear, quadratic or spline. That's what I have in mind.

Please see the plot below, I have manually drew red-lines around the spikes.

I hope that clarifies my question.

[Image Analyst]

Not really, other than we now know the shapes are called "spikes". We don't know the answers to my questions about whether they are at the same index, or same location.

And also, what if the "spikes" take up a much larger portion of the shape? Like what if the spikes are 30%, 50% or 75% of the image? What parameters do you have restricting the problem. For example if the spikes are taking up 80% then you'd have inward pointing spikes and some algorithms would identify the outward pointing 80% as the spikes. Is that what you want? That the outward pointing ones are always what you want no matter if they end up being the "dominant' smooth shape and not spikes at all?

[BeeTiaw]

I can assure you that the spikes will not more than, even, 10%. Let's just stick to the problem I have at the moment. Thanks.

Answer 1

[Image Analyst]

Well this works for "the problem I have at the moment", though there are even simpler methods, and more sophisticated methods. I think this one is a good mix:

clc;	% Clear command window.
clear;	% Delete all variables.
close all;	% Close all figure windows except those created by imtool.
workspace;	% Make sure the workspace panel is showing.
fontSize = 18;
markerSize = 10;

% Read in data.
data = load('Data.txt');
x = data(:,1);
y = data(:,2);

% Smooth the shape with a median filter.
windowWidth = 9;
% for windowWidth = 9 : 2 : 9  % Try different values
	smoothedx = medfilt1(x, windowWidth);
	smoothedy = medfilt1(y, windowWidth);
	
	figure;
	plot(x, y, 'b.-', 'MarkerSize', markerSize);
	hold on;
	plot(smoothedx, smoothedy, 'r.-', 'MarkerSize', markerSize);
	grid on;
	legend('Original', 'Smoothed');
	caption = sprintf('Median Window Width = %d', windowWidth)
	title(caption, 'FontSize', fontSize);
	% Enlarge figure to full screen.
	set(gcf, 'Units', 'Normalized', 'OuterPosition', [0, 0.04, 1, 0.96]);
% end

% Find where shape deviates from smoothed shape the most.
figure;
residuals = sqrt((x-smoothedx) .^ 2 + (y - smoothedy) .^ 2);
plot(residuals, 'b.-', 'MarkerSize', markerSize);
title('Residuals', 'FontSize', fontSize);
grid on;
hold on;
% Enlarge figure to full screen.
set(gcf, 'Units', 'Normalized', 'OuterPosition', [0, 0.04, 1, 0.96]);

% Find all residuals more than 0.01.
spikeIndexes = residuals > 0.01;
xSpike = x(spikeIndexes);
ySpike = y(spikeIndexes);
% Plot them
figure;
plot(x, y, 'b.-');
grid on;
hold on;
plot(xSpike, ySpike, 'ro', 'MarkerSize', markerSize);

% Enlarge figure to full screen.
set(gcf, 'Units', 'Normalized', 'OuterPosition', [0, 0.04, 1, 0.96]);

[BeeTiaw]

Thanks!

[Image Analyst]

You're welcome. Thanks for accepting. Just realize that it may not work for shapes where the spikes are of different widths, and you might need to adjust the parameters until you get exactly 4 regions. You can determine the number of regions with either bwlabel() or findgroups().