Script to Find Repeating Data within Tolerance
4 views (last 30 days)
Show older comments
Hello,
I have a table that has over 6,000 rows of data with 8 columns, with each column representing a different data.
Many of the rows of data are similar but not exactly the same as other rows.
In other words, plotting column 1 vs column 2, and column 1 vs column 3, etc. gives a sinusoidal shape with a periodic behavior, but it is not exactly repeating, but slightly increasing.
With that in mind, I wrote a script such that it finds similar data within a tolerance, and removes that row of data if it is deemed as similar.
In short, the code is something like this:
for J=1:Length
for I=1:Length
if I~=J
if abs(table.colA(J)-table.colA(I))<=tolerance && abs(table.colB(J)-table.colB(I))<=tolerance && % Repeat for other columns
table.colA(I)=-1;
table.colB(I)=-1;
% Repeat for other columns
else
end
else
end
end
end
% Then I keep only the rows whose values are greater than -1
What I have works, I got it to go from over 6,000 rows to about 150 rows
However it is very computationally expenisve, meaning it takes several hours to run
Does anyone have any advice to speed this up or make it more efficient?
Or even just a better way to do this task?
I know there is an easy function that removes duplicate rows in a table, but since none of the rows are exactly the same, this won't work.
Is it possible to do this to a tolerance, like two rows not exactly duplicates, but "close enough" to each other.
I was looking into it here and the in maual but couldn't find any easier way to do it than what I did above.
Any help or suggestions is appreciated, thanks.
0 Comments
Accepted Answer
More Answers (0)
See Also
Categories
Find more on Logical in Help Center and File Exchange
Products
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!