Clear Filters
Clear Filters

Matching two arrays that have similar values

2 views (last 30 days)
I have two folders that have time stamped files. The time stamps are in this format: MMDDYY HHMMSS.### where ### is the number of milliseconds.
I need to compare the contents (file names) of the two folders and generate a new array (or structure) that contains all the files from both folders that have time stamps within 12 milliseconds of each other.
I have manipulated the file name strings and generated numeric arrays with the file stamps but then I still have to compare two large numeric arrays to each other with double loops etc.
That just takes up too much time (I have dozens of folders each containing thousands of files).
I was wondering if there is a quicker easier way to do it?
Thank you in advance
  2 Comments
dpb
dpb on 18 Jul 2014
Not sure it would be any faster in the end, but a sorted combined vector of all would allow a single processing pass from the beginning to pick the ones sequentially that are no farther than M millisec from present. Then increment to the next beyond that and repeat.
Cedric
Cedric on 18 Jul 2014
Edited: Cedric on 18 Jul 2014
Sort first as proposed by dpb, and then you can cluster based on time difference (using DIFF and e.g. ACCUMARRAY).

Sign in to comment.

Accepted Answer

Arun Mathew Iype
Arun Mathew Iype on 24 Jul 2014
Edited: Arun Mathew Iype on 24 Jul 2014
It seems sorting the data first and then using some matching algorithm is faster by around 65 times on my system.
What I did was to sort the two numeric arrays using “sortrows”. Then I match each entry in the first array with entries which fall in 12 millisec duration in the second array. Since the array is sorted, the search can be stopped if you get a non-match after a match. Additionally, the next element in the first array may start the search from the first matched element in second array for the previous entry in the first array.
The following is the test script based on an input of around 1000 rows in each file array. >> array_grouping Time for generating matches using simple for loops =5.2695 Time for generating matches using sorting =0.082277
Image : Input files
Image : Output matching

More Answers (0)

Categories

Find more on Large Files and Big Data in Help Center and File Exchange

Products

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!