Fastest way to save and retrieve floating point vectors

9 views (last 30 days)
SKP
SKP on 11 Mar 2021
Commented: SKP on 15 Mar 2021
I am writing a code that generates a row vector of size 1x600 (containing floating point numbers) in each iteration. There are about 100,000 such iterations performed in the entire code. Also, I have to read these saved vectors (row wise) in a different code at a later point of time. I have been using 'dlmwrite' so far for appending the vectors row wise to a .csv file, and the function 'load' for retrieving this data. I found that saving and retrieving data in this way is consuming a lot of time. I wanted to know if there is a better way of saving and retrieving these vectors.
I have read some posts in the forum suggesting to save the data in a binary file/.mat file as some alternatives. But I am not sure which would be the best for my case. Could someone please suggest me a faster way to do the above exercise?

Accepted Answer

Walter Roberson
Walter Roberson on 11 Mar 2021
Simplest way, that will take very low overhead for your program (such as might be needed for real-time work):
fid = fopen(OutputFileName, 'w'); %not 'wt'
for ...
fwrite(fid, YourBufferOfDoubles);
end
fclose(fid)
However, you have about 480 megabytes worth of raw data, and that is enough that unless you are writing to SSD, that the file overhead is going to add up.
Because of that, you might want to consider methods that compress data as you go. You would buffer some data and call compression routines sometimes, and write out the compressed data. One way to do that would be https://docs.oracle.com/javase/7/docs/api/java/util/zip/GZIPOutputStream.html
A third way... just store all the values in memory as you produce them (having pre-allocated the array), and afterwards save() them to a .mat file. .mat does compression.
  2 Comments
SKP
SKP on 15 Mar 2021
The suggested methods are very time efficient and occupy lesser memory compared to the ones I was previously using. Thank you.

Sign in to comment.

More Answers (0)

Products


Release

R2020b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!