- MATLAB code writes MATLAB variables to RAM drive/txt file.
- Ancient software reads RAM drive/txt file, processes content, outputs outcome to RAM drive/txt file.
- MATLAB code reads RAM drive/txt file, processes content, stores outcome in variable.
- Loop to 1 if not done.
Fast way to read text file of formatted data
57 views (last 30 days)
I have a text file formatted as (just showing the first, second, second to last and last lines):
0.00 68.000 68.000
1.00 68.001 68.000
1923.00 1871.164 1869.803
1924.00 1871.484 1870.134
The data values are delimited by spaces (number varies, depending by data values).
I want to import these as floating numbers, eventually in a 3 column array. I will always know ahead of time how many columns there are but I will not know ahead of time how many rows there will be.
I can input this easily using one of many commands: dlmread, importdata, textscan, fscanf. For a resulting 1925x3 array, fscanf is the fastest and takes around .004 sec. Since I will have to do this import over a hundred thousand times in my MATLAB script, is there a faster way to do this? Thanks
Cedric Wannaz on 27 May 2014
Edited: Cedric Wannaz on 27 May 2014
You should/could perform this operation using a RAM drive/disk. There is little interest in saving temporary files on disk when you have a lot of them. Once you have the data in MATLAB, save it in one or a few .mat files, to avoid having to deal with that many files in the future. Your processing flow becomes:
When done: store variable in unique .mat file. If too large to hold in memory, split into a few blocks, e.g. 1GB, to minimize the number of files. As mentioned by Star Rider, the .mat format is well suited for storing large amounts of data; it is based on HDF5 [ ref ] (from version 7.3 on). Yet, if you want to push even further, this post is not uninteresting.
To optimize read/write operations on a very large number of files, stick to low level functions, and try to be as specific as possible during calls (i.e. it is more efficient to specify a separator than to let the function find it out by testing, it is more efficient to specify a date format than to let a function finding it out, etc).
Star Strider on 26 May 2014
I would import them once and save them as a ‘.mat’ file, with their variable names included. Then load the ‘.mat’ each time instead.
rifat on 27 May 2014
Edited: rifat on 27 May 2014
I did a similar thing before. Hope this helps. result will be on the variable mat.
count = 1;