Fast way to read text file of formatted data

57 views (last 30 days)
Ted
Ted on 26 May 2014
Commented: rifat on 27 May 2014
I have a text file formatted as (just showing the first, second, second to last and last lines):
0.00 68.000 68.000
1.00 68.001 68.000
...
1923.00 1871.164 1869.803
1924.00 1871.484 1870.134
The data values are delimited by spaces (number varies, depending by data values).
I want to import these as floating numbers, eventually in a 3 column array. I will always know ahead of time how many columns there are but I will not know ahead of time how many rows there will be.
I can input this easily using one of many commands: dlmread, importdata, textscan, fscanf. For a resulting 1925x3 array, fscanf is the fastest and takes around .004 sec. Since I will have to do this import over a hundred thousand times in my MATLAB script, is there a faster way to do this? Thanks

Answers (3)

Cedric Wannaz
Cedric Wannaz on 27 May 2014
Edited: Cedric Wannaz on 27 May 2014
You should/could perform this operation using a RAM drive/disk. There is little interest in saving temporary files on disk when you have a lot of them. Once you have the data in MATLAB, save it in one or a few .mat files, to avoid having to deal with that many files in the future. Your processing flow becomes:
  1. MATLAB code writes MATLAB variables to RAM drive/txt file.
  2. Ancient software reads RAM drive/txt file, processes content, outputs outcome to RAM drive/txt file.
  3. MATLAB code reads RAM drive/txt file, processes content, stores outcome in variable.
  4. Loop to 1 if not done.
When done: store variable in unique .mat file. If too large to hold in memory, split into a few blocks, e.g. 1GB, to minimize the number of files. As mentioned by Star Rider, the .mat format is well suited for storing large amounts of data; it is based on HDF5 [ ref ] (from version 7.3 on). Yet, if you want to push even further, this post is not uninteresting.
To optimize read/write operations on a very large number of files, stick to low level functions, and try to be as specific as possible during calls (i.e. it is more efficient to specify a separator than to let the function find it out by testing, it is more efficient to specify a date format than to let a function finding it out, etc).
  1 Comment
Ted
Ted on 27 May 2014
Thanks for the RAMdisk suggestion. I did test a RAM disk when I was trying out different data import methods (dlmread, importdata, textscan, fscanf). I did a simple large import and looped it for 10000 times. The RAM disk was only 5% faster--I suspect most of the execution time was in parsing the imported txt file. Now that I have my code done, I will try some timing tests again. Even if there is just a small difference in time, reducing HDD wear and tear is a good idea.
I settled on using fscanf, it seemed to be the fastest but not by much. I tried several different formatspecs, but they all timed about the same. Don't have to provide a delimiter so I think that I've made fscanf as fast as it can be.
I will also try rift's code below.

Sign in to comment.


Star Strider
Star Strider on 26 May 2014
I would import them once and save them as a ‘.mat’ file, with their variable names included. Then load the ‘.mat’ each time instead.
  9 Comments
Star Strider
Star Strider on 27 May 2014
I rarely program in FORTRAN now (haven’t in more than a decade) but I have a compiler in my DVD software library that will run on this machine (Win 8) that I keep partially out of nostalgia. I was still writing FORTRAN code for my neural nets and genetic algorithms about 20 years ago because MATLAB was very slow on those machines. FORTRAN was significantly faster, and probably still is for large projects.
I strongly suggest you consider Cedric’s idea of a RAM drive for the temporary files. It is much faster in terms of read-write time — you can also wipe the files from the RAM drive quickly — and you don’t have to worry about HDD file fragmentation that would likely slow your processes.

Sign in to comment.


rifat
rifat on 27 May 2014
Edited: rifat on 27 May 2014
I did a similar thing before. Hope this helps. result will be on the variable mat.
fclose('all');
fid=fopen('fileName.txt','rt');
frewind(fid);
lnum=0
count = 1;
while(count==1)
line=fgetl(fid);
lnum=lnum+1;
if ~ischar(line)
count=0;
else
count=1;
line=strtrim(line);
len=length(line);
a=line==' ';
loc=find(diff(a)~=0);
num1=line(1:loc(1));
num2=line(loc(2)+1:loc(3));
num3=line(loc(3)+1:end);
mat(lnum,1)=str2num(num1);
mat(lnum,2)=str2num(num2);
mat(lnum,3)=str2num(num3);
end
end
  2 Comments
rifat
rifat on 27 May 2014
yeah.. i wasnt worried about performance.. Thats why this rough implementation

Sign in to comment.

Categories

Find more on Large Files and Big Data in Help Center and File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!