Write and read numeric data line by line

5 views (last 30 days)
Here is a working example of what I have in mind:
N=10000; M=10000;
x=rand(1,M);
dlmwrite('file',x,'delimiter',' ')
for k=1:N-1
x=rand(1,M);
%In the real application x is the result of some calculations
dlmwrite('file',x,'delimiter',' ','-append')
end
toc
tic
for k=1:N
u=dlmread('file',' ',[k-1 0 k-1 M-1]);
%calculations using vector u
end
toc
The writing part is fine (about 5 seconds on my PC), but the reading is
awfully slow - about 43 seconds. My guess is that it would help if reading
is resumed automatically after the last line read. The command "fgetl"
appears to do this, but is designed for character, rather than numeric,
variables.
Any other suggestions?
(The problem entails generating an array
of size 10^5 X 10^5, which is used by another program.)

Accepted Answer

Michael Soskind
Michael Soskind on 5 May 2020
Hi Chris,
If I were you, I would try and use fprintf and fscanf rather than dlmwrite and dlmread.
These are quite a bit faster in my experience, and testing with smaller example arrays, give a much better result.
I compare the two in the following code:
%% Using fprintf and fscanf, which seem order of magnitude faster
N=100; M=100;
x=rand(1,M);
dlmwrite('file',x,'delimiter',' ')
tic
fid = fopen('file.txt', 'a');
for k=1:N-1
x=rand(1,M);
%In the real application x is the result of some calculations
fprintf(fid,repmat('%f ', 1,M), x);
end
fclose(fid);
toc
tic
fid = fopen('file.txt', 'r');
for k=1:N
u=fscanf(fid,repmat('%f ', 1,M),[1 M]);
%calculations using vector u
end
fclose(fid);
toc
%% Using dlmwrite and dlmread
N=100; M=100;
x=rand(1,M);
dlmwrite('file',x,'delimiter',' ')
tic
for k=1:N-1
x=rand(1,M);
%In the real application x is the result of some calculations
dlmwrite('file',x,'delimiter',' ','-append')
end
toc
tic
for k=1:N
u=dlmread('file',' ',[k-1 0 k-1 M-1]);
%calculations using vector u
end
toc
You can test this with any data size you like, I suppose my computer is not nearly as fast as yours at the file processing.
The append feature of fprintf is particularly useful.
Hopefully that helps,
Michael
  2 Comments
Michael Soskind
Michael Soskind on 5 May 2020
Sorry, a few notes
  1. I did not remove the dlmwrite at the beginning of the code there.
  2. Further, the speed up is quite good for a 1000 element case on my computer.
I have ~2.5 seconds for reading and writing with fprintf, and 5.6 and 60 seconds for writing and reading, respectively with dlmread, using a 1000 element array.
Chris Koen
Chris Koen on 6 May 2020
Dear Michael - That does indeed help, thank you very much!
I made one change to get each x on a different line:
fid = fopen('file2.txt', 'a');
q=strcat( repmat('%f ', 1,M),'\n');
for k=1:N
x=rand(1,M);
fprintf(fid,q, x);
end
fclose(fid);

Sign in to comment.

More Answers (0)

Products


Release

R2015a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!