Efficient way to read in large dataset
1 view (last 30 days)
Show older comments
Hi all.
I am to read in a set of data to Matlab which has 6 columns and more than 2 million rows. I tried reading the data as a tab delimited .txt file with the dataset function as such:
pricedata = ...
dataset('File',file,'Delimiter','tab','ReadObsNames',true) ;
This has gone on for more than 3 hours. Can someone suggest a better way to do it?
0 Comments
Accepted Answer
Peter Perkins
on 1 Nov 2011
Mads, you will most likely find that this:
>> help dataset/dataset
dataset Create a dataset array.
[snip]
DS = dataset('File',FILENAME,'Format',FORMAT, ...) creates
a dataset array using the TEXTSCAN function to read column-
oriented data in a text file. Specifying a format can
improve speed significantly for large files.
will read the file much faster. In your case, FORMAT might be something like '%f%f%f%f%f%f' if all the columns are numeric. Hope it helps.
More Answers (0)
See Also
Categories
Find more on Large Files and Big Data in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!