Clear Filters
Clear Filters

Matlab uses much more memory reading a file

7 views (last 30 days)
Hi, I have a csv file with 22 columns and 871,000 rows. Columns are separated by comma. Each value is a text or number embraced by quote. The size of the file is about 150 MB. But after I read the file into MATLAB using textscan, the variable which stores the data takes up about 2GB of memory! For another csv file with 47 columns and 7,000,000 rows which have similar structure and is about 2GB in hard disk, MATLAB takes forever to read it using textscan. However, R is able to read these two files and the memory used is approximately the same as the space needed in hard disk. Are there any explanations for this? Thanks.
PS: Thank you for your help guys. Yes I can process the files line by line and discard the columns that I do not need. I am just curious why Matlab uses so much more memory than R or than the original file in the hard disk.
per isakson
per isakson on 26 Apr 2017
Edited: per isakson on 26 Apr 2017
You write "Each value is a text or number embraced by quote." Given the quotes, there is no alternative to your format specifier - afaik.
Recently Matlab has become better to handle large files and strings. Which release do you use?
Are the sizes of the different strings approximately equal?
Walter Roberson
Walter Roberson on 26 Apr 2017
If there are fields known to be numeric but enclose by quotes anyhow, then you can code a literal " character in the textscan format.

Sign in to comment.

Answers (1)

AstroGuy1984 on 25 Apr 2017
Do you need to read the whole matrix at once or are you just doing a routine line by line? If you are just running something on each line, I'd suggest NOT reading the whole file at once and doing something like:
fid = fopen('file.dat');
tline = fgetl(fid);
while ischar(tline)
tline = fgetl(fid);
fclose(fid); %DONT FOGET THIS!
Unfortunately, handling huge data files you eventually have to get creative in how you handle things. I would also suggest looking into using properties such as fseek and frewind if you're needing to pull out specific entries from a huge database. It will save you a ton of time and effort if you're just referencing the textfile for something.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!