How to import large text file data in to workspace?
5 views (last 30 days)
Show older comments
Hi, I am having a text file with following format
First Line is a string contains device parameters
rest is 18000x12 Lines format is double
I am using uigetfile to access my file then fscanf to read data. But this text file should reside in the same workscpace where the reading script is placed.
Is there any way to read above file without placing .txt file into current directory. The command importdata is not useful in my case only.
0 Comments
Accepted Answer
Walter Roberson
on 4 Feb 2017
[filename, pathname] = uigetfile(.....)
fullname = fullfile(pathname, filename);
ncol = 12;
fmt = repmat('%f', 1, ncol);
fid = fopen(fullname, 'rt');
data_cell = textscan(fid, fmt, 'HeaderLines, 1, 'CollectOutput', 1);
fclose(fid);
data = data_cell{1};
4 Comments
Walter Roberson
on 5 Jul 2022
datatime = [datatime; newline];
That is slow. Each time you append data to the end of an array, MATLAB has to allocate space for the new larger array, then copy the old values into the array, copy in the new value, then release the old space. The first time that would involve copying 0 existing rows; the second time it would involve copying 1 existing row; the third time, 2 existing rows, the 4th time, 3 existing rows, and so on. After N times that would be 0+1+2+3+...(N-1) existing rows copied, which would be (N-1)*N/2 rows copied from the beginning. It is an N^2 copying process.
Instead, if you can establish a maximum size, pre-allocate the memory, and assign into the array,
maxents = 50000;
datatime = zeros(maxents, 6);
...
datatime(counter,:) = newline;
Note: you are asking for the entire format of six %f64 items to be processed 6 times, so you are trying to read in 6 rows of 6 columns at a time. Are you sure that the data has a multiple of 6 rows in each block?
LeChat
on 6 Jul 2022
Dear Walter, thank you very much for your answer.
Would you know how I can get the maximum row number to load in order to pre-allocated the memory and assign into the array? I am thinking of something similar to "wc -l filename" in shell, but here in Matlab, with -if possible- the possibility to get the respective size of each of the two blocks of data...
I actually don't remember why I was processing the data by sets of 6 rows... I know the number of columns in each block (the first block has 20 columns, the second one has 6 columns), but the number of rows is a priori unknown.
Thanks you very much for your help and interest.
More Answers (0)
See Also
Categories
Find more on Large Files and Big Data in Help Center and File Exchange
Products
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!