Is there any limit to the number of rows that can be read by datastore function?

Hi I'm trying to parse a very large *.csv file into multiple *.csv file by MATLAB.

The size of *.csv file is about 47GB and has about 2Million rows.

The picture below is my workspace in Matlab and you can see the target csv file of 47GB in red rectangle

I set the read size of datastore as 10^5 by ReadSize,

you can see rows of data loaded to table 'T'.

I thought, by repetitively executing 'read' command, I can load all the data in 'FAT_ANALYSIS_REAL_ALL1.csv'.

Here's the 1st recommanding result, yo can check 'clear' ,'clc', datastore and readsize command deactivated,

and from red rectangle above, we can check that the read range of 'ttds' has move about 10^5 row by the increase of row index from '82630' to '171914'.

I reactivate the same code, and expected another 10^5 rows of data loaded form target file.

but you can see from the figure below that only 96438 row loaded to 'T' array

and I cannot load further data from 'read' function.

I hope to know why this happened.

It's not definitely cause by the shortage of the database, if i open up the file 's.csv' in high-end text editor,

you can check the scroll bar is only 1/5 to the totall but the row number is already around 2*10^5 in the lefthandside.

Here's the info of my computer, I hope this info help you guys figure out why this happened.

And the figure below is the info of target file 'FAT_ANALYSIS_REAL_ALL1.csv'

[Mohammad Sami]

According to the documentations, the ReadSize is the upper limit. So it can return less then number of rows in ReadSize.

https://www.mathworks.com/matlabcentral/answers/422446-why-doesn-t-the-read-tabulartextdatastore-variable-function-return-the-correct-number-of-rows-set-b?s_tid=answers_rc1-1_p1_MLT

[daeyeong Lee]

Thank you for your answer!

please check the file attached for further description.

But my understanding to your reply is that

you are saying the error is due to 'red box size limit' in the the drawings in attached pdf.file.

but I think it is due to 'red dash edged box size limit' in the the drawings in attached pdf.file.

So to take a re check,

you mean that there's no way to change the cover range of 'dashed red box' or datastore coverage, right?

[Mohammad Sami]

So what you are saying is that after the third read command, you are not able to load more data, eventhough the underlying csv file has more data available.

[daeyeong Lee]

Yes! precisely.

[Mohammad Sami]

After the third read if you run the hasdata function on datastore, what does it return.

tf = hasdata(ttds);

[Mohammad Sami]

The internal matlab file readdata.m is located in the following folder in C:\Program Files\MATLAB\RXXXX

toolbox\matlab\datastoreio\+matlab\+io\+datastore\@TabularTextDatastore.

you can try adding a debug in the readdata.m and see what may be happening.