append table to a exist csv file

Hi, I have a massive table which need to export to a csv file. The reason of append the table to the file rather grouping the table together and write it as a csv is because: the table size is 242*20X300*300 which is a large table and matlab can't handle the data in the memory.(This could be my computer fault) Anyway, my table is look like this

>> temp_T(1,1:10)

ans =

          patient_ID           R1C1    R1C2    R1C3    R1C4    R1C5    R1C6    R1C7    R1C8    R1C9
    _______________________    ____    ____    ____    ____    ____    ____    ____    ____    ____

    'CMC 009 OD 2008-04-08'    '0'     '0'     '0'     '0'     '0'     '0'     '0'     '0'     '0'

so is it possible to write it to the csv file and then append the data again so for example:

%%this is the csv file

          patient_ID           R1C1    R1C2    R1C3    R1C4    R1C5    R1C6    R1C7    R1C8    R1C9
    _______________________    ____    ____    ____    ____    ____    ____    ____    ____    ____

    'CMC 009 OD 2008-04-08'    '0'     '0'     '0'     '0'     '0'     '0'     '0'     '0'     '0' 

    'CMC 009 OS 2008-04-08'    '0'     '0'     '0'     '0'     '0'     '0'     '0'     '0'     '0'

[Marco Yu]

I know dlmwrite can append the result, I tried but it gives me a error message.

Undefined function 'real' for input arguments of type 'table'.

[Marco Yu]

names(1,1)={['patient_ID']};

values(1,1)=cellstr(imageName(1,count_image));

names(m,n)={['R' num2str(m) 'C' num2str(n)]};

temp_x=temp_image(m,n);

values(m,n)={[num2str(temp_x)]};

temp_T=cell2table(temp_table(2,:));

temp_T.Properties.VariableNames=cellstr(temp_table(1,:));

%%Here is the class of the data.

[Andrew McLean]

I have the same problem. I have a long running program that generates data that I want to write to a CSV file. The data are generated in blocks of a few thousand rows at a time. I want to write the data as it is generated in case the program crashes, as it can run for days. The data are mainly numeric, but includes some non-numeric data like datetime objects.

What I woul like to see is something like:

• an option to append to a file in writetable
• have writetable take a file handle as an input in place of a filename
• a writer object like Python's csv.writer

Has anything moved on since the original responses to this question?

At the moment my best option seems to be to wite out hundreds of CSV files and join them together with a Python script.

[Christoph Pieper]

Because I was also looking for this: In R2020b there is now an option to append with the writetable function:

'WriteMode','append'

not sure when it was added but 2018b does not have it yet.

[Brent F]

WriteMode='append' is a great idea, but it causes writetable to suppress the column headers.

The fix is to add: `WriteVariableNames=true`

Answer 1

[Peter Perkins]

Marco, it's quite possible (you don't provide enough info to know for sure) that the reason why your table is so big is because you are storing your numeric values in cell arrays of strings. This:

>> x = randn(242*20,300*300);
>> t = array2table(x);
>> whos
  Name         Size                    Bytes  Class     Attributes

    t         4840x90000            3506018448  table               
    x         4840x90000            3484800000  double

is not all that big. Patient_ID is not numeric, but even there you might benefit form using a categorical if the IDs are not all unique.

Answer 2

[Guillaume]

You cannot use dlmwrite or csvwrite to write tables. You would have to convert the table to a matrix (using table2array). Even then, dlmwrite and csvwrite can only write matrices of numbers and your table contain text, so they're completely out of the equation.

The normal way to write a table to a csv file is to use writetable. Unfortunately, there's no append option.

So, in the end, you can either:

• write the tables to different ranges of an excel spreadsheet (with writetable). However, if the merged table uses too much memory in matlab, it's likely that it'll be the same for excel.
• write the tables to individual files (with writetable), and merge all these file together afterward.
• use low level functions ( fopen, sprintf, etc.) to write your tables.

[Marco Yu]

Hi Guillaume. I am interested in the write tables to individual files, but how can I merge all these files together? should I do it in excel?

Or in other hand, is the low level functions is easier?

I only need to transfer the data like above to 1 csv file. Can I have some suggestion please?

[Guillaume]

The low levels functions are not particularly complicated, the hardest part would be to figure out the format string for fprintf.

Merging text files together can easily in any OS using the built-in command line tools. On windows, you can use copy. Thus, the code would be something like this:

finalfile = 'hugecsvfile.csv';
tempfileprefix = 'tempfile';  %whatever you want. could include fullpath. file number and extension added in the loop
tempnames = cell(1, numtables); %for stocking file names of temporary files

for tableiter = 1 : numtables
   %...
   %construct temporary table any way you want, e.g.
   temp_t = cell2table(temp_table(2,:), 'variableNames', cellstr(temp_table(1,:)));
   %save table to temporary text file:
   tempnames{tableiter} = fprintf('%s%02d.csv', tempfileprefix, tableiter);  %or any other format you wish
   writetable(temp_t, tempnames{tableiter}, 'WriteVariableNames', tableiter == 1); %only write header for first file
end

%build string for Windows copy command:
copystr = sprintf('copy %s%s %s /b', tempnames{1}, sprintf(' +%s', tempnames{2:end}), finalfile);
system(copystr);
%delete temporary files
delete(tempnames{:});