Matlab crashes with large matrix

I need to create a large matrix F=[F1, F2, ... , FN] where each Fn has the same number of rows (about 12,000) but different number of columns. What I am doing now is to create F as a cell array using a loop with element F{n} = Fn. The problem is that when the number of columns of F is very large, Matlab crashes: at some point it shuts down (I run this on a server, and I find matlab closed). I suspect that it shuts down when I convert the cell array into a single matrix, F=cell2mat(F);

Notice that everything works ok when F has a number of columns less than say, say 1,000,000,

Any help is much appreciated. Thanks!

Answer 1

[Walter Roberson]

Well, that is certainly possible... if you make the arrays large enough then there might just not be enough memory available.

Is it possible to predict the number of columns that will be needed for each Fn before doing the actual callculations? If so then you could consider doing that calculation and totaling them, and pre-allocating the output array, then writing directly into the output array instead of building the cell array. If you pre-build the sizes then you can use cumsum() to create an array that will tell you which indices to write at

As a side note: it is more efficient to use

F = horzcat(F{:});

than F = cell2mat(F); -- but this would not solve your memory problems.

[valentino dardanoni]

Thank you very much Walter. Unfortunatley I cannot predict ex ante the number of columns in each Fn. I guess I could check them afterwrods looking at the cell array, then stop the program, and rerun it by preallocation... It would not be fun, but it is doable... I was wondering, however, if it is normal that Matlab crashes in such a situation, rtaher than giving some sort of warning and stopping....

[Bruno Luong]

No it's not normal. But when all memory are consumed, many tasks are slowing down and requesred memory get swap to hard drive. The PC are in a "non standard" functioning mode and things can get slippy from there. Even pop up a warning message or windows become a hercules task.

Answer 2

[Bruno Luong]

For 1e6 column the memory required in Gb for one array is

memgGb = 1e6*12e3*8 / 1e9

memgGb = 96

And you have the cell array + the concatenate array (single contiguous emory blockà, so you need 192 Gb of free memory.

You might consider using single-class array, write directly to the allocated matrix as Walter suggests or using tall array assuming you store the transposed of the original intended array.

[valentino dardanoni]

Grazie Bruno, will explore the tall array route... BTW by coincindence the next step in my program is to use "licols", which I understand stands form an idea of yours!

[Bruno Luong]

licols is Matt's FEX, but yeah I quite participate to popular the idea. Thanks