large sparse matrices fail with accumarray

Large sparse matrices are easily created by 'sparse':

a = sparse(1e6, 1e6);

is working fine. However,

b = accumarray([1 1], 1, [2^31 1], @sum, [], true);

fails with "maximum variable size allowed by the program is exceeded." Matrix sizes with less than 2^31 elements are going through.

Why is this stupid limitation and how to circumvent it? I need accumarray to work on matrices of that size. The solution must be fast.

Using R2007a 32-bit Linux.

Cheers,

Balint

Answer 1

[the cyclist]

I believe this particular memory issue is related to the addressing of elements of the array. This limitation is mentioned (although not discussed in detail) in Section 1.5 of this guide to memory use: http://www.mathworks.com/support/tech-notes/1100/1107.html. There is also discussion in this document: http://www.mathworks.com/support/tech-notes/1100/1106.html

[Balint Takacs]

The fact that arrays of this size are finely created by 'sparse' shows that it is not a general limitation of MATLAB, rather indicates that accumarray is unable to handle large sparse matrices.

[the cyclist]

Are you able to upgrade to a more recent version? This statement from the documentation:

"Note that in MATLAB 7.4 (R2007a) and later you can create arrays with more than 231-1 elements but this was not officially supported until MATLAB 7.5."

suggests to me that your version was a bit transitional as they implemented the new functionality. For example, in R2012a I am able to execute the statement

b = accumarray([1 1], 1, [2^47 1], @sum, [], true);

without error.

[Balint Takacs]

Thank you. Unfortunately I have no option to upgrade.

I guess the problem is related to accumarray creating some temporary, which is limited by this bound you are referring to.

I have solved my problem by slightly changing the data structures, and using sparse(i,j,val,m,n) instead accumarray.

Answer 2

[Richard Crozier]

I'm not sure how your two statements here are equivalent?

a = sparse(1e6, 1e6);

creates a sparse matrix of all zeros of dimensions (1e6, 1e6), making only 1e12 elements, and nothing needs to be allocated since there are no non-zero values

b = accumarray([1 1], 1, [2^31 1], @sum, [], true);

Creates a sparse vector of size (2^31, 1) with a single nonzero value allocated at (1,1). The equivalent call to this is surely:

>> a = sparse(1, 1, 1, 2^31, 1)

which fails with:

??? Error using ==> sparse Sparse matrix sizes must be non-negative integers less than MAXSIZE as defined by COMPUTER. Use HELP COMPUTER for more details.

Which is probably the same error in another guise.

EDIT:

in fact even

a = sparse(2^31, 1)

fails with this error.

[Balint Takacs]

Both of your lines execute without error on my MATLAB, while the accumarray one does not.

[Richard Crozier]

You're right by the way, it does indeed have this stupid limitation, I've now come across it elsewhere. Not sure why it doesn't happen in the exact case above on my machine though.