Bypass maximum matrix size for ones()

I have a program in which I create a large matrix of ones using the ones() function, but apparently the matrix is too large for the function. I get the error 'Maximum variable size allowed by the program is exceeded.' I use this matrix later in my code and overwrite some entries, but most of the entries stay one, so it would be most efficient if I have the matrix be initialized with ones from the start. Is there someway that I can expand the maximum variable size of this function or is there another way to create a large matrix pre-populated with ones. (Just for reference, by large I mean 40,859,280 elements and I think the maximum number of elements is 524,288). Any input would be helpful and thanks in advance!

[James Tursa]

40M elements is not that big for a modern computer. Can you show us the exact syntax you are using to create the matrix? Maybe you are inadvertently trying to create a 40M x 40M matrix.

[Sana Ahmed]

I'm using S = ones(420,402,242);

Answer 1

[Walter Roberson]

Should you perhaps be using a sparse() array?

Your array is only about 312 megabytes; I would expect any modern system to be able to handle an array that size; I would only expect trouble with the 32 bit versions of MATLAB (it was common to have only about a gigabyte of free memory for those.)

Anyhow, go into Preferences -> Workspace -> MATLAB Array Size limit.

You can configure the fraction of RAM that you permit, and you can turn off the check.

[Sana Ahmed]

I tried that, but i'm still getting the same error.

[Walter Roberson]

Are you using Windows? If so what shows up when you command

memory()

?

[Sana Ahmed]

It says

"Maximum possible array: 9499 MB (9.960e+09 bytes) *

Memory available for all arrays: 9499 MB (9.960e+09 bytes) *

Memory used by MATLAB: 2638 MB (2.766e+09 bytes)

Physical Memory (RAM): 16282 MB (1.707e+10 bytes)

* Limited by System Memory (physical + swap file) available."

This is after I took off the size limit

[Walter Roberson]

Unless you had a quite small setting for the Array Size limit preference, I cannot see any reason it would think that your array was too large.

Is it definitely the ones() call that is causing the problem, or is it further on? For example if you were calling pca() on an array that size then you would have a problem.

[Sana Ahmed]

I just realized I was looking at another matrix in my code, I'm actually trying to make a 40M x 40M array. Is that too large for the memory?

[Bruno Luong]

Memory size needed for an (double) array MxNxP

• in Megabyte is M*N*P*8 / 2^20
• in Gigabyte is M*N*P*8 / 2^30

Now you know everything to answer your own question.

[Sana Ahmed]

I see, thanks for the help!

[Walter Roberson]

>> log2(40000000^2*8)
ans =
          53.5069933284231

You would require 54 address bits to address that much memory. However, all publicly known implementations of the Intel x64 architecture have a limit of 48 address bits. The upper bytes of addresses is currently used by the operating system for its internal purposes (including access to privileged memory), so even if you had a chip with more than 48 address lines, you would have problems.

I just tested on my system, and I was able to

          foobar=spalloc(40E6, 40E6,2064000000000);  %0.00129 occupancy      

but not

          foobar=spalloc(40E6, 40E6,20640000000000); %0.0129 occupancy

The 0.0129 occupancy was tested because that is the fraction you had asked for earlier when you wanted 524,288 out of 40,859,280.

So if you use sparse matrices, it is not the 40E6 on a side that is the problem, but the amount of memory requested to be allocated.

It appears that the allocated memory according to whos() is

 (columns+1) * 8 + nonzeros * 16

and

 >> log2(33024320000008)
ans =
          44.9085940889452
          

whereas if I had asked for 10 times as much that would add another 3 to 4 bits... which would take it just over 2^48.

Therefore you can spalloc() requesting up to 2^48 bytes according to (columns+1) * 8 + nonzeros * 16 . But you are going to swap very badly once you go beyond your physical memory.