Computer crashed when calculating matrix (mixed with sparse and full) multiplication and summation

In a Matlab script, I have to multiply a large sparse matrix with a full matrix, as follows:

A1 = B * C.

B is a full matrix, its size is 5000-by-4800 double. C is a sparse matrix, its size is 4800-by-3E5 sparse double, and its nnz/numel = 2.08E-4, so it’s very sparse.

My desktop computer crashed at this line of code. I thought it was because matrix C is too large, so I “broke down” matrix C along its columns, and let them multiply matrix B one by one, such as follows:

for j=1:3E5

Aj=B*C(:,j);

A1(:,j)=Aj;

end

The computer did not crash this time, it was able to calculate the multiplication and provide matrix A1. However, it was slow (elapsed time was about 10 seconds), and the “Physical Memory Usage” in the Task Manager was alarmingly approaching the maximum.

Immediately following this “for” loop, I have one more line of code, which is a simple summation of two matrices:

A = A1 + A2.

Matrix A1 (a full matrix) is obtained from above. Matrix A2 is a sparse matrix of the same size (5000-by-3E5), it is very sparse.

As I mentioned, the computer did not crash calculating the “for” loop. But it crashed at this summation line of code afterwards. Could it be because too much “Physical Memory Usage” was used up already prior to the summation?

In conclusion, the whole thing is a mess, and I appreciate any help to be able to calculate these large matrix multiplication and then summation. Or could it be that the matrix C is simply too large and there’s no way to solve it?

Thank you!

-------------------------

Update: I just tried something else: Without running any code, I entered in the Command window: A1 = zeros(5000, 3E5). This simply creates a full matrix of zeros, and I see that the “Physical Memory Usage” in the Task Manager immediately goes up to about 14GB, approaching the maximum (16GB).

Does this mean a full matrix that has 3E5 number of columns is just too big for the computer to handle, and there's nothing I can do to solve it?

[James Tursa]

What does the following calculation yield:

sum(any(C))

That would tell you how many columns of C have at least one non-zero number, and consequently would tell you how sparse you would expect the result of B*C to be.

[Paula]

sum(any( C )) = 3E5, meaning each column has at least a non-zero value. Is that why the line "A1 = B * C" uses all the Physical Memory? But I checked matrix C, quite a few columns have only a small number of non-zero entries.

What do you mean "...and consequently would tell you how sparse you would expect the result of B*C to be"? Matlab automatically makes the result of B*C a full matrix, so it's not sparse.

Thanks!

[James Tursa]

Remember that each element of the B*C result is simply the dot product of a row of B with a column of C. If that column of C has any non-zeros in it (even if it is only one non-zero), then in general you would expect the dot product to be non-zero, the only exception being if you had very fortunate cancellation. The fact that sum(any( C )) = 3E5 for your particular case means that all of the columns of C have at least one non-zero, so the result of B*C would be expected to be essentially a full matrix. Hence your runtime and memory problems ...

[Paula]

OK. Just to make sure I understood, you are saying that the computer simply doesn't have enough space to handle a full matrix that has 3E5 number of columns, so I should never create a full matrix this big, is that correct?

I just tried something else, and I think it's consistent with your point: Without running any code, I entered in the Command window: A1 = zeros(5000, 3E5). This simply creates a full matrix of zeros, and I see that the “Physical Memory Usage” in the Task Manager immediately goes up to about 14GB, approaching the maximum (16GB).

I guess I should look for another implementing method then.. Or maybe I can "break up" the large matrix into several ones, or make it into an array? Would you have any suggestions please?

[James Tursa]

You have pretty much come to the conclusion yourself ... you will need to chunk out your calculations to avoid the memory issue and simply live with the calculation runtimes, or figure out another way to solve your problem. We don't have any insight into what that problem actually is, but maybe you could solve a smaller problem that approximates your actual problem within acceptable error, etc.

[Paula]

Understood. Thank you very much for the help and suggestions!

I would like to "accept" your answer to end this question, but I don't know how to. I think it's because you answered as a "Comment", not as an "Answer"? But I'm not sure, sorry.

If you don't mind clicking on "Answer this question", and copy pasting one of your answers to there, I would gladly accept it.

Answer 1

[James Tursa]

Remember that each element of the B*C result is simply the dot product of a row of B with a column of C. If that column of C has any non-zeros in it (even if it is only one non-zero), then in general you would expect the dot product to be non-zero, the only exception being if you had very fortunate cancellation. The fact that sum(any( C )) = 3E5 for your particular case means that all of the columns of C have at least one non-zero, so the result of B*C would be expected to be essentially a full matrix. Hence your runtime and memory problems ...

You will need to chunk out your calculations to avoid the memory issue and simply live with the calculation runtimes, or figure out another way to solve your problem. We don't have any insight into what that problem actually is, but maybe you could solve a smaller problem that approximates your actual problem within acceptable error, etc.

Answer 2

[Steven Lord]

What release of MATLAB are you using?

What operating system? I'm suspect you're going to say Linux. If I'm right, the OOM Killer may have struck again. The page linked as the first footnote says how to check if this was the case.

[Paula]

I don't think it's Linux. It's "Windows 7 Enterprise", RAM is 16GB, and 64-bit operating system.

Does it mean I should not have the "out of memory" problem? Thanks!

Matlab release is R2018b.

[Steven Lord]

Can you clarify did your entire computer crash (Windows shut down, you received a Blue Screen of Death, etc.) or did MATLAB crash? In the latter case, there should be a crash log file in the temporary directory:

ls(fullfile(tempdir, '*crash*'))

If one exists that was created at the time MATLAB crashed please post it or send it to Technical Support.

[Paula]

What I meant by "crash" was, the "Physical Memory Usage” in the Task Manager hit the maximum. Matlab keeps calculating, but I'm not able to stop it, there's no response if I click anywhere in the Matlab window. I have to click "End Task" in the Task Manager to force Matlab to shut down.

As for the computer, there's no blue screen, nor does it shut down. I think I should be able to use other applications on the computer, even when Matlab "crashed".

However, I did not find a corresponding crash log file, sorry. I entered the command "ls...", and I did find a list of crash files in the temporary directory, but all of them are of previous times when Matlab crashed.

I have a new "finding" to report please:

I just tried to run my script again:

...

A1 = B * C;

A = A1 + A2;

I put breakpoints so that these two lines of code were run individually. This time, the problem is different:

Matlab did not crash at the line "A1 = B * C" this time. But the “Physical Memory Usage” in the Task Manager approaches the maximum.

Then when I click "Continue" and let the line "A = A1 + A2" to be calculated, Matlab stopped and an error message showed: "Out of memory."

At the time when I posted my original question, I had the exact same code and the same matrices, and the problem was that Matlab "crashed", I had to end it from the Task Manager. I don't know why it's a different problem and a different error message when I ran it this time.

I will keep looking for crash log files.

Thank you!