# Desired object produced but still receive "out of memory error"

2 views (last 30 days)
I encountered something quite odd and was hoping someone could provide some insight.
I have a very large variable ("W") that I need to run through some logical indexing in the form of a for loop. The end result should be a 110000 X 5000 X 401 3D matrix, but 220.5 billion numbers is a bit much for memory to handle, even if I make it a sparse array. You can see I've broken "W" up into 11 parts to see if that works.
This code ran for me and produced the object "b_1", but I still got an out of memory error. The out of memory error suggests to me that the object should have never been produced. I'm not sure that I trust the produced object "b_1" because of the out of memory error. Am I being too paranoid, or would it be wise to look for a different solution?
T = 5000
W = zeros(110000, 5000);
for trial = 1:T
X2ri = X;
X2ri(10001:20000, 46:50) = randi([1,5], 10000, 5);
X3ri = X2ri;
X3ri(20001:30000, 41:50) = randi([1,5], 10000, 10);
X4ri = X3ri;
X4ri(30001:40000, 36:50) = randi([1,5], 10000, 15);
X5ri = X4ri;
X5ri(40001:50000, 31:50) = randi([1,5], 10000, 20);
X6ri = X5ri;
X6ri(50001:60000, 26:50) = randi([1,5], 10000, 25);
X7ri = X6ri;
X7ri(60001:70000, 21:50) = randi([1,5], 10000, 30);
X8ri = X7ri;
X8ri(70001:80000, 16:50) = randi([1,5], 10000, 35);
X9ri = X8ri;
X9ri(80001:90000, 11:50) = randi([1,5], 10000, 40);
X10ri = X9ri;
X10ri(90001:100000, 6:50) = randi([1,5], 10000, 45);
X11ri = X10ri;
X11ri(100001:110000, 1:50) = randi([1,5], 10000, 50);
XFri = X11ri;
W(:,trial) = var(XFri,0,2);
end
W_1 = W(1:10000,:);
W_2 = W(10001:20000,:);
W_3 = W(20001:30000,:);
W_4 = W(30001:40000,:);
W_5 = W(40001:50000,:);
W_6 = W(50001:60000,:);
W_7 = W(60001:70000,:);
W_8 = W(70001:80000,:);
W_9 = W(80001:90000,:);
W_10 = W(90001:100000,:);
W_11 = W(100001:110000,:);
varinc = (0:0.01:4);
b_1 = ndSparse.build([10000,5000,401]);
for i = 1:length(varinc)
b_1(:,:,i) = W_1 >= varinc(i);
end

Walter Roberson on 26 Sep 2020
you are using https://www.mathworks.com/matlabcentral/fileexchange/29832-n-dimensional-sparse-arrays
You need to use nzmax to preallocate the object. Otherwise every time you assign into the object, matlab needs to create a copy of the array with one more value slot and copy the old one over.
Hi Walter,
Thank you again for your insight! This has given me a lot to think about, but it will make my future code much better.