out of memory error

7 views (last 30 days)
Mark Bodner
Mark Bodner on 28 Sep 2022
Commented: Jan on 29 Sep 2022
I'm trying to join to large arrays (rawS=43914x11 and rawP=23772085x11). My code is
out1=cellfun(@num2str,rawS,'un',0)
out2=cellfun(@num2str,rawP,'un',0)
[idxP,idxS] = ismember(out2(:,1),out1(:,1));
out=rawS(idxS(idxP),[1,2,3,4,5,6,7,8,9,10,11]);
out(:,[12,13,14,15,16,17,18,19,20,21,22]) - rawP(idxP,[1,2,3,4,5,6,7,8,9,10,11]);
This ran just fine the first time I ran it, but runs out of memory now (seemingly at different points each time) on the second line for "out2" . No other programs or processes are running. I Tried restarting the computer of course, using tall arrays, deleting variables after they are not needed and so forth, but nothing seems to works. Short of going next to distributed arrays to parallel workers, I would appreciate greatly any ideas or suggestions. Thanks.
  5 Comments
Mark Bodner
Mark Bodner on 28 Sep 2022
Sorry, I had accidently mistyped the code--the last line is actually
out(:,[12,13,14,15,16,17,18,19,20,21,22]) = rawP(idxP,[1,2,3,4,5,6,7,8,9,10,11]);
The installed RAm is 64 GB.
The code with the tall approach was simply:
t=tall(rawP);
out1=cellfun(@num2str,rawS,'un',0)
out2=cellfun(@num2str,t,'un',0)
[idxP,idxS] = ismember(out2(:,1),out1(:,1));
out=rawS(idxS(idxP),[1,2,3,4,5,6,7,8,9,10,11]);
out(:,[12,13,14,15,16,17,18,19,20,21,22]) = rawP(idxP,[1,2,3,4,5,6,7,8,9,10,11]);
Jan
Jan on 29 Sep 2022
@Mark Bodner: Again, the code would be better to read, if you replace [12,13,14,15,16,17,18,19,20,21,22] by 12:22 .

Sign in to comment.

Accepted Answer

Walter Roberson
Walter Roberson on 28 Sep 2022
Pre-allocate out to the final size. This will reduce temporary memory use. Also, using : is more memory efficient than indexing at the list of columns.
out = zeros(numel(idxP), 22);
out(:, 1:11) = rawS(idxS(idxP), :);
out(:, 12:24) = rawP(idxP, :);

More Answers (0)

Categories

Find more on Startup and Shutdown in Help Center and File Exchange

Products


Release

R2022a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!