Why I got an error like :"Not enough memory resources when MATLAB runs on a super computer with 512GB RAM

I have a script that reads an excel file load their data into the MATLAB workspace. The script then does some calculation and creates a new big excel file. My script breaks almost at the end of the process, right before creating the result excel file. the error is as follow:

Invoke Error, Dispatch Exception: Not enough memory resources are available to complete this operation.

but why I am getting this error when my computer has a RAM of 512GB. (is a cloud computer)

I have put a breakpoint and see the variables loaded in the workspace, see below:

the maximum size is only 1GB (fourth from the top), is this the issue?

Thanks

[Ismael]

just digged into this and found that the code creates an excel Object (in memory) in which the data, read in buch of > 2000000 points, is being populated. I believe that this is the issue as this excel object is getting bigger at each loop and fails at the end of the code when it tries to save the object (with all data size as shown above) into a file.

How to avoid using the excel object?

[Walter Roberson]

A table that is 1 gigabyte in binary would typically expand to more than 4 gigabytes in text form. Text form would be used internally in creating xlsx files -- xlsx files are internally a "zip" of a directory of XML files, with the XML files being text files.

Answer 1

[Nathan]

Hi Ismael,

Your script seems to be working with an excel COM object created by an actxserver call, and then writing your table. The invoke error comes from an excel COM object (workbook, sheet, range, etc.) trying and failing to write the table. You could try a few things here:

1. Instead of using COM/excel objects to import data, you can try parsing the input excel files with the 'readtable(file_name, FileType='spreadsheet')' function. This may require some rewriting, and may not work if the data in your sheet comes from excel features like graphs or pivot tables; 'readtable' is good for reading rows and columns of information.
2. If you have access to the computer's resource monitor program, I'd recommend watching the RAM usage over time to see when it spikes. If it constantly goes up over time, you are likely correct that the excel object is opening, reading, and then not closing its data. It may be opening many workbooks in the application object, and not closing, or may sheets in the workbook object, etc. I would specifically see if you can force the problematic object to close in each loop. You can call the objects' methods similarly to Visual Basic for Applications if you're familiar, and browse the properties/methods (functions) with MATLAB's intellisense. You can also check out the Microsoft data for these objects, for example https://learn.microsoft.com/en-us/office/vba/api/excel.worksheet for worksheets. It's probably object.Close(), or something similar depending on what it is.
3. Instead of using COM/excel objects to generate the new spreadsheet, use the 'writetable' function, which can generate excel files or other formats like .csv that the excel application can easily import.

Hope that helps!

-Nathan