Efficient work on Array of objects (User-defined class) larger than memory

My code relies on the creation of objects from a user-defined class (Signal), which can be a few MB each.

Until now it stores them in a simple array. But with multi thousands of objects I run quickly out of memory. This should be avoidable as each object is worked on consecutively, so it could be created, written and read to/from the SDD without influencing the others.

But here comes the problem. How do I do that, and keep the code fast? I don't want to write out each file seperately as that's a mess and very slow.

So my first solution was to use matfile which can just partially read and write files

signalmat= matfile(['Simulator_Results',Folder,'\DataGeneratorOutput\signalArray'],'Writable',true);
...
signalmat.signal(i,1)=signal

This does not work as matfile does not support indexing INSIDE user defined classes? (I don't want to go inside the class, just access the array of classes)

So I got it to work by tricking it into a cell array.

signal={}
for i=1:(steps*samplesperSet)
   signal{i}=Signal(width,height);
end
save(['Simulator_Results',Folder,'\DataGeneratorOutput\signalArray'],'signal','-v7.3');
signalmat= matfile(['Simulator_Results',Folder,'\DataGeneratorOutput\signalArray'],'Writable',true);
...
signalmat.signal(i,1)={signal}

Obviously pre allocation in memory is a terrible idea, as it will also run out of memory. But this code is also crazy slow when updating objects. And maybe even worse it seems that with each new object that is written it does exactly what it was not supposed to doo and reads in part of the file, filling up more and more of the memory.

So I thought I try the forbidden route using eval

eval(['signal',num2str(i),'=signal;']);
nameFile=['signal',num2str(i)];
save(['Simulator_Results',Folder,'\DataGeneratorOutput\signalArray'],nameFile,'-v7.3','-append');
eval(['clear ','signal',num2str(i)]);

But the same behaviour. Appending the file is very slow, and consumes more and more RAM as you add more and more files.

What is the solution? I do want to keep the system based on objects from my class. How can I store them efficiently?

PS: Would be also nice to keep the compression efficiency of storing a simple V7.3 file, as the data can be well compressed.

Answer 1

[Nipun]

Hi Clemens,

I understand that you are looking for a way to store the system based objects from your class efficiently, when the variable size may exceed the memory size. I should remark that, modulo compression, swapping is an efficient way of retrieving and storing data chunks that exceed memory size.

Based on the provided information, here are few recommendations to compress chunk size:

1. Use a Database: If your data is becoming too large to fit in memory, consider using a database system. MATLAB supports various database interfaces, and you can store your objects in a database table. This allows you to retrieve and update individual objects efficiently.
2. Serialization: Serialize your objects into a binary format and store them as separate files. You can use the save function with the '-struct' option to save the properties of your objects. This way, you can load only the necessary data when needed. I have attached a simplified example using serialization at the end.
3. MAT-File Compression: When using the save function to store data in MAT-files, you can specify the '-v7.3' option, which uses HDF5 format. This format supports compression. Additionally, you can specify compression options using the '-nocompression', '-zlib', or '-lz4' flags. Experiment with different compression options to find the balance between compression ratio and performance.

Example using serialization

% Save objects individually
for i = 1:(steps * samplesperSet)
    signal = Signal(width, height);
    save(['Simulator_Results', Folder, '\DataGeneratorOutput\signal_', num2str(i), '.mat'], 'signal', '-v7.3');
end

% Load a specific object when needed
load(['Simulator_Results', Folder, '\DataGeneratorOutput\signal_', num2str(k), '.mat'], 'signal');

Saving data:

save('data.mat', 'yourData', '-v7.3');
% You may try this too:
save('data.mat', 'yourData', '-v7.3', '-zlib'); % Use zlib compression

Hope this helps.

Regards,

Nipun