Storing 3 large data sets in one workspace

Hello i have 3 data sets from NESO for System Inertia, System Frequency and Historic generation mix and carbon intensity.

They are quite large in size, just wondering how to put into one workspace for data analysis.

[Aquatris]

what is the problem you encounter when you try to import all 3 datasets to your workspace?

Unless you have memory limitations, this should not be a problem

Answer 1

[Aastha]

Hi Mark,

I understand that you are working with three large datasets, and you are looking for a way to load them into the workspace while managing memory constraints.

You can use "datastore" function in MATLAB, which is specifically designed for handling collections of data that are too large to fit into memory.

Kindly refer to the example mentioned below illustrating the use of "datastore" function for large datasets:

1. Configure the datastore function using the MATLAB code given below:

ds = tabularTextDatastore("large_file.csv"); % Create a datastore for tabular text data 
ds.SelectedVariableNames = {'Column_Name_1', 'Column_Name_2'}; % Specify columns to be read from the file 
ds.TreatAsMissing = 'NA'; % Replace missing data with NA 
ds.ReadSize = 15000; % Define the number of rows to be read at once based on memory constraints 

2. Use a while loop along with a "hasdata" function to determine if there is data available to read the data from the "datastore" function as shown in the MATLAB code below:

reset(ds); % reset the datastore to the initial state 
while hasdata(ds) 
    T = read(ds); % Read data from the datastore 
    % Process the intermediate data 
    % Insert your processing code here 
end 

This method allows you to handle datasets too large to fit into memory efficiently. Additionally, you can explore other methods for managing large datasets by referring to the link given below:

https://www.mathworks.com/help/matlab/large-files-and-big-data.html?s_tid=CRUX_lftnav

For more information on the "datastore" and "hasdata" function, kindly refer to the link of MathWorks documentation mentioned below:

-datastore: 

https://www.mathworks.com/help/matlab/datastore.html

-hasdata: 

https://www.mathworks.com/help/matlab/ref/matlab.io.datastore.hasdata.html

I hope this is helpful!