How to save .mat files on the disk after each iteration?
Show older comments
I am facing difficulties how to save the data as struct after each loop.
I have huge image matrix and its respective labels matrix ,I want to save them after each iteration in a folder, and then delete them after each iteration from workspace, any suggestion?
Please note that I want to save the data in the disk and clear them after each iteration and save
Accepted Answer
Dyuman Joshi
on 14 Nov 2023
The values will be over-written in each iteration, so you don't need to clear them.
for k = 1:number_of_iterations
%get images and labels for this iteration
%save them as MAT files, the variable names should be provided as a string or a char vector
%As you have mentioned that there is a large data for each iteration
%use the (MAT File) version 7.3
save("images"+k+".mat", "images", "-v7.3")
save("labales"+k+".mat", "labels", "-v7.3")
end
23 Comments
M
on 14 Nov 2023
Stephen23
on 14 Nov 2023
"how to create a folder with a specific name to save them into it? "
M
on 14 Nov 2023
Dyuman Joshi
on 14 Nov 2023
I assume you want to make a new folder in the current directory -
%Make a new folder
mkdir FolderName
%Change the current directory to the new folder
cd FolderName
Dyuman Joshi
on 14 Nov 2023
"also I read that the Variables Should Not Be Named Dynamically, instead we have to use structure to save them"
Yes, it's best to not store them dynamically.
"Do you have any idea about this matter please?"
What do you want to do with the saved data?
Do you want to do some Image Processing with the image data?
"I was reading your TUTORIAL: Why Variables Should Not Be Named Dynamically"
It is unclear how that relates to your question. Are you trying to SAVE some MAT files each with unique variable names? That would be ... bad data design.
Much better: simply use the same variable name (and SAVE any meta-data together in the same file).
"Will the above solution be efficient in case of huge dataset?"
Some users asking questions on this forum think that their 1000x1000 matrices are "huge". Some people think that their 20 Terabyte datasets are "huge". You can read here what different MAT file versions support:
Note v7.3 uses data compression which of course takes more time than no data compression.
"...and then delete them after each iteration from workspace, any suggestion? "
In most cases CLEARing (and other attempts by users to micro-manage MATLAB's memory management) is counter-productive and inefficient. Dyuman Joshi is most likely correct to advise that you do not need to clear variables. But ... it really does depend on what you are doing and how much memoery you have.
Dyuman Joshi
on 14 Nov 2023
You can write the image data as images and use the images accordingly.
Stephen23
on 14 Nov 2023
You appear to be overthinking this. What you are doing boils down to this:
A = rand(120,120,1,7000);
save test.mat A
I just ran it now on this thread. It works. Repeat for each array. Name the MAT files as you wish. Of course importing and exporting (with the corresponding data compressing and decompression) takes a finite amount of time.
Then use a DATASTORE.
Note that MAT files do not seem to be a supported image file format of IMAGEDATASTORE.
"by this I will save only the final iteration?"
Yes, which is exactly why I wrote "Name the MAT files as you wish."
"How to use it in for loop? Any suggestion please?"
See Dyuman Joshi's answer from over one hour ago:
"When I want to load some matrices and combine them in a matrix ,all of their names are the same, how can I overcome this issue?"
Having all of the names the same is good: it means that you can write simple, efficient, robust code.
Remember to always LOAD into an output variable (which is a scalar structure):
S = load(..);
and then access its fieldnames if required. Then simply use indexing to store the imported data (either the structure S or any of its fields) in an array, e.g. a numeric array, a cell array, a structure array:
"I have multiple folders, the .mat files inside are with different names, each folder contains two varibales, each with the same name (Images_A, Labels_A) but with different .mat names"
That sounds like something the DIR could handle quite easily. What have you tried so far?
"The other folders have the same concept, (Images_B, Labels_B) but with different .mat names..."
So you have "multiple folders" and also "the other folders". What exactly are "the other folders" , how are they distinguished from the "multiple folders" ?
You do not provide suffiicent information to give clear and complete advice (or testable, working code).
How to LOAD these MAT files is probably better discussed in one of your other recent threads, e.g.:
As discussed in that last thread, your data will (probably) not all fit into memory at once.
Stephen23
on 15 Nov 2023
"what I meant is that I have 12 folders, each folder contains 60 files, 30 Images 30 Labels, the .mat files have different names but the variables inside have the same name."
That sounds like something DIR would easily handle.
"For example folder1 .. 30 variables with Images_A name, and 30 variables with labels_A name but their .mat files name are all different .. folder2 .. 30 variables with Images_B name, and 30 variables with labels_B name but their .mat files name are all different"
Folders contain files and other folders. Folders do not contain variables (these only exist in MATLAB memory).
It would probably be simpler and faster if you:
- posted a screenshot of your File Explorer clearly showing the file structure,
- use WHOS() to show us a list of the content of several of the MAT files.
P = 'absolute or relative path to the parent directory';
S = dir(fullfile(P,'*Data','*.mat'));
If you want the folder and filenames sorted into alphanumeric order then either 1) add sufficient lrading zeros to the folder/filenames or 2) sort them yourself, e.g. by downloading NATSORTFILES:
S = natsortfiles(S); % DOWNLOAD https://www.mathworks.com/matlabcentral/fileexchange/47434-natural-order-filename-sort
S contains all filenames together with their folders and other file meta-data. You could loop over all files:
for k = 1:numel(S)
F = fullfile(S(k).folder,S(k).name);
D = load(F);
.. do whatever with D (except concatenate them into a 64 GB array)
end
Of course you could also filter S for specific file/folder names. Or modify the DIR match expression to only match "labels" or "TRO" at the start of the filenames... whatever.
"I cant find clear method to deal with this data properly , I want to use this data in SVD and NN but before that I want to do some computations as I mentioned in the previous thread"
If this is the same data discussed in your other threads then you (probably) cannot fit all of that data into memory. So there is likely no point in trying to concatenate all of those arrays together. But you could supply those filenames to DATASTORE (of some kind) and use tall arrays to process it.... if you have sufficient time to spare (perhaps days... or weeks... or more... who knows).
Before you decide to attempt SVD on a 64GB array you should probably do an internet search on SVD of large arrays.
I suspect that you do not really appreciate that computers do not have infinite memory and infinite speed.
"I cant do that since the name inside each .mat file is the same "
How exactly do the variable names prevent you from processing your data?
Having exactly the same variable names in every MAT file makes your data easier to work with. It also makes your code simpler, more efficient, and more robust:
What exactly is stopping you from LOADing into an output variable and accessing its fields?
Show us the exact code that you tried.
"I want to pick up from each folder 15 files of images (4D double) and their respective files of labels and combine the images and combine the labels"
In this thread you wanted to combine around 400 "images" into a 64GB array:
Now you only want to combine "15 files of images".
Which is correct? When your explanations keep changing then it makes it hard to understand what you want.
More Answers (1)
Often, you would put all images with the same label in the same folder -
Folders={'Label1','Label2','Label3'};
for j=1:numel(Folders)
mkdir(Folders{j});
for i=1:10
image=rand(10);
save( fullfile( Folders{j}, "Image_"+num2str(i,'%3d') ) , 'image')
end
end
2 Comments
Catalytic
on 14 Nov 2023
We are not here to refine code for you down to the last detail. The example clearly shows how you can create file names in a loop to use with the save() command. Adapt it as you see fit.
Categories
Find more on Images in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!
