ImageDataStore and tall array, How to use to save Labels and 4D Matrices in for loop?
5 views (last 30 days)
Show older comments
I generate 4D matrices of images (Double) and their respective labels (Double) with huge sizes in each iteration,
for example in one iteration I get images with size 120*120*1*6000 where 6000 the number of images, 1 is the number of channels, and their respective lables 1*6000
How to use ImageDataStore and tall array to store Labels and 4D Matrices in for loop in each iteration to use it then in further analysis in machine learning?
Also, Is there any other efficient solution to deal with huge data?
Note that the number of images and their labels vary in each iteration
4 Comments
Answers (3)
Jorrit Montijn
on 7 Nov 2023
I'm not sure what your workflow exactly looks like, but if you transform your images from double-precision floating numbers (64 bits per pixel) to uint8 (8 bits per pixel) you'll only need 1/8th of the memory capacity.
In most cases this won't even cause a loss of data, as images are often encoded in 8-bit depth anyway. In each iteration, you can then transform the (set of) image(s) you're working on back to double-precision floats so you don't have to change your workflow.
Transform the whole stack:
Ims_uint8 = cast(Ims_double*255,'uint8');
Then in each iteration:
Ims_double = cast(Ims_uint8,'double')/255;
2 Comments
Matt J
on 7 Nov 2023
imageDatastores are provided for situations like that.
8 Comments
Matt J
on 13 Nov 2023
do you mean that I have to save all the files first then put them in the imds? What is the benefits of doing that??
Yes. That way, the images are stored on disk, rather than consuming RAM.
Image Analyst
on 7 Nov 2023
Cell arrays are very inefficient compared to regular numerical arrays. They use a lot more memory. You could even use single instead of double to preallocate using half the memory.
Not sure what you're doing but you may be doing what a lot of beginners do and that is to read ALL the input images into one huge array in a loop, then after the loop process each slice of the huge array as an individual image. This is usually NOT the way to process a sequence of images. You usually (if it's a series of 2-D images you want to analyze and not a volumetric image that you're reading in slice-by-slice) want to have a loop where you read in one image at a time and then analyze it immediately in that same iteration. See the FAQ:
If you must read all the images into memory at once time, then try to use a single array rather than a cell array.
1 Comment
See Also
Products
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!