How can I train a VDSR model with Thermal (or Grayscale) images?

Is it possible to train a VDSR model using images that are not RGB?

[Benjamin Thompson]

Please define VDSR for us.

[Ioanna Thoma]

I am sorry, I was trying to follow this example: https://www.mathworks.com/help/images/single-image-super-resolution-using-deep-learning.html but I would like to use thermal images or temperature maps

Answer 1

[KSSV]

Offcourse it is possible. You can go ahead with your images with that example. If you are stuck you are welcome with your questions.

[Ioanna Thoma]

My data include NaN values, which correspond to land (I am using sea surface temperature maps). When extracting the patches, is there a way to limit the patches where there are no NaNs?

[Image Analyst]

Yes.

TUTORIAL: How to ask a question (on Answers) and get a fast answer

[Ioanna Thoma]

which is...?

[Image Analyst]

You can get a mask of nans by using isnan()

nanMask = isnan(thermalImage);

Beyond that I'm not sure what you mean by "extracting". You forgot to define it. Do you mean cropping non-nan regions with the bounding box to a new sub-image? And then maybe saving out to disk? Or maybe just getting the mean value or stdev value in the non-nan regions?

meanValue = mean(thermalImage(:), 'omitnan');

I have no idea what "extract" means to you. Nor did you attach any images. Nor did you answer any of the questions I asked in my answer below. Maybe you need to read this link again.

[Ioanna Thoma]

Thank you for your reply. Let me try to explain a little bit better. So I am following this example:

https://www.mathworks.com/help/images/single-image-super-resolution-using-deep-learning.html?searchHighlight=VDSR&s_tid=srchtitle_VDSR_4#VeryDeepSuperResolutionDeepLearningExample-1

As input I have normalized temperature maps, instead of RGB images.

(Unfortunatelly I don't know how to upload an example as the limit is 5KB, can we consider the following matrix as an inmput example?

tst=rand(180, 456);

tst(1:5,35:89)=nan;

tst(89:160,300:410)=nan;)

At one point I need to extract patches that will be used to train the network, according to the example we have:

patchSize = [41 41];

patchesPerImage = 64;

dsTrain = randomPatchExtractionDatastore(upsampledImages,residualImages,patchSize, ...

"DataAugmentation",augmenter,"PatchesPerImage",patchesPerImage);

but as my raw data contain Nans, the extracted patches cannot be used for training.

What I would like is to use the randomPatchExtractionDatastore but limit it to the regions were there are no Nans.

[Image Analyst]

Why can't you just reassign the nan pixels to zero? Would that ruin the training? It's worth a try since it's so easy.

[Ioanna Thoma]

I tried by replacing the nan pixels to 100 as zeros also correspond to actual data but the results were pretty bad close to the coast line:/ Then I just cropped the original data so that they do not include any land, but I don't think this is a good way to solve the issue:/

Answer 2

[Image Analyst]

The demo doesn't look like it provides earth shattering improvement. Why do you think you need super high resolution? What can't you get from your image in the original resolution that you think you could if you had much higher resolution? Would you care to share your thermal image and tell us what you are trying to find in the image?