How to set up a deep neural network input so that it inputs both an image (X by Y greyscale) AND 5 numerical values for each output.

1 view (last 30 days)
Thomas Hyatt
Thomas Hyatt on 6 Jun 2021
Hello, I am trying to train a deep neural network that classifies based on the data present in an [X by Y] grayscale image. However, prior to this, 5 regression values are being calculated from the image as well. I am trying to make a network that takes both the image, and the 5 regression values as inputs for each output.
I currently am using a table where the first column is the path to each image, and the second column is the classifier. However, I believe I learned that the trainNetwork function can take an arbitrary number of inputs paired with each output.
However, how should I format the network when the first column is an image file, and the next 5 are regression values, followed finally by the classifier? The image input layer can only take the image of course, so how do I also tell the network to input the 5 regression values?
I would really like to avoid breaking down the image into a sequence of values followed by the regression values for use in a sequenceInputLyaer - the images are large and there are hundreds of thousands of them.
Thanks for the advice!

Answers (1)

Niccolò Dal Santo
Niccolò Dal Santo on 8 Jun 2021
Hi Thomas,
Currently trainNetwork does not support networks which have multiple heterogeneous inputs like your case (an image and a set of feature). However you can achieve what you want with a custom training loop.
This example might be helpful to start with: Train Network on Image and Feature Data
It trains a classification network with two heterogeneous inputs: a feature input (which can be used for your 5 regression values) and an image input.
Hope this helps.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!