Data format error with training a ResNet for semantic segmentation

I am trying to train a ResNet for the purpose of a pixel classification problem with two classes. The data consists of 256x256 RGB images, groundtruth is 256x256 Categorical. However there seems to be a conflict of data formats in regard to the final softmax layer of the network. Here is an excerpt from my code along with the error message.

input_size = [256 256 3];
numClasses = 2;

resNet = resnetNetwork(input_size, numClasses);

options = trainingOptions('adam', ...
    'InitialLearnRate',1e-3, ...
    'MaxEpochs',30, ... 
    'MiniBatchSize',64, ...
    'Metrics','accuracy', ...
    'Plots','training-progress', ...
    'ValidationFrequency',25, ...
    'ValidationData',validationData);


trainedResNetGreen = trainnet(trainingData,resNet,@(Y,T) modelLoss(Y,T,classWeights),options);

Error using trainnet (line 46)

Error forming mini-batch of targets for network output "softmax". Data interpreted with format "BC". To specify a different format,

use the TargetDataFormats option.

Error in ResNetGreen (line 37)

trainedLCUnetGreen = trainnet(trainingData,resNet,@(Y,T) modelLoss(Y,T,classWeights),options);

Caused by:

Batch dimension of datastore must match the format batch dimension (1).

I already tried to specify the Target Data Format as "SSBC" which corresponds to "Spatial, Spatial, Batch, Channel" but it did not fix the problem. Am I missing something?

[Matt J]

Since the problem is in the data format, we need to see the data you are giving it.

[Jaimin]

Hi @Jannik Menke,

Could you please provide relevant data for better understanding?

Answer 1

[Milan Bansal]

Hi Jannik Menke,

From the mentioned error, I understand that the program is not able to minibatches due to differences in data formats while performing semantic segmentation using resnetNetwork.

By default, resentNetwork gives you a neural network equivalent to Resnet-50, which can only be used for classification instead of semantic segmentation. Due to this reason, the model expects the target to be a categorical array, hence the output layer of the model has "BC" data format. You can verify this using the analyzeNetwork function. The target data provided by you are label images, which will have the data format "SSBC" and will not be compatible with the model.

If you wish to use the Resnet Network for semantic segmentation, you will need to modify the model manually. Instead, it is better to use the inbuilt models in MATLAB, which can use Resnet as the backbone. You can use deeplabv3plus, which can have the Resnet-18 or Resnet-50 as the backbone.

Please refer to the following documentation links to learn more about resnetNetwork and deeplabv3plus.

Please refer to the following documentation links to learn more about resnetNetwork and deeplabv3plus and analyzeNetwork.

• https://www.mathworks.com/help/releases/R2024a/deeplearning/ref/resnetnetwork.html
• https://www.mathworks.com/help/releases/R2024a/vision/ref/deeplabv3plus.html
• https://www.mathworks.com/help/releases/R2024a/deeplearning/ref/analyzenetwork.html

Hope this helps!

[Jannik Menke]

I already figured out that the problem originates from the spatial information getting lost as this specific Resnet is intended for classification. I almost came to terms with having to modify the model manually but your recommendation of deeplabv3plus comes very handy. Thank you very much!