Image Classification Returning Different Results on Different Computers

Hello,

I am running a neural network for image classification using the Matlab deep learning toolbox. The algorithm contains three instances of convolution2dLayer, and it trains using TrainNetwork. If I run the exact same algorithm (identical code) on a different computer, my algorithm returns vastly different results: one computer yields a very high accuracy, around 99%, while the other never learns at all (the learning curve never increases above 50%). The only difference between the two computers is the Matlab version. The 99% accurate computer is on Matlab 2019a (Deep Learning Toolbox, version 12.1), while the 50% accurate computer is on Matlab 2019b (Deep Learning Toolbox, version 13.0). Is it expected that these two versions, which are so close together, would return such different results? I also find it surprising that the older version is better, but as machine learning is a blackbox, perhaps this is entirely possible? Are there any other reasons the results could be so different?

Answer 1

[Mahesh Taparia]

Hi

This is an unexpected behaviour. Check if all the data preprocessing steps are same, also the training parameters, number of epochs in both the network, weight initialization and other hyperparameters should be same. If the problem still remains, then share your code and relevant information, it will be helpful.

[Sofia Brown]

Hi,

The parameters are indeed the same. Here are the important parts of my code. The images I am training on are of optical fiber output patterns at different polarization angles, 0 and 4. These different angles are already very similar, which makes the training difficult. I am also a bit new to machine learning, so any advice is much appreciated.

rootFolder = 'TrainingAll5Sets';
categories = {'0deg', '4deg'};
imds = imageDatastore(fullfile(rootFolder, categories), 'LabelSource', 'foldernames','FileExtensions','.png');
%Define Layers
layers = [
    imageInputLayer([256 320 1])

    convolution2dLayer(20,8,'Padding',2)
    batchNormalizationLayer
    reluLayer

    maxPooling2dLayer(2,'Stride',2)

    convolution2dLayer(40,16,'Padding','same')
    batchNormalizationLayer
    reluLayer

    maxPooling2dLayer(2,'Stride',2)


    convolution2dLayer(40,32,'Padding','same')
    batchNormalizationLayer
    reluLayer

    fullyConnectedLayer(2)
    softmaxLayer
    classificationLayer];

%Set training options 
options = trainingOptions('sgdm', ...
    'InitialLearnRate',0.0001, ...
    'LearnRateSchedule', 'piecewise', ...
    'LearnRateDropPeriod', 10, ...
    'LearnRateDropFactor', 0.1, ...
    'MaxEpochs',15, ...
    'Shuffle','every-epoch', ...
    'Verbose',false, ...
    'Plots','training-progress');


%Train
[net, info] = trainNetwork(imds, layers, options);

%Test one at a time
figure();
labels = classify(net, imds_test);

ii = randi(2);
im = imread(imds_test.Files{ii});
imshow(im);
if labels(ii) == imds_test.Labels(ii)
   colorText = 'g'; 
else
    colorText = 'r';
end
title(char(labels(ii)),'Color',colorText);

%Test all the data at once and assess the accuracy
confMat = confusionmat(imds_test.Labels, labels);
confMat = confMat./sum(confMat,2);
mean(diag(confMat))

Thank you!

[Mahesh Taparia]

Hi

Try with setting the same random seed at the begining of your code using rng command in both the systems, then it won't be much difference. It will generate same random weights. Different weights initialization may take different number of epochs in loss convergence.

[Sofia Brown]

Hello,

Thank you very much for this response. I am sorry for my months long delay in writing back - I took a break from this project for the summer. I hope you are still able to be of assistance.

So, if I understand correctly, I just need to initialize a random seed value at the beginning of my code, like this (Using, say, 0 as the random seed):

rootFolder = 'TrainingAll5Sets';
categories = {'0deg', 'eighthdeg'};
rng(0);
%imds = imageDatastore(fullfile(rootFolder, categories), 'LabelSource', 'foldernames');
imds = imageDatastore(fullfile(rootFolder, categories), 'LabelSource', 'foldernames','FileExtensions','.png');
%Define Layers
layers = [
    imageInputLayer([256 320 1])

    convolution2dLayer(1,5,'Padding',2)
    batchNormalizationLayer
    reluLayer

    maxPooling2dLayer(2,'Stride',2)

    convolution2dLayer(6,15,'Padding','same')
    batchNormalizationLayer
    reluLayer

    maxPooling2dLayer(2,'Stride',2)


    convolution2dLayer(12,40,'Padding','same')
    batchNormalizationLayer
    reluLayer

    fullyConnectedLayer(2)
    softmaxLayer
    classificationLayer];

%Set training options - use default options from 7.15.20
options = trainingOptions('sgdm', ...
    'InitialLearnRate',0.00001, ...
    'MaxEpochs',300, ...
    'Shuffle','every-epoch', ...
    'Verbose',false, ...
    'Plots','training-progress');

%Train
[net, info] = trainNetwork(imds, layers, options);
    

I have not changed any parts of the code except line 4. By just making this small change, will it generate the same random weights during each run, as you mentioned? I'm not sure I completetly understand how rng works in relation to the rest of the code and if I need to incorporate it anywhere else.

Thank you!