Layer-structure prediction of fitrnet not yielding same answer as predict(Model,input)

Greetings,

I've created a single-layer regression neural network using the fitrnet command. The network has 3 inputs and 1 hidden layer comprised of 2 nodes. The activation is rectified linear unit.

In trying to understand how all the layers link, I followed the instructions at the link Predict Using Layer Structure of Regression Neural Network Model. The result I get using that flow do not agree with the prediction made using the predict tool, nor do they agree with my hand-written math.

Input = [30.9 30.7 0.244]

Mdl.LayerWeights{1} = [0.9914 1.4559 -1.0186; 10.2493 -7.8115 -6.3606]

Mdl.LayerBiases{1} = [-1.6804 0.3631]

Mdl.LayerWeights{end} = [-1.1448 3.3748]

Mdl.LayerBiases{end} = [8.1000]

Predict(Mdl,[30.9 30.7 0.244]) = 21.7515

firststep=(Mdl.LayerWeights{1})*Input'+Mdl.LayerBiases{1} = [73.4008; 75.7016]

relustep=max(firststep,0) = [73.4008; 75.7016]

finalstep=(Mdl.LayerWeights{end})*relustep+Mdl.LayerBiases{end} = 179.5503

I get the same value as finalstep when I do the math by hand. The "Predict" response is the correct response as compared to my actual result.

I'm at a loss at this point, and I'm not sure where to look in the workspace for what might have gone wrong. Are there any dot-notation properties I should inspect? Thank you in advance.

[Anjaneyulu Bairi]

can you provide the entire code and reproductions steps?

Thanks.

[Kevin]

I'm not sure what else to provide. The neural network is built with the fitrnet tool:

Mdl=fitrnet(XTraining,YTraining,"Standardize",true,"Activations","relu","LayerSizes",[2])

From there it trains until it reaches the default end-points, and then I call the stepwise calculation of the prediction using the "firststep," "relustep," and "finalstep" prompts.

Can you please be more specific as to what you might want to see?

[Sam Chak]

Hi @Kevin

I believe that @Anjaneyulu Bairi attempted to assist you by requesting that you provide the complete executable code, as shown in the example below, so that the prediction deviation issue can be thoroughly investigated by users in this forum. You can click the indent icon to toggle the coding field and paste the code. Then, click the Run icon to execute the code in the browser.

Example:

load patients
tbl = table(Diastolic, Height, Smoker, Weight, Systolic);

rng("default")
c               = cvpartition(size(tbl, 1), "Holdout", 0.30);
trainingIndices = training(c);
testIndices     = test(c);
tblTrain        = tbl(trainingIndices, :);
tblTest         = tbl(testIndices, :);
Mdl             = fitrnet(tblTrain, "Systolic", "Standardize", true, "IterationLimit", 50);
predictedY      = predict(Mdl, tblTest);

plot(tblTest.Systolic, predictedY, ".")
hold on
plot(tblTest.Systolic, tblTest.Systolic)
hold off
xlabel("True Systolic Blood Pressure Levels")
ylabel("Predicted Systolic Blood Pressure Levels")

●

[Kevin]

Apologies. I did not see the notifications for these responses. I've attached my training data, which has been relabeled, as well as a new version of my code. I had to create new code and in doing so, could not recreate my original issue with a 2-node model: the relu step would reduce the interim response to zero, leaving the model to output the final layer's bias as the response. I created a 6-node model and pasted the code. Thanks in advance!

Data=xlsread('datamatlabhelp.xlsx');
input1=[Data(:,1)];
input2=[Data(:,2)];
input3=[Data(:,3)];
output=[Data(:,4)];
X=[input1 input2 input3];
Y=output;
rng("default");
c=cvpartition(length(Y),"Holdout",0.2);
trainingIdx=training(c);
XTrain=X(trainingIdx,:);
YTrain=Y(trainingIdx);
testIdx=test(c);
XTest=X(testIdx,:);
YTest=Y(testIdx);
Mdl=fitrnet(XTrain,YTrain,"Standardize",true,"Activations","relu","LayerSizes",[6]);
Input=[30.9 30.7 0.244];
firststep=(Mdl.LayerWeights{1})*Input'+Mdl.LayerBiases{1}

firststep = 6×1

6.4763 -86.2128 8.6041 -14.4905 -0.9122 57.4965

<mw-icon class=""></mw-icon>

<mw-icon class=""></mw-icon>

relustep=max(firststep,0)

relustep = 6×1

6.4763 0 8.6041 0 0 57.4965

<mw-icon class=""></mw-icon>

<mw-icon class=""></mw-icon>

finalstep=(Mdl.LayerWeights{end})*relustep+Mdl.LayerBiases{end}

finalstep = 181.1441

predict(Mdl,Input)

ans = 22.0551

Answer 1

[Sam Chak]

Hi @Kevin

The reason for the discrepancy is that you have set "Standardize" to true, which standardizes the predictor data by centering and scaling each numeric predictor variable. If you leave "Standardize" at its default setting, both predictions will match each other.

T   = readtable("datamatlabhelp.xlsx", VariableNamingRule="preserve")

T = 140×4 table
    Input1    Input2    Input3     Output
    ______    ______    _______    ______

    115.7     115.7     0.17073     40.9 
    115.7     115.7     0.17073     42.1 
    115.7     115.7     0.17073     38.8 
    115.7     115.7     0.17073     41.7 
    115.7     115.7     0.17073       41 
    115.7     115.7     0.17073     39.7 
    115.7     115.7     0.17073     40.7 
    115.7     115.7     0.17073     42.9 
    115.7     115.7     0.17073     39.5 
    115.7     115.7     0.17073     40.2 
    115.7     115.7           1     12.2 
    115.7     115.7           1     11.7 
    115.7     115.7           1      9.8 
    115.7     115.7           1      9.2 
    115.7     115.7           1     11.4 
    115.7     115.7           1      8.5 

%% Data
Data=xlsread('datamatlabhelp.xlsx');
input1=[Data(:,1)];
input2=[Data(:,2)];
input3=[Data(:,3)];
output=[Data(:,4)];
X=[input1 input2 input3];
Y=output;

%% Training of Neural Nets
rng("default");
c=cvpartition(length(Y),"Holdout",0.2);
trainingIdx = training(c);
XTrain      = X(trainingIdx,:);
YTrain      = Y(trainingIdx);
testIdx     = test(c);
XTest       = X(testIdx,:);
YTest       = Y(testIdx);
Mdl         = fitrnet(XTrain, YTrain, "Activations", "relu", "LayerSizes", 6)

Mdl = 
  RegressionNeuralNetwork
             ResponseName: 'Y'
    CategoricalPredictors: []
        ResponseTransform: 'none'
          NumObservations: 112
               LayerSizes: 6
              Activations: 'relu'
    OutputLayerActivation: 'none'
                   Solver: 'LBFGS'
          ConvergenceInfo: [1×1 struct]
          TrainingHistory: [3×7 table]

testMSE     = loss(Mdl, XTest, YTest)

testMSE = 149.9581

%% Test prediction
Input       = [30.9 30.7 0.244];
firststep   = Mdl.LayerWeights{1}*Input'     + Mdl.LayerBiases{1};
relustep    = max(firststep, 0);
finalstep   = Mdl.LayerWeights{end}*relustep + Mdl.LayerBiases{end}

finalstep = 18.0152

predictedY  = predict(Mdl, Input)

predictedY = 18.0152

% Test if the prediction by formula matches the one returned by the predict() object function
isequal(finalstep, predictedY)

ans = logical
   1

[Kevin]

This is interesting, thank you, @Sam Chak. Is there a way to "standardize" my test inputs ("Input") in order to get the correct response? I have a series of models with standardized inputs that work quite well, and I'd like to demonstrate how they work in that stepwise fashion.

One thing I forgot to mention (I realized this just now): the predict(Mdl,Input) response in my prompt is the correct answer. I updated my original prompt in case anyone else sees this thread.

Answer 2

[Sam Chak]

Hi @Kevin

If you want to use the satisfactorily trained neural network models with standardized inputs, you will need to apply the standardized inputs instead of the original input values. The formula remains unchanged.

T   = readtable("datamatlabhelp.xlsx", VariableNamingRule="preserve")

T = 140×4 table
    Input1    Input2    Input3     Output
    ______    ______    _______    ______

    115.7     115.7     0.17073     40.9 
    115.7     115.7     0.17073     42.1 
    115.7     115.7     0.17073     38.8 
    115.7     115.7     0.17073     41.7 
    115.7     115.7     0.17073       41 
    115.7     115.7     0.17073     39.7 
    115.7     115.7     0.17073     40.7 
    115.7     115.7     0.17073     42.9 
    115.7     115.7     0.17073     39.5 
    115.7     115.7     0.17073     40.2 
    115.7     115.7           1     12.2 
    115.7     115.7           1     11.7 
    115.7     115.7           1      9.8 
    115.7     115.7           1      9.2 
    115.7     115.7           1     11.4 
    115.7     115.7           1      8.5 

%% Data
Data    = xlsread('datamatlabhelp.xlsx');
input1  = [Data(:,1)];
input2  = [Data(:,2)];
input3  = [Data(:,3)];
output  = [Data(:,4)];
X       = [input1 input2 input3];
Y       = output;

%% Training of Neural Nets
rng("default");
c           = cvpartition(length(Y), "Holdout", 0.2);
trainingIdx = training(c);
XTrain      = X(trainingIdx,:);
YTrain      = Y(trainingIdx);
testIdx     = test(c);
XTest       = X(testIdx,:);
YTest       = Y(testIdx);
Mdl         = fitrnet(XTrain, YTrain, "Standardize", true, "Activations", "relu", "LayerSizes", 6)

Mdl = 
  RegressionNeuralNetwork
             ResponseName: 'Y'
    CategoricalPredictors: []
        ResponseTransform: 'none'
          NumObservations: 112
               LayerSizes: 6
              Activations: 'relu'
    OutputLayerActivation: 'none'
                   Solver: 'LBFGS'
          ConvergenceInfo: [1×1 struct]
          TrainingHistory: [59×7 table]

testMSE     = loss(Mdl, XTest, YTest)

testMSE = 3.1202

%% Test prediction
Input       = [30.9, 30.7, 0.244];      % original input values
muX         = mean(XTrain)              % mean

muX = 1×3

110.4939 110.9779 0.5895

<mw-icon class=""></mw-icon>

<mw-icon class=""></mw-icon>

sigmaX      = std(XTrain)               % standard deviation

sigmaX = 1×3

45.9075 46.1296 0.2877

<mw-icon class=""></mw-icon>

<mw-icon class=""></mw-icon>

InputStd    = (Input - muX)./sigmaX     % center and scale the input data

InputStd = 1×3

-1.7338 -1.7403 -1.2008

<mw-icon class=""></mw-icon>

<mw-icon class=""></mw-icon>

% Prediction using the formula by applying the standardized input
firststep   = Mdl.LayerWeights{1}*InputStd'  + Mdl.LayerBiases{1};
relustep    = max(firststep, 0);
finalstep   = Mdl.LayerWeights{end}*relustep + Mdl.LayerBiases{end}

finalstep = 22.0551

% Prediction using the predict() object function
predictedY  = predict(Mdl, Input)

predictedY = 22.0551

% Test if the prediction by formula approximately matches the one returned by the predict() function
TF          = isapprox(finalstep, predictedY, AbsoluteTolerance=1e-10)

TF = logical
   1

[Kevin]

That is perfect. Thank you, @Sam Chak!