How can I train a neural network manually choosing a specific set of initial weights and biases?

18 views (last 30 days)
My goal is to reproduce many times exactly the same training in order to do some tests about some training parameters variations. So what i tried first was to define a neural network, configure it, initialize it. Then I "switched off" the init function in order to keep fixed the initial values of weights and biases and use them again later in another training. I succeeded in this by setting
net.initFcn = 'initlay';
net.layers{1,1}.initFcn = 'initwb';
net.layers{2,1}.initFcn = 'initwb';
net.inputWeights{1,1}.initFcn = '';
net.layerWeights{2,1}.initFcn = '';
net.biases{1,1}.initFcn = '';
net.biases{2,1}.initFcn = '';
and it actually worked. In this way even calling the function "init" or "configure" does not affect in any way the weights and biases of the net. So I thought that if I manually set two nets in this way and with the same initial values, training them would result in exactly the same final weights/biases...but it doesn't... My question is why?
Does "trainlm" re-initialize the net in some way different than calling "init"? Has it some randomness other than initialization?

Accepted Answer

Greg Heath
Greg Heath on 3 Feb 2016
If you want to reproduce initial weights, reinitialize the rng to the same state before using configure.
If any non-zero weights are present, TRAIN will treat all weights as initial weights. Otherwise it will self-initialize using random weights.
Hope this helps.
Thank you for formally accepting my answer
Greg
  1 Comment
Roberto
Roberto on 3 Feb 2016
I'm sorry, I don't understand. If train doesn't initialize the net if a non-zero weight is present, then two networks initialized to the same state will have exactly the same training and should result in exactly the same final weights/biases right? But look:
>> net2 = net;
>> net.IW{1,1}
ans =
-4.199999999999999
-4.199999999999999
-4.199999999999998
>> net2.IW{1,1}
ans =
-4.199999999999999
-4.199999999999999
-4.199999999999998
>> net = train(net,Input,Output); >> net2 = train(net2,Input,Output); >> net.IW{1,1}
ans =
-5.175970179589142
-2.034413485697540
-2.720340992563584
>> net2.IW{1,1}
ans =
-6.053550714637582
0.375020219358226
-3.628899940303217
the 2 nets are identical and both have clearly non-zero weights before training...so how is possible that final weights are different? Maybe trainlm has some other "randomness" somewhere? I know Levenberg-Marquardt algorithm and I thought that it was purely deterministic...no randomness at all...am I wrong?

Sign in to comment.

More Answers (1)

Greg Heath
Greg Heath on 3 Feb 2016
After net is trained the state of the rng is changed. Therefore the weight initialization of net2 is different.
% help fitnet
close all, clear all, clc
[ x, t ] = simplefit_dataset;
net1 = fitnet; net2 = fitnet; net3 = fitnet;
isequal(net1,net2), isequal(net1,net3)% 1,1
WB0 = [getwb(net1) getwb(net2) getwb(net3) ] % zeros(10,3)
rng(0), net1 = train(net1,x,t); net2 = train(net2,x,t);
rng(0), net3 = train(net3,x,t);
WB = [getwb(net1) getwb(net2) getwb(net3)]
WB =
-16.568 -11.025 -16.568
-5.5931 -7.8365 -5.5931
3.6185 -3.8022 3.6185
-4.0666 4.3355 -4.0666
-1.85 1.6122 -1.85
0.07357 -0.094195 0.07357
-2.834 3.1095 -2.834
4.3672 -5.1322 4.3672
8.4144 7.4463 8.4144
12.07 13.83 12.07
12.118 12.197 12.118
6.6852 9.9585 6.6852
-6.9822 7.4495 -6.9822
10.482 -11.442 10.482
12.836 -11.914 12.836
-12.557 -14.228 -12.557
-7.3149 7.7502 -7.3149
7.6948 -8.879 7.6948
9.7115 8.1667 9.7115
12.196 12.392 12.196
-0.84428 -1.9359 -0.84428
-0.71237 0.25416 -0.71237
0.72471 0.36081 0.72471
0.81622 -0.766 0.81622
-0.18894 0.17421 -0.18894
0.090501 -0.095302 0.090501
-0.051973 -0.040525 -0.051973
0.11345 -0.13442 0.11345
-0.21845 0.19223 -0.21845
0.31931 0.53145 0.31931
0.41655 1.956 0.41655
  2 Comments
Nga Dao Thi
Nga Dao Thi on 4 Jan 2017
Thank you! I experienced the similar situation. I define a neural network (NN) with initial weights are 0s. However, when this NN trains the same dataset differently, i.e., the results are different when I run program many times. I also don't know why it happens. Later, I find that the training function trainscg will update weight and bias values according to the scaled conjugate gradient method. If I set biases are 0s, my NN has the same result. In contrast, If I set biases are not 0s, trainscg will update biases and weights. So that, the NN trains differently.

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!