How can I train a neural network manually choosing a specific set of initial weights and biases?
6 views (last 30 days)
Show older comments
My goal is to reproduce many times exactly the same training in order to do some tests about some training parameters variations. So what i tried first was to define a neural network, configure it, initialize it. Then I "switched off" the init function in order to keep fixed the initial values of weights and biases and use them again later in another training. I succeeded in this by setting
net.initFcn = 'initlay';
net.layers{1,1}.initFcn = 'initwb';
net.layers{2,1}.initFcn = 'initwb';
net.inputWeights{1,1}.initFcn = '';
net.layerWeights{2,1}.initFcn = '';
net.biases{1,1}.initFcn = '';
net.biases{2,1}.initFcn = '';
and it actually worked. In this way even calling the function "init" or "configure" does not affect in any way the weights and biases of the net. So I thought that if I manually set two nets in this way and with the same initial values, training them would result in exactly the same final weights/biases...but it doesn't... My question is why?
Does "trainlm" re-initialize the net in some way different than calling "init"? Has it some randomness other than initialization?
0 Comments
Accepted Answer
Greg Heath
on 3 Feb 2016
If you want to reproduce initial weights, reinitialize the rng to the same state before using configure.
If any non-zero weights are present, TRAIN will treat all weights as initial weights. Otherwise it will self-initialize using random weights.
Hope this helps.
Thank you for formally accepting my answer
Greg
More Answers (1)
Greg Heath
on 3 Feb 2016
After net is trained the state of the rng is changed. Therefore the weight initialization of net2 is different.
% help fitnet
close all, clear all, clc
[ x, t ] = simplefit_dataset;
net1 = fitnet; net2 = fitnet; net3 = fitnet;
isequal(net1,net2), isequal(net1,net3)% 1,1
WB0 = [getwb(net1) getwb(net2) getwb(net3) ] % zeros(10,3)
rng(0), net1 = train(net1,x,t); net2 = train(net2,x,t);
rng(0), net3 = train(net3,x,t);
WB = [getwb(net1) getwb(net2) getwb(net3)]
WB =
-16.568 -11.025 -16.568
-5.5931 -7.8365 -5.5931
3.6185 -3.8022 3.6185
-4.0666 4.3355 -4.0666
-1.85 1.6122 -1.85
0.07357 -0.094195 0.07357
-2.834 3.1095 -2.834
4.3672 -5.1322 4.3672
8.4144 7.4463 8.4144
12.07 13.83 12.07
12.118 12.197 12.118
6.6852 9.9585 6.6852
-6.9822 7.4495 -6.9822
10.482 -11.442 10.482
12.836 -11.914 12.836
-12.557 -14.228 -12.557
-7.3149 7.7502 -7.3149
7.6948 -8.879 7.6948
9.7115 8.1667 9.7115
12.196 12.392 12.196
-0.84428 -1.9359 -0.84428
-0.71237 0.25416 -0.71237
0.72471 0.36081 0.72471
0.81622 -0.766 0.81622
-0.18894 0.17421 -0.18894
0.090501 -0.095302 0.090501
-0.051973 -0.040525 -0.051973
0.11345 -0.13442 0.11345
-0.21845 0.19223 -0.21845
0.31931 0.53145 0.31931
0.41655 1.956 0.41655
2 Comments
Nga Dao Thi
on 4 Jan 2017
Thank you! I experienced the similar situation. I define a neural network (NN) with initial weights are 0s. However, when this NN trains the same dataset differently, i.e., the results are different when I run program many times. I also don't know why it happens. Later, I find that the training function trainscg will update weight and bias values according to the scaled conjugate gradient method. If I set biases are 0s, my NN has the same result. In contrast, If I set biases are not 0s, trainscg will update biases and weights. So that, the NN trains differently.
See Also
Categories
Find more on Function Approximation and Nonlinear Regression in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!