How do I improve the performance of a closed loop NARNET network?

I want to predict the next 10 steps for a time series. The single time series (T), has multiple values per time period.

The design criteria, are 1. Forecast accuracy, (next prediction point is more accurate than the 10th prediction point) 2. NN Execution speed. (Training speed is a secondary).

I used the 'Workflow for Neural Network Design' as a template, and the 'Narnet Tutorial On Multi Step Predictions' as a working model.

The code is below. (I specify some default parameters, so that I know what to set them back to when testing different network configurations).

The only relevant (i.e. multi value) dataset I could find for testing is 'oil_dataset'.

The code & results are below:

%%1. Collect & Prepare the data
clear all; clc; plt = 0;
PredictSteps = 10;

T = oil_dataset;
N = length(T);

%%2. Create the Neural Network
FD = 2;                 % Optimize this later with nncorr
H = 10;                 % Optimize this later with trial/error loops

feedbackDelays = 1:FD;  % Default = 1:2
hiddenLayerSize = H;    % Default = 10
trainFcn = 'trainbr';   % Default = 'trainlm'
net = narnet(feedbackDelays,hiddenLayerSize,'open',trainFcn);

%%3. Configure the network
% Choose Feedback Pre/Post-Processing Functions
net.input.processFcns = {'removeconstantrows','mapminmax'};

% Setup dynamic weights & biases
net.adaptFcn = 'adaptwb';       % Default = 'adaptwb'

% Setup Division of Data for Training, Validation, Testing
net.divideFcn = 'divideblock';  % Divide data into blocks
net.divideMode = 'time';        % Divide up every value
net.divideParam.trainRatio = 70/100;
net.divideParam.valRatio   = 15/100;
net.divideParam.testRatio  = 15/100;
net.trainParam.epochs = 1000;    % Default = 1,000

% Choose a Performance Function
net.performFcn = 'mse';         % Mean squared error

% For MSE, Normalize errors of multiple outputs in different ranges
net.performParam.normalization = 'standard';

% Choose Plot Functions
net.plotFcns = {'plotperform','plottrainstate','plotresponse', ...
  'ploterrcorr','plotinerrcorr','plotwb','ploterrhist','plotfit', ...
  'plotregression',};

%%4. Initialize weights & biases
rng('default');

% In this case set 'weighted Error Weights'
ind = 1:N;                   
ew = 0.99.^(N-ind);  

% Plot the Error Weights
plt = plt + 1;
figure(plt), hold on
plot(ew)
title('Error Weights');
ew = con2seq(ew);

%%5.a) Train the network
[Xo,Xio,Aio,To,EWo] = preparets(net,{},{},T, ew);

% Plot original data
to = cell2mat(To);
plt = plt + 1;
figure(plt), hold on
plot(FD+1:N, to)
legend('Upper', 'Lower')
title('INITIAL TRAINING DATA');

% Train the NN
[net,tr,Yo,Eo,Afo,Xfo] = train(net,Xo,To,Xio,Aio,EWo);

%%5.b) Plot the OPENLOOP traing results
yo = cell2mat(Yo);
plt = plt + 1;
figure(plt), hold on
plot(FD+1:N, to, 'b') 
plot(FD+1:N, yo, 'r--')
legend('TARGET',  'OUTPUT') % Fix the legend colors, need to make legend for each line
title('OPENLOOP NARNET RESULTS');

%%5.c) Validate the trained 'Open' Network
[Yo, Xfo, Afo] = net(Xo,Xio,Aio);
Eo = gsubtract(To,Yo);
OpenLoopMSEperformance = mse(net, To, Yo);              % 728.0917
OpenLoopPERFORMperformance = perform(net,To,Yo, EWo);   % 0.0063
% NMSEopen = mse((Eo)/var(To,1)); % Error: Undefined function 'sum' for input arguments of type 'cell'
view(net);

%%5.d) Close the Network
[netc, Xic,Aic] = closeloop(net,Xio,Aio);
netc.name = [net.name ' - Closed Loop'];
view(netc);

[Xc,Xic,Aic,Tc,EWc] = preparets(netc,{},{},T,ew); % should we use EWo instead of ew?
[Yc,Xfc,Afc] = netc(Xc,Xic,Aic);

%%5.e) Plot the CLOSELOOP results
tc = cell2mat(Tc);          % tc isequal to
yc = cell2mat(Yc);
plt = plt + 1;
figure(plt), hold on
plot(FD+1:N, tc, 'b') 
plot(FD+1:N, yc, 'r--')
legend('TARGET',  'OUTPUT') % Fix the legend colors
title('CLOSELOOP NARNET RESULTS')

%%5.f) Validate the closed network
Ec = gsubtract(Tc,Yc);
CloseLoopMSEperformance = mse(netc, Tc, Yc);            % 9.8945e+04
CloseLoopPERFORMperformance = perform(netc,Tc,Yc,EWc);  % 0.5823
% NMSEclosed = mse((Ec)/var(Tc,1)); % Error: Undefined function 'sum' for input arguments of type 'cell'
view(netc);

%%5.g) ReTrain the 'Closed' network for better accuracy
% 'preparets' performed in section 5.d)
[netc2, trc, Yc2, Ec2, Xfc2, Afc2] = train(netc, Xc, Tc, Xic, Aic, EWc);
[Yc3,Xfc3, Afc3] = netc2(Xc,Xfc2,Afc2);

%%5.h) Plot the retrained CLOSELOOP results
yc3 = cell2mat(Yc3);
% tc = to                   % tc isequal to
plt = plt + 1;
figure(plt), hold on
plot(FD+1:N, tc, 'b') 
plot(FD+1:N, yc3, 'g--')
legend('TARGET',  'OUTPUT') % Fix the legend colors
title('TRAINED CLOSELOOP NARNET RESULTS');

%%6. Validate the trained 'Closed' Network
Ec2 = gsubtract(Tc,Yc3);
TrainedClosedLoopMSEperformance = mse(netc2, Tc, Yc3);              % 6.3859e+04
TrainedClosedLoopPERFORMperformance = perform(netc2,Tc,Yc3,EWc);    % 0.2147
% NMSEs = mse((Ec2)/var(Tc,1)); % Error: Undefined function 'sum' for input arguments of type 'cell'
view(netc2);

%%7.a) Use the network (for multi-step prediction)
Xc3 = cell(1, PredictSteps); % Empty cells for prediction
[Yc4,Xfc4,Afc4] = netc2(Xc3,Xfc3,Afc3);

%%7.b) Plot the predicted results
yc4 = cell2mat(Yc4);
plt = plt+1; figure(plt), hold on
plot(FD+1:N, tc)
plot(FD+1:N, yc3)
plot(N+1:N+PredictSteps, yc4, '+')
legend('TARGET', 'OUTPUT' , 'TARGETLESS PREDICTION')
title('NARNET PREDICTION RESULTS');

%%Save the NN and the final input and layer delay states, Xfc and Afc
Xis = Xfc4;
Ais = Afc4;
save('NARNETfcn.mat', 'Xis', 'Ais');  % Ready for use next time

As you can see, once the narnet is closed, the performance is bizarre.

I ran the 'Narnet Tutorial On Multistep Ahead Predictions' on oil_dataset and produced similar results.

1. What am I doing wrong?

2. How do I correct the error calculating NMSEs?

Thank you in anticipation.

[Greg Heath]

 1. Why did you use TRAINBR instead of the default TRAINLM?

2. Cannot have valratio = 0.15 with TRAINBR. Must use valratio = 0!

3. Why mse instead of msereg??? Combining TRAINBR with MSE is unusual.

4. Why need EW? Have you compare w and w/o?

5. Xfo, Afo erroneously in reverse order in training equation

6. What is the performance function? perform(neto,To,Yo, EWo). Similarly for netc

7. NMSEopen = mse(Eo)/mean(var(to',1)) . Similarly for NMSEclosed

Hope this helps.

Greg

[Gary]

Greg,

Thank you for your prompt response, yes it helps enormously. If I may, I would like to answer your questions in a slightly different order, for reasons which I hope will become clear:

1. I used TRAINBR to improve NN generalization and avoid overfitting. (As per the documentation, but I haven't mastered the 'early stopping' yet though!)

3. Originally I considered 'msereg' but 'help msereg' stated "Obsoleted in R2010b NNET7.0. Last used in 2010a NNET6.0.4.". I have since updated "net.performFcn = msereg".

2. I removed all the NET.performParam's. (As per 'help msereg' - Set NET.performFcn to 'msereg'. This will automatically set NET.performParam to msereg's default performance parameters.)

4. I used Error Weights for prediction accuracy, i.e. because I wanted the 'predictions' to reflect the more recent history, while remaining generalized without overfitting. At this point I have not tested without them, because I still have to optimize the feedbackDelays & hiddenLayerSize, so this was part of my 'tuning' plan.

5. Thank you for identifying that error.

7. Thank you for that clarification

6. Here are the results:

MSEo = 728.0917 MSEREGo = 655.5660 PERFORMo = 0.0063 NMSEo = 0.0231

MSEc = 9.9845e+04 MSEREGc = 8.9051e+04 PERFORMc = 0.5823 NMSEc = 3.1392

MSEr = 6.3859e+04 MSEREGr = 5.4462e+03 PERFORMr = 0.2147 NMSEr = 2.0260 (r = retrained)

The current performance levels are just not usable. What do you recommend as the way forward? Should I work on the feedbackDelay and hiddenLayerSize parameters to improve the NN performance, or am I missing something else?

Thanks and regards Gary

Answer 1

[Greg Heath]

Greg,

1. I used TRAINBR to improve NN generalization and avoid overfitting. (As per the documentation, but I haven't mastered the 'early stopping' yet though!)

 GEH1 = It is better to train with defaults first. 
Early Stopping is a default. Training stops when 
the error on the validation set increases for 
6(default) consecutive epochs.

Simple as that. Dealing with TRAINBR is somewhat complicated because of various idiosyncracies of the MATLAB code.

3. Originally I considered 'msereg' but 'help msereg' stated "Obsoleted in R2010b NNET7.0. Last used in 2010a NNET6.0.4.". I have since updated "net.performFcn = msereg".

 GEH2 = msereg is a TRAINBR default. Therefore you 
needed to do nothing.

2. I removed all the NET.performParam's. (As per 'help msereg' - Set NET.performFcn to 'msereg'. This will automatically set NET.performParam to msereg's default performance parameters.)

 GEH3 = Not clear. However probably
 irrelevant because of GEH2.

4. I used Error Weights for prediction accuracy, i.e. because I wanted the 'predictions' to reflect the more recent history, while remaining generalized without overfitting. At this point I have not tested without them, because I still have to optimize the feedbackDelays & hiddenLayerSize, so this was part of my 'tuning' plan.

 GEH4: It is best to begin with all defaults. 
Then make "improvements" one at a time.

5. Thank you for identifying that error.

7. Thank you for that clarification

6. Here are the results:

MSEo = 728.0917 MSEREGo = 655.5660 PERFORMo = 0.0063 NMSEo = 0.0231

MSEc = 9.9845e+04 MSEREGc = 8.9051e+04 PERFORMc = 0.5823 NMSEc = 3.1392

MSEr = 6.3859e+04 MSEREGr = 5.4462e+03 PERFORMr = 0.2147 NMSEr = 2.0260 (r = retrained)

The current performance levels are just not usable. What do you recommend as the way forward? Should I work on the feedbackDelay and hiddenLayerSize parameters to improve the NN performance, or am I missing something else?

Thanks and regards Gary

I can only recommend what I do.

 1. use all defaults; use an outer for loop to vary

H and an inner loop to vary random 
initial weights

and random data-divisions

 2. use DIVIDEBLOCK; repeat double loop search
 3. Obtain significant lags
 4. Choose a sensible subset of siglags then try
    to minimize H subject to the constraint

   NMSEo <= 0.005 (or 0.001)

using the doule loop search

5. It is often difficult to obtain a low NMSEc with closeloop if NMSEo is any higher.

6. If none of the CL nets is successful, try training them further otherwise go back and lower the NMSEo goal.

Hope this helps.

Thank you for formally accepting my answer

Greg

[Greg Heath]

I have designed several nets using the oil dataset.

No problem getting NMSEo = 0.023 using DIVIDEBLOCK with the default FD=1:2, H = 10 and a single random initial weight setting.

However, before closing the loop, or investigating other settings, I decided to estimate a lower bound by using DIVIDETRAIN. The result was NMSEo = 0.002.

However, closing the loop on the latter net resulted in a disastrous NMSEc = 11.2 !!! An attempt to train the CL net was also a disaster: Training died after ONE EPOCH! via a MAXIMUM MU error message.

Obviously, varying FD, H and random initial weights should yield much better solutions. However, I was so surprised by my disastrous CL results , in addition to yours, that I felt it was important enough to post this comment.

More later.

Greg

[Gary]

Greg,

I have re-run the process with 1:2<FD<1:50 and 0<H<50, using 'trainlm' & 'trainbr', and with & without weights.

I even went so far as to split the data up into 2 series and created a NN for each series. Every time the results demonstrated the same pattern, i.e. disastrous CL results. Further CL training did not improve the results.

I am using MATLAB 2015a with a number of toolboxes, including Neural Network toolbox and Statistics & Machine Learning toolbox.

I look forward very much to the results of your investigation.

Regards Gary

[Greg Heath]

After calculating the two autocorrelation functions and cross- correlation function I have found that the feedback delays FD = 1:35 are all significant and sufficient for a good CL solution. Furthermore, good results are obtained with H <= 5.

Will probably post the solution in the NEWSGROUP when it is thoroughly checked out.

I'll let you know wnen it is posted.

Greg

[Greg Heath]

Sorry for the delay. Home repairs and vacation with family took precedence.

[Gary]

Greg,

Completely understood!!

Would very much appreciate notification when posted to the newsgroup.

Thank you very much for your attention.

Kind regards Gary

[Gary]

Greg,

Did you ever post the autocorrelation/cross correlation solution mentioned above to the NEWSGROUP?

It's been a while since I have been on MATLAB Central, and things seem to have changed somewhat!

I'm looking for the calculations for the autocorrelation and cross-correlation functions that suggested the feedback delays FD = 1:35

When I tried using:

ACFt = nncorr(zt, zt, N-1, 'biased');
[pks locs] = findpeaks(ACFt(N+1:end), 'MinPeakHeight', 0);

The resulting autocorrelation peak came back at location "158" only.

Your assistance would be very much appreciated.

Thanks Gary

[Greg Heath]

I have not solved the OL --> CL problem for this and several other MATLAB datasets obtained from

 help nndatasets

and

 doc nndatasets

30AUG2017:

Sorry. I have been distracted because of more non-serious but time-consuming family stuff.

Will have to go back to square 1 to refresh the ageing memory.

Sorry for the delay.

Greg

[Greg Heath]

You forgot the negative peak near lag = 98!

Greg

[Greg Heath]

Well, I couldn't find my previous stuff .. so I started from scratch with the default 1:2,10 inputs. No change in results.

 1.  Good OL
 2. Bad CL from closing the loop
 3. No improvement from training CL.

 Stopped at one epoch via maximum mu.

 Will try to find my significant lag stuff 
instead of doing it over.

Greg

[Gary]

Greg,

Many thanks for the update. I'm pretty comfortable with the NN training, (thanks to your posted tutorials, comments etc.) what I'm looking for guidance on, is the methodology for using autocorrelation and cross-correlation to support the FD = 1:35

While there is a -ve correlation peak at lag 98, I still can't reconcile that with your earlier calculations for FD = 1:35

I was looking for +ve correlations because I understood a -ve correlation to be an "inverse" relationship between the two data series, i.e. the opposite to what I'm looking for.

Would you suggest I look at "smoothing" the data to reduce the "noise", before looking for correlation?

Thank you in advance

Gary

[Greg Heath]

Look for significant ABSOLUTE values.

1. Negative correlations are just as effective as positive correlations. The only difference is that the sign of some of the NN weights will be reversed.

2. Significant absolute values are all absolute values above a specified threshold. Not just the peak values.

3. I consider signal correlation coefficients significant if their absolute values are not exceeded by the absolute values of noise/noise correlation coefficients 95% of the time.

Hope this helps.

Greg