Neural Networks extrapolation (using closed network - multistep prediction) can not even predict a line ?

3 views (last 30 days)
Hi; Is it possible to predict EFFECTIVELY into the future using NAR neural network. I used Neural Networks to predict a simple line using multisteps (closed loop ), but it turns out that it is very good but only till the training part whereas it miserably fails in the multistep (predict ahead ) part. here is my code: Please help, where am I wrong ? %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% MULTISTEP PREDICTION USING NAR AND CLOSED LOOP NEURAL NETWORK
clc; clear all; close all;
% A simple line which I want to predict
DATA= (1:1000)';
%% Create a Nonlinear Autoregressive Network
feedbackDelays = 1:5;
hiddenLayerSize = 5;
net = narnet(feedbackDelays,hiddenLayerSize);
net.divideParam.trainRatio = 85/100;
net.divideParam.valRatio = 15/100;
net.divideParam.testRatio = 0/100; % NO TEST DATA as I only want to test the multistep part in future
net.divideFcn = 'dividerand'; % Divide data randomly
% TRAINING
N=500; % Number of bars used for training
TDATA = DATA(1:N); % Training Data
T = tonndata(TDATA,false,false);
[x,xi,ai,t] = preparets(net,{},{},T); % prepare data
[net,tr] = train(net,x,t,xi,ai); % Train the Network
y = net(x,xi,ai);
% MULTISPTEP PREDICTION - Closed Loop Network
MSDATA = DATA(N-feedbackDelays(end):end); % multistep predict ahead data
T = tonndata(MSDATA,false,false);
netc = closeloop(net);
[xc,xic,aic,tc] = preparets(netc,{},{},T);
yc = netc(xc,xic,aic);
%% PLot training and multistep predicted part
hold on;
plot(cell2mat(t)); % targets in training part
plot(cell2mat(y),'r'); % nn results in training part
plot([nan(1,length(t)'),cell2mat(tc)],'g'); % targets in multistep predicted part
plot([nan(1,length(y)'),cell2mat(yc)],'r'); % nn in multistep predicted part

Accepted Answer

Greg Heath
Greg Heath on 22 Sep 2015
Edited: Greg Heath on 22 Sep 2015
Since the target is a straight line, no hidden layer is used. (In general, I would find the minimum number of hidden nodes via a double-for-loop multiple design search).
Testing the CL net on the OL design data for 500 timesteps results in an increase of normalized- mean-square-error from 8e-32 to 5e24 !!!. However, the monotonically increasing CL NMSE only exceeds 0.01 after time = 275.
Similarly, testing the CL net on the nondesign prediction data (501 to 1000 ) yields nmse > 0.01 only after time = 757.
clc; clear all; close all; plt = 0;
T = con2seq(1:1000); d=5
FD = 1:d; H = []; % H changed to prevent overfitting
neto = narnet(FD,H);
neto.divideParam.trainRatio = 85/100;
neto.divideParam.valRatio = 15/100;
neto.divideParam.testRatio = 0/100; % NOTE
% neto.divideFcn = 'dividerand'; Default
Nd = 500 % Design (Train + Val)
Td = T(1:Nd); % Design data
Tp = T(Nd+1:end); % Prediction data
Np = length(Tp) % 500
% OPEN LOOP PERFORMANCE
[Xo,Xoi,Aoi,To] = preparets(neto,{},{},Td);
to= cell2mat(To); varto = var(to,1) % 20419
rng('default')
[neto,tro Yo Eo Xof Aof] = train(neto,Xo,To,Xoi,Aoi);
yo = cell2mat(Yo);
NMSEo = mse(Eo)/varto % 8.3362e-32
plt=plt+1, figure(plt)
subplot(311), hold on
plot(to,'LineWidth',2)
plot(yo,'r--','LineWidth',1)
title('OPEN LOOP PERFORMANCE')
% CLOSED LOOP PERFORMANCE
[ netc Xci Aci ] = closeloop(neto,Xoi,Aoi);
[ Xc Xci Aci Tc ] = preparets(netc,{},{},Td);
tc = to; % isequal(Tc,To) = 1
[Yc Xcf Acf ] = netc(Xc,Xci,Aci);
ec = cell2mat(gsubtract(Tc,Yc));
NMSEc = mse(ec)/varto % 5.3035e+24 WOW!
% Is abs(ec) monotonic?
u = find(diff(abs(ec) <= 0)) % Empty matrix: 1-by-0
necsq = ec.^2/varto;
cumNMSEc = cumsum(necsq)./(1:Nd-d);
v = find(cumNMSEc <= 0.01);
Nv = length(v) % 275
NMSEcv = cumNMSEc(Nv) % 0.0094645
yc = cell2mat(Yc);
subplot(312), hold on
plot(tc(v),'LineWidth',2)
plot(yc(v),'r--','LineWidth',1)
title('CLOSED LOOP PERFORMANCE')
subplot(313),
plot(ec(v),'LineWidth',2)
title('CLOSED LOOP ERROR')
% MULTISTEP PREDICTION
Tm = T(Nd-5:Nd+Nv-1);
[ Xcm Xcmi Acmi Tcm ] = preparets(netc,{},{},Tm);
tcm = cell2mat(Tcm); vartcm = var(tcm,1) % 6302
[Ycm Xcmf Acmf ] = netc(Xcm,Xcmi,Acmi);
ecm = cell2mat(gsubtract(Tcm,Ycm));
NMSEcm = mse(ecm)/vartcm % 1.3771
% The error amplitude is not monotonic
u = find(diff(abs(ecm) <= 0)) %[1,12:19]
necmsq = ecm.^2/vartcm;
cumNMSEcm = cumsum(necmsq)./(1:Nv);
vm = find(cumNMSEcm <= 0.01);
Nvm = length(vm) % 257
NMSEcmv = cumNMSEcm(Nvm) % 0.0090988
plt = plt+1, figure(plt)
subplot(211), hold on
plot(tmc(z),'LineWidth',2)
plot(ymc(z),'r--','LineWidth',1)
title('TARGET(Blue) & OUTPUT(Red)')
subplot(212),
plot(emc(z),'LineWidth',2)
Hope this helps.
Thank you for formally accepting my answer
Greg

More Answers (2)

Greg Heath
Greg Heath on 17 Oct 2014
Edited: Greg Heath on 17 Oct 2014
1. Do you really expect to confidently predict ahead when you use 'dividerand' ???
2. Both training and validation data are used for design. Therefore if you do not have test data you do not have an UNBIASED estimate of future performance.
3. Conclusion
a. Use a divideFcn which preserves order and constant spacing.
b. Obtain an unbiased estimate of performance on new data using a test set.
Hope this helps.
Thank you for formally accepting my answer
Greg
P.S. IT IS VERY UNFORTUNATE THAT MATLAB USES DIVIDERAND TO BE THE DEFAULT DIVISION FUNCTION IN TIMESERIES FUNCTIONS
  1 Comment
Muhammad Hasnat
Muhammad Hasnat on 17 Oct 2014
Thankyou for the response.
I should have mentioned earlier that I have already tried different basic variants, e.g divideblock, different number of layers as well as different number of delays. but the line is never matched effectively in multistep prediction ahead. After repeated training one may get a curve at best. Also, shouldn't 70% be enough to find constant spacing of a line, even if 30% is taken out randomly. I say that also because, results were better with random division of data when predicting a line.
Regards Hasnat

Sign in to comment.


Greg Heath
Greg Heath on 22 Sep 2015
1. You do not need a hidden layer for a straight line. Therefore, you are overfitting. It is good practice to try to minimize the number of hidden layers and nodes. Otherwise use validation stopping and/or regularization.
2. Test the CL configuration on the OL design data. If you compute the cumulative MSE, you can get an estimate of an upper bound for how many timesteps you can expect to accurately predict.
Hope this helps.
Greg

Categories

Find more on Sequence and Numeric Feature Data Workflows in Help Center and File Exchange

Products

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!