How to find the best performance values for multistep ahead prediction?

2 views (last 30 days)
With narnet in a loop I am looking for the best hidden layer size for my network in terms of future predictions. I do a multistep ahead prediction with narnet and the predicted values are good (R squared > 0.8).
My problem is that the train, validation, test and closedloop performances of my network are not correlated with the R squared value, so if I do my trials or prediction for an unknown segment, then I can't decide which hidden layer size to choose.
How can I solve this problem?
Here is the correlation matrix (the rows and columns: performance, trainPerformance, valPerformance, closedLoopPerformance, testPerformance, MSE of predicted values, R squared of predicted values)
1.0000 0.9702 0.2953 0.1610 0.0780 -0.0401 0.0401
0.9702 1.0000 0.2507 0.1158 -0.1657 -0.0240 0.0240
0.2953 0.2507 1.0000 -0.0626 0.1469 -0.0918 0.0918
0.1610 0.1158 -0.0626 1.0000 0.1826 0.2678 -0.2678
0.0780 -0.1657 0.1469 0.1826 1.0000 -0.0622 0.0622
-0.0401 -0.0240 -0.0918 0.2678 -0.0622 1.0000 -1.0000
0.0401 0.0240 0.0918 -0.2678 0.0622 -1.0000 1.0000

Accepted Answer

Greg Heath
Greg Heath on 12 Sep 2015
To optimize the number of hidden nodes, I routinely design multiple candidates in a loop over numH hidden node values via h = Hmin:dH:Hmax. The value of Hmax <= Hub (upper bound) is determined with respect to the difference of the number of unknown weights (NID/NFD = "N"umber of "I"nput/"F"eedback "D"elays )
Nw = ( NID*I + NFD*O + 1)*H + (H+1)*O
and number of training equations for O-dimensional "O"utput targets
Ntrneq = Ntrn*O ~ 0.7*N*O
However, because initial weights are random I use an inner loop over Ntrials candidates that differ w.r.t. random initial weights and, most times, random data divisions. Typically, I design 100 candidates using Ntrials = 10. However, sometimes when dH > 1, I have had to add another search using a smaller dH.
Displaying Rsq values in a Ntrials-by-numH matrix yields an excellent overview of the performance dependence on H.
For examples in the NEWSGROUP and ANSWERS, try searching with Ntrials and/or Hub
Hope this helps.
Thank you for accepting my answer
Greg
  1 Comment
Greg Heath
Greg Heath on 12 Sep 2015
For time-series: Before the search for H, it I find it extremely useful to determine "significant" values of delays using the output autocorrelation function and input/output cross-correlation function.
Search the NEWSGROUP and ANSWERS using
greg nncorr
Hope this helps.
Greg

Sign in to comment.

More Answers (0)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!