how can i analyze my neural network ?

i have create a newpr network with two hidden layers, and i want to analysis the the netwrok, i see the plot confusion and see the result, and i see also the plot performance, but i do not understand how can i analyze the result is it good or not. the mean square error is going down and the test confusion matrix is 83.5 is it good or what?

another thing why there is a validation matrix? and what is the meaning of best validation performance is 0.10113 at epoch 26 is that good or what ?

need help if anyone can help me

thanks a lot

Answer 1

[Greg Heath]

See the comp.ai.neural-nets FAQ.

<ftp://ftp.sas.com/pub/neural/FAQ.html>

Consider

1. Randomly draw all of the data from the same probability distribution. Use all of the data to

a. Train many candidate models: Typically, by just using different numbers of hidden nodes, H, and many different weight initializations for each value of H; but sometimes also varying other design parameters.

b. Choose the best of all the models.

c. Use the performance of the best model to estimate performance on unseen data.

Using the same data to perform all three tasks often leads to naive, optimistically biased, estimates of performance on unseen data.

The final result of this can be disasterous performance on unseen data, loss of clients, reputation and employment, bankruptcy of the company and failure of the economy.

A simple way to avoid the optimistic estimates and associated repercussions is to use a sufficiently large data set. However, in most real world situations, that much data is either not available or too unwieldly.

The most common approach is to perform the three tasks using three separate subsets of the data.

Now, consider the following

1. Randomly split the data into a design subset (for tasks a and b) and a test subset (for task c).

total = design + test

2. Put the test subset set aside to be used ONCE and ONLY ONCE for task c.

3. Randomly split the design set into training (task a) and validation (task b) subsets.

design = training + validation

4. Perform tasks a, b, and c.

5. If the estimated performance is unsatisfactory, either accept what you have or return to 1.

6. Steps 1 to 5 can be repeated 32 or more times so that histograms and confidence levels on the performance estimate can be obtained.

Very often the training set is too small to be representative of unseen data and/or too small to support the accurate estimation of a large number of weights. Consequently, as training proceeds, the tendency for MSEtrn to decrease is not representative of performance on unseen data.

There are several ways to avoid this. They are discussed in the CANN FAQ. The method used here is called Early Stopping (aka "Stopped Training"): I like my own term "Validation Stopping". Training stops when the nontraining validation set error reaches a minimum. MATLAB's default definition for Early Stopping is when the validation set error stops decreasing for 6 consecutive epochs. The default value of 6 can be overwritten if you wish.

Finally, an unbiased estimate of performance on unseen data is obtained from the test set.

The design set should be used repeatedly (randomly splitting into training + validation), to find the best combination of hidden nodes, initial weights, number of training epochs and other training parameters.

The test set is used to obtain an unbiased estimate of performance on unseen data.

Many repetitions can be used to obtain confidence intervals.

Hope this helps.

Thank you for officially accepting my answer.

Greg

P.S. There are many valid variations of the above design procedure

Answer 2

[Greg Heath]

>i have create a newpr network with two hidden layers, and i want to analysis >the the netwrok, i see the plot confusion and see the result, and i see also >the plot performance, but i do not understand how can i analyze the result is >it good or not. the mean square error is going down and the test confusion >matrix is 83.5 is it good or what?

Depends on the nature of the problem (How many people die for each error?), the number of classes and the complexity of the target distribution.

>another thing why there is a validation matrix?

Default training stops 6 epochs after the validation error reaches a minimum. However you can overwrite the default.

>and what is the meaning of best validation performance is 0.10113 at epoch 26 >is that good or what ?

See above re "good". I like MSEval/MSE00 < 0.01 (MSE00 = var(t,1,2))

For c classes(categories) target matrix t should columns of eye(c) where the row index of the 1 indicates the correct class.

[ I N ] = size(x)

[ O N ] = size(t) % O = c

Neq = N*O % Number of scalar training equations.

MSE00 = var(t,1,2) % MSE for constant output

2. Sufficient to use one hidden layer. For H hidden nodes, there are

Nw =(I+1+*H+(H+1)*O = O+(I+O+1)*H % Number of unknown weights to estimate.

3. Require Neq > Nw but desire Neq >> Nw. Therefore try

H << (Neq-O)/(I+O+1)

4. MSEgoal = MSE00/200

Hope this helps.

Thanks for formally accepting my answer.

Greg

[ahmad]

thanks alot for your help but i still dont understand i have three classes and from the confusion matrix i can understand the number of correct classification and false classification but from the performance the MSE is going down and the Best validation performance is 0.10113 at epoch 26 but i dont know what is that mean? can u illustrate the meaning of best validation performance number and is it good or bad? another thing what is the purpose of validation set in neural network ? and what is the validation checks = 6 at epoch 32?

iam a master student and my doctor asking to put the result of experiment but i dont know what to put ?!

sorry about my lot of questions but i really need help.

thanks a lot Greg

Answer 3

[Nida]

ahmad i was going through the same prb a few days back. MSE is basically needed to be as much minimum as it can be . In ur case, the value of 1.011e-01 is somewhat acceptable if ur output is correctly classified. The value of 83.5 for test confusion matrix is if the overall accuracy, it can be improved. A detailed answer provided by greg hearth on my problem can be found at

http://www.mathworks.es/matlabcentral/answers/79653-performance-estimate-of-pattern-recognition-tool

As you are using neural networks, try to increase ur dataset and also check the performance parameters for neural networks that are precision, recall etc. and from that calculate F-Score that tells u abt the performance of ur model. It should be towards 1 in the range of 0-1

[Nida]

In my case, using a larger dataset and altering a few parameters, i reached an overall accuracy of 97.5%

[afef]

Hi, Nida i'm stuck with improving the accuracy of my neural network tha i created using nprtool i tried to use larger dataset i mean for the input which is a matrix with size 9*1012 or changing the number of neurons in the hidden layer or changing the training function but still the accuracy is 66% so ca you please help me i need this for my master project .Thanks

[Walter Roberson]

It is unlikely that Nida is still reading here, 4 years later. Nida will not be notified about your comment. You could try clicking on Nida's name above to reach their profile and then use the Contact button to send Nida email.