Why neural network sometimes performs worse when additional input variables are used?

Why neural network sometimes performs worse when additional input variables are used?

I would expect a neural network trained with n+1 input variables not to perform worse than a neural network (exactly same structured) trained with n input variables (the n inputs are the same). If the added input variable has no value then during the training step shouldn't all the weights associated to it be 0 and the overall performance not be worse. Is my expectation wrong? If not, why I am getting worse performance sometimes (even with seeded random generator)?

Answer 1

[Greg Heath]

1. You don't say what n is

2. How is the net suppose to know that n inputs is sufficient?

3. All it does is start somewhere amongst a bunch of hills. Now since it is blind and can only feel within it's reach to determine where down is, it's lucky to get anywhere.

4. If you were kidnapped and dumped in a 4-dimensional mountain range do you think you would do better if you were in 3-D??

Hope this helps.

Greg tries to roll down in the steepest direction

[Kamuran Turksoy]

See below please

1. You don't say what n is

I dont think the value of n is important. To more specific lets say n is 3. So here is the case: x1, x2, x3, x4 are the inputs and y1 is the output. Lets define two same structured neural networks: net1 and net2. The same initial values are used for both nets

net1 is trained with [x1 x2 x3]-[y1] and net2 is trained with [x1 x2 x3 x4]-[y1]. The question is, why net2 performs worse than net1 as it already has all the information used in net1. I would expect the worst performance of net2 to be equal to the performance of net1 in which case all weights/biases associated to x4 to be zero and the two nets to be identical. In my opinion, net2 should perform better or equal to net1 but not worse.

2. How is the net suppose to know that n inputs is sufficient?

I am not saying n inputs is sufficient. I am just comparing n inputs vs n+1 inputs. if n+1 inputs performs better (or equal) than n, I am fine but the problem is why sometimes its worse ?

3. All it does is start somewhere amongst a bunch of hills. Now since it is blind and can only feel within it's reach to determine where down is, it's lucky to get anywhere.

If this was the case, then increasing validation check or number of epochs should let net2 to converge to the performance of net1. But this did not work for me.

4. If you were kidnapped and dumped in a 4-dimensional mountain range do you think you would do better if you were in 3-D??

If the 4th dimension has no value, then my worst performance would be identical to the 3 dimension case but not worse. Its understandable to have better or equal performence with 4-D but not worse.

[Greg Heath]

There is a dreaded NN phenomenom called

OVERTRAINING AN OVERFIT NET.

An overfit net is a net that has more unknown parameters than equations used to solve those parameters. The result is that, without other constraints to satisfy, there are extra degrees of freedom that can do what ever they want. As a result, the answer between input data points is somewhat arbitrary.

Simple case: Solve

    x1 + 2*x2 + 3*x3 = 6
  3*x1 + 2*x2 +   x3 = 6

Whatever answer you give, it is not the one I wanted... UNLESS you chose the one with x2 = 0.

Hope I got the idea across

Greg