Check Gradient (Jacobian) of objective function -- what is the meaning of absolute difference (1e-6) if component of gradient is less than 1 ?

I use lsqnonlin to solve my data-fitting problem and provide the Jacobian, which I verify using CheckGradients option.

As stated here, if a component of the Jacobian is less than 1, gradient check is successful if the absolute difference between the user-shipped Jacobian and Matlabs finite-difference approximation of that component is less than 1e-6.

Example: Jacobian_USER(1,1)=2e-7, Jacobian_MATLAB(1,1)=1e-7.

Here, Jacobian_USER(1,1) is clearly wrong, but Jacobian check would be successful. Is that true?

[Torsten]

What is Jacobian_USER and Jacobian_MATLAB ?

[SA-W]

Jacobian_USER is the Jacobian computed by the user and passed to lsqnonlin by setting ' SpecifyObjectiveGradient' to true.

Jacobian_MATLAB is the finite-difference approximation to the Jacobian calculated by Matlab.

[Torsten]

So you only have one equation in one unknown since the Jacobian only has one element ?

[SA-W]

No, my Jacobian is 10x2.

I picked J(1,1) arbitrarily in my question.

[Torsten]

I don't think Jacobian errors are computed elementwise.

But let's assume this is the case. Then the difference between the Jacobian computed by the user and the Jacobian computed by MATLAB is 1e-7, so smaller than the error tolerance and the user Jacobian will thus be accepted.

Why do you think the user Jacobian is clearly wrong ?

[SA-W]

In the link from my question is stated that Check Gradients looks at individual components of the Jacobian.

Let's set Jacobian_USER(1,1)=1e-9. This would also be accepted although it is wrong by two orders of magnitude, given that 1e-7 is correct.

[Torsten]

Yes.

Would you feel better if the errors were estimated as

(2e-7-1e-7)/1e-7 = 1

or

(1e-7-1e-9)/1e-7 = 0.99

?

I don't.

[SA-W]

I would actually feel better with this kind of estimation, because then I knew that there might be something wrong with my Jacobian as Check Gradients fails. :)

[Torsten]

You shouldn't assume something is wrong if the result of one computation gives 2e-7, the result of another computation gives 1e-7.

You should be alarmed if the result of one of the computations moves significantly towards a larger value, e.g. 1e-5 or 1e-4. But these are cases where the error check would inform you about the difference.

[SA-W]

"You shouldn't assume something is wrong if the result of one computation gives 2e-7, the result of another computation gives 1e-7."

Why should I not be alarmed in that case?

If I know my values are so small (around 1e-7) per nature, a difference between 1e-7 and 2e-7 is significant.

[Torsten]

Why should I not be alarmed in that case?

If I know my values are so small (around 1e-7) per nature, a difference between 1e-7 and 2e-7 is significant.

The values you investigate are numerical derivatives (usually of complex functions). Already when choosing the h in approximating the derivative as (f(x+h)-f(x))/h can cause such differences.

Answer 1

[Matt J]

If I know my values are so small (around 1e-7) per nature, a difference between 1e-7 and 2e-7 is significant.

Not necessarily. The element-wise relative error is large, but remember lsqnonlin uses the Jacobian J in matrix products of the form J*x and J.'*y to compute its updates. If the average value of abs(J(m,n)) is 1000 and the average values of abs(x(n)) or abs(y(m)) is 10, then an absolute error of 1e-7 in any entry J(m,n) would likely contribute negligibly to J*x or J.'*y.

Regardless, if you don't have full confidence in DerivativeCheck, you can implement your own finite difference computation, using for example,

https://www.mathworks.com/matlabcentral/fileexchange/13490-adaptive-robust-numerical-differentiation

and then do the comparison according to a criterion you prefer.