Significance level of goodness of fit tests (Chi Square)

Dear forum,

I am a bit confused about the significance level of the Chi Square test in Matlab. I try to do some curve fitting and I use the Chi Square test to test whether data come from a Gamma distribution or not. However, the lower I set the significance level, the fewer rejections of the null hypothesis I get. I tested 20 datasets at the default 5% (0.05) significance level and I got 5 rejections of the null hyopthesis. At the 1% level (0.01) I only get 2 rejections. This seems to be strange to me as I thought a 1% level was more conservative? Shouldn't it be the other way around? The higher alpha the fewer rejections?

A brief reply would be appreciated! - Peter