One problem is that mle's search function (fminsearch) may try negative parameter values, in which case your pdf function returns a negative value. You could avoid that problem by squaring the parameter values, like this:
[phat,pci] = mle(A,'pdf',@(A,x,y) x^2*y^2*exp(-y^2*A),'start',[1,1])
If you do that, just keep in mind that the "real" x and y values you want are the squares of the phat values returned by mle.
Another problem, though, is that these are not legitimate pdf and cdf functions except when a=1. So, I think you really only have one parameter to estimate here--namely, your y parameter. It looks to me, anyway, like you are fitting an exponential distribution to these A values and looking for the best-fitting rate parameter. That value is 0.03333 = 1/mean(A)