Given that the same key is used to encrypt and decrypt the image, then the algorithm is symmetric (simple xor with the image and the key is all that is happening).
It is the generation of the key that is interesting. From keyGen.m, the difference equation
x(n) = 1 - 2*x(n-1)*x(n-1)
is used to generate an 8-bit binary "key" for each pixel in the image (software seems to assume that the image data type is uint8). The values x(n) are in the interval [-1,1], with negative values mapped to zero, and positive values to 1.
If you add the above line of code to the software (it doesn't exist right now, and instead relies on two local variables for x(n) and x(n-1)) you can plot the data and look at the frequency of the values in the sequence
figure;plot(x,'b.');
figure;hist(x,100);
x(n) = r*x(n-1)*(1-x(n-1))
for r equal to 4. Pay particular attention to the section (given by the link) on Chaos and the logistic map.
While I can't tell you right out what the algorithm is called or which papers to reference, I suggest that you Google logistic map, crypotography, image and review what appears.
Else contact the author of the code! ;)
