Interesting. This is an application of the detrended fluctuation analysis (DFA) to a 2D image. Based on what your screenshot shows, it implements the algorithm similarly like being implemented to a time series -- cut into segments based on a time scale s (or here a time-spatial scale), integration (cumulative sum), linear fitting to get residual, and finally there should be a log-log fit between s and F(s).
To answer the specific question you asked, the residual is nothing but the point difference between G_{m,n} and \tilde{G}_{m,n}. This means, the y variable y_{m,n} = G_{m,n} - \tide{G}_{m,n}. F is simply the average of sum of squared y.