How to map the disparity image values to the original image?

Using stereo camera calibration and Matlab's instructions I produce the disparity map. Now I want to create a Kinect-like depth map for which I need to map the depth image to the original image (e.g., Camera 1). The problem is that the "disparity" and "reconstructScene" output matrices with the dimensions of the image produced by the rectifyStereoImages function and not the original image. I wonder how I can map the disparity values/image to the pixels of the original image. Below is an example of the code (a simplified version and not the actual one I am running, but mostly the same).

    I1 = imread(name1); %original image
    I2 = imread(name2);
    [J1,J2]=rectifyStereoImages(I1,I2,stereoParams, 'OutputView', 'full');
    disparityMap = disparity(rgb2gray(J1), rgb2gray(J2), 'DisparityRange', ...
    disparityRange);
    point3D = reconstructScene(disparity, stereoParams);
% Where:
% size of I1   is      5208x3476x3
% size of disparity is 3672x7112
% size of J2   is      3672x7112x3

And now I want to get a depth image or at least a disparity image with the size 5208x3476 which corresponds pixel-wise to the original image (I1)

I appreciate if anyone has an idea/suggestion.

Best wishes,

Answer 1

[Dima Lisin]

You would have "un-rectify" the image, and there is no easy way to do that because rectification is a non-linear transformation.

[Shida]

I was fearing that would be the case. So basically what you're saying is that there's no way one can get a kinect-like depth images which correspond to the original RGB image. Right?

[Dima Lisin]

Does it have to be the original RGB image, or can it be the undistorted RGB image? If it is the latter, you can undistort your stereo images, instead of rectifying them. Then compute dense optical flow from one into the other. Then use triangulate to get a dense 3D reconstruction, and compute the depth from that.

Out of curiosity, why not just compute the depth map for the rectified image?