Mapping RGB and depth (Kinect v2)

Hi everyone, for my project, I'm working with the Kinect v2 and I have to map the depth information onto the RGB images to process them: in particular, I need to know which pixels in the RGB images are in a certain range of distance (depth) along the Z axis; I'm acquiring all the data with a C# program and saving them as images.

I had a look on the internet and I found some equations to map depth and RGB info here. I am actually able to get the intrinsic parameters for the single camera calibration using a checkerboard recorded from both the RGB camera (1920x1080) and the IR sensor (512x424) and processing them with MatLab Camera Calibrator app, but I'm stuck doing the stereo calibration (to find the rotation and translation matrices), because MatLab Stereo Camera Calibrator doesn't accept pair of images with different resolution.

I've also tried some other method with no results; It would be great if anyone could help me figuring out what I have to do in order to solve my problem. Thanks very much!

[dodesheide]

Matlab Stereo Camera Calibrator is not the right way, because it is not a Stereo Calibration. Take a look at this matlab calibration: https://github.com/kapibara/ToF-Calibration or take a look at the paper "Joint Depth and Color Camera Calibration with Distortion Correction" from Herrera. That would help you to map color to depth.

[Walter Roberson]

https://www.codefull.org/2016/03/align-depth-and-color-frames-depth-and-rgb-registration/

[Muhammad Hammad Malik]

hello all, i want to do depth-color image registration but do not know how to do it, kindly guide me for this. if anyone have working code or wahtever, kindly help me. thanks

[Walter Roberson]

Muhammad Hammad Malik: the link I posted for codefull has MATLAB code for RGB registration with depth for Kinect.

[Muhammad Hammad Malik]

i tried to use that link but i am unable. kindly check it what things i need to do.see attach code. thanks

function [aligned] = ...
                    depth_rgb_registration('F:\kinectdata\cloud\tomato.ply', 'F:\kinectdata\color\tomato1.jpg',...
                    1.1752e+03, 1.2589e+03, 849.6059, 726.8106,...
                    1.0585e+03, 1.1341e+03, 965.1123, 583.2686,...
                    [1.0000   -0.0055    0.0003  -78.2239;
    0.0055    1.0000    0.0002   31.3206;
   -0.0003   -0.0002    1.0000   -8.0845])
fx_rgb_len =   3.28135;
fy_rgb_len =   3.51571;
fx_d_len =   3.64132;
fy_d_len =   3.90295;
 depthHeight=424;
  depthWidth=512;
    depthHeight = size('F:\kinectdata\cloud\tomato.ply', 1);
    depthWidth = size('F:\kinectdata\cloud\tomato.ply', 2);
    % Aligned will contain X, Y, Z, R, G, B values in its planes
    aligned = zeros(depthHeight, depthWidth, 6);

      for v = 1 : (depthHeight)
          for u = 1 : (depthWidth)
              % Apply depth intrinsics
              z = single('F:\kinectdata\cloud\tomato.ply'(v,u)) / depthScale;
              x = single((u - 849.6059) * z) / 1.1752e+03;
              y = single((v - 726.8106) * z) / 1.2589e+03;

              % Apply the extrinsics
              transformed = ([1.0000   -0.0055    0.0003  -78.2239;
      0.0055    1.0000    0.0002   31.3206;
     -0.0003   -0.0002    1.0000   -8.0845] * [x;y;z;1])';
              aligned(v,u,1) = transformed(1);
              aligned(v,u,2) = transformed(2);
              aligned(v,u,3) = transformed(3);
          end
      end

      for v = 1 : (depthHeight)
          for u = 1 : (depthWidth)
              % Apply RGB intrinsics
              x = (aligned(v,u,1) * 1.0585e+03 / aligned(v,u,3)) + 965.1123;
              y = (aligned(v,u,2) * 1.1341e+03/ aligned(v,u,3)) + cy_rgb;

              % "x" and "y" are indices into the RGB frame, but they may contain
              % invalid values (which correspond to the parts of the scene not visible
              % to the RGB camera.
              % Do we have a valid index?
              if (x > rgbWidth || y > rgbHeight ||...
                  x < 1 || y < 1 ||...
                  isnan(x) || isnan(y))
                  continue;
              end

              % Need some kind of interpolation. I just did it the lazy way
              x = round(x);
              y = round(y);

              aligned(v,u,4) = single('F:\kinectdata\color\tomato1.jpg'(y, x, 1);
              aligned(v,u,5) = single('F:\kinectdata\color\tomato1.jpg'(y, x, 2);
              aligned(v,u,6) = single('F:\kinectdata\color\tomato1.jpg'(y, x, 3);
          end
      end    
  end

[Walter Roberson]

A function statement cannot have any numeric constants or any quoted strings or any {}. You need to put back the variable names and you need to call the function passing in the necessary information.

Also note that taking size() of a single quoted string is an operation that is asking about the number of characters in the literal string. No attempt is made to see if the string resembles a file name or to see how much information might be stored in the file.

[Muhammad Hammad Malik]

alright, thanks, but i can understand a little bit, because i dont know much about programming. if you can then kindly solve the issue. i shall be very thankful to you.thanks

[Walter Roberson]

Following some links I see that someone has created a small toolbox at https://sourceforge.net/projects/kinectcalib/

[Muhammad Hammad Malik]

thanks for this, i used this but it is always saying that device is not detected but when i run in some other code then device is working

Answer 1

[Shida]

Hi alheka, I used to have the same problem. It's a pity that Matlab still hasn't solved this issue. But there's a solution to that. It took me quite a while to figure it out. Here you have the steps you need:

1. Stereo Calibration:

a) Find checkerboard points:

    [pointsRGB, boardSizeRGB, imagesRGBused] =detectCheckerboardPoints(imagesRGB);
    [pointsIR, boardSizeIR, imagesIRused] = detectCheckerboardPoints(imagesIR);

b) Remove the pairs where in one of the sets the points were not detected. Matlab does that automatically for the stereo calibration. But you need to write your code. I did make such code, but it's too unreadable to share here !

c) Now you need to manually combine the two sets.

    imagePoints = cat(4, poitnsRGB, pointsIR);

d) Now the word points: where squaresize is the width of each of the squares in your calibration target (unit is mm).

    worldPoints = generateCheckerboardPoints(boardSizeRGB, squaresize);

e) Calibration:

    [stereoParams,pairsUsed,estimationErrors] = estimateCameraParameters(imagePoints,...
    worldPoints,'EstimateSkew', false, 'EstimateTangentialDistortion', false, ...
    'NumRadialDistortionCoefficients', 2, 'WorldUnits', 'mm');
figure; showReprojectionErrors(stereoParams);
figure; showExtrinsics(stereoParams);

f) If needed, remove pairs of images with high error and re-calibrate until you're satisfied with the error.

2) Map the images: Actually the link you have is correct for this part. Just follow those instructions.

3) Remember that some artifacts (repeated pixels) can be caused in the mapped image due to parallax and the fact that some pixels are visible in one camera and not in the other. There are ways to fix that.

I hope it works for you.

Good luck.

[Shida]

P.S: You might want to change the calibration parameters according to you camera to decrease the error.

[alheka]

Thank you very much for the help, Shida. I still have some problems:

1) in the figure that comes out from the stereo calibration, camera 1 (RGB camera) is represented on the left respect to camera 2 (depth sensor), but in the Kinect is the opposite

2) I still don't manage to map depth information onto RGB images.

Through the calibration I have calculated the rotation (R) and translation (T) matrix between the depth sensor and the RGB camera, as well as their intrinsic parameters.

I have created P3D_d (depth pixels in world coordinates related to depth sensor) and P3D_rgb (depth pixels in world coordinates related to rgb camera).

row_num = 424;
col_num = 512;

P3D_d = zeros(row_num,col_num,3);
P3D_rgb = zeros(row_num,col_num,3);
cont = 1;
for row=1:row_num
    for col=1:col_num
        P3D_d(row,col,1) = (row - cx_d) * depth(row,col) / fx_d;
        P3D_d(row,col,2) = (col - cy_d) * depth(row,col) / fy_d;
        P3D_d(row,col,3) = depth(row,col);

          temp=[P3D_d(row,col,1);P3D_d(row,col,2);P3D_d(row,col,3)];

          P3D_rgb(row,col,:) = R*temp+T;
      end
  end

I have created P2D_rgb_x and P2D_rgb_y.

P2D_rgb_x(:,:,1) = (P3D_rgb(:,:,1)./P3D_rgb(:,:,3))*fx_rgb+cx_rgb;
P2D_rgb_y(:,:,2) = (P3D_rgb(:,:,2)./P3D_rgb(:,:,3))*fy_rgb+cy_rgb;

but now I don't understand how to continue.

Assuming that the calibration parameters are correct, I've tried to click on a defined point in both the depth (coordinates: row_d, col_d) and rgb (coordinates: row_rgb, col_rgb) images, but the return of P2D_rgb_x(row_d, col_d) is totally different from row_rgb, as well as the return of P2D_rgb_y(row_d, col_d) is totally different from col_rgb.

So, what do exactly mean P2D_rgb_x and P2D_rgb_y? How can I use them to map depth value onto rgb images or just to get the depth of a certain RGB pixel?

I'll apreciate any suggest or help!

[Shida]

Hi, About the first point, could you first confirm that you have already mirrored the Kinect images before doing the calibration. If so, then it could be the coordinate system of Matlab which causes this. If it's not either of them, then I need to check my own setup and see what I've done. That was a while ago!

About the mapping, I didn't have time to read all of your steps. But I attached my own (experimental) code here. I know that's not a nice code to look at as I wrote it back then in a hurry, but it might help you get an idea. I also don't guarantee if the code is correct. I won't be surprised if there are some stupid mistakes which I have overlooked. One should definitely be able to write it much simpler and more optimized.

Keep in mind that since depth image has a lower resolution, your results will be sparse (as some pixels won't have a value). One last thing, don't forget the parallax problem and don't worry if some pixels are repeated. One can write another code to fix that.

Good luck!

[alheka]

Thank you very much Shida! For the first point, you were right: I forgot to mirror the image. For the mapping, your solution is working quite well and now I can optimize it for my project. You have been really very kind, thanks very much again!

[Shida]

You're welcome! I'm glad it helped.

[Somrita Chattaopdhyay]

Hi,

I am also trying to align the depth and color frames of a kinect 2.0 sensor. However, my final results seems to be off. Can you guys share your results of calibration with me; so that I can check mine.

Thanks. Somrita

[Dominik Melcher]

Hi Shida,

I am trying to map the Color Pixel to Cameracoordinates. I followed your step you gave above, but I stuck at step 2, where I have to use the functions of the link. For some reason the depth Pixel doesnt match to the calculated colorpixel and I dont know why. Could you send me you matlab code of this matching pls?

Thanks, Dominik

[Franz-Josef Siegemund]

Hi there,

i know this is a really old entry, but i would like to know if anybody did map from rgb->depth? I did get the transformation from depth->rgb working. But i am having trouble understanding the whole extrinsic/instrinsic camera parameters and how to transform with each. (A book or some reference would be enough).

In particular, i want to transform the position of 4 points from rgb image space -> depth.

Answer 2

[Muhammad Hammad Malik]

hello all, i want to do depth-color image registration but do not know how to do it, kindly guide me for this. if anyone have working code or wahtever, kindly help me. thanks

[Ankit Jaiswal]

Hi, Did you get success in this?

[Muhammad Hammad Malik]

see the following link

https://www.codefull.org/2016/03/align-depth-and-color-frames-depth-and-rgb-registration/