nonlinear 2d calibration of line-camera using known pattern - how to align world coordinates to points?

I rephrased the question, see last comment

--------------------------------------------

Hi all,

I'm currently working with a highres line camera scanner and we want to use it for measurements. Therefore we need to be able to calibrate it properly in 2d.

We have an accurate test-pattern made by photolithography. We can assume it to be 'exact'. I have managed to come up with a very robust subpixel circle recognition to find the points of interest so I have no question regarding image recognition.

What is unclear to me at this point is how to use the found image coordinates together with the known spatial correlation in order to come up with a distortion map which ultimately would be used to correct the distortion.

The example shows a cropped subset of a typical image of circles with 400µm xy spacing. The recognition is already done (red circles with blue centercross) leaving out structures that are too close to the edge. Now, how do I calculate a distortion-map from that?

I played around with some algorithms like knnsearch in order to look at the lokal distances and I can see that they vary significantly. I don't want to simply create a coordinate system using arbitrary points and then measure relative distances to that system because I fear that might induce large errors over big distances.

My guess is that using a combined approach of the local information (distances) in combination with the 'far field' (straight lines, orthogonality etc.) should result in the most robst outcome but I'm stuck on how to continue. In the end we want to do many measurements in order to determine the measurement errors caused by vibration and thermal influences i.e.

I'm fairly certain this has been solved (many) times but as a non-expert on the field I just don't know what to search for exactly. I would appreciating any help!

regards

Arnold

[arnold]

out of clarification. This camera is a line camera, so we expect local nonlinearities as well since only one dimension is fixed in silicon, the other dimension defined by timing a linear stage that moves the sample. So I don't know what to call this problem in computer science terms, to me it's a nonlinear, not necessarily continuous distortion meaning: not just lens distortions but an 'arbitrary jitter' at least in the second dimension is all I can think of.

I can think of pros and cons for a global or relative approach I guess one should consider both or a mix for this usage scenario, but I am no expert.

Looking at some of the function matlab provides like estimateCameraParameters or imref2d - I wouldn't know how to define world coordinates since all I have is the point distance xy which is constant.

I'd be happy if anyone could point me in the right way.

[Walter Roberson]

Is there a possibility of a second pass through the camera, preferably with the same sample rotated?

[arnold]

sure, sorry for the confusion. As stated in the answer to Image Analyst's question, that's part of the plan in order to get statistics. The calibration sample is calibrated down to 100nm, 10 times the resolution we want so rotation shouldn't be necessary but we'll do it too.

[arnold]

Rephrased question:

I don't want to reinvent the wheel. I want to determine the camera distortion so I can use this to unwarp the image (and other images). I use an image of an calibration pattern. The pattern is larger than the field of view in order to cover the enitre area. This pattern has a known grid distance (real world). The locations of the pattern marks are determined accurately giving image coordinates. From here on out I have the following question(s):

1. A function that takes image coordinates, known grid spacing and then gives the image distortion vectorfield would be the final goal.

I'm at a loss. I'm sure though, that I must be looking at it the wrong way. Anyways, people must have solved this many times over but I don't find an implementation that's general enough - apparently.

Any help greatly appreciated.

[arnold]

no idea, anyone?

[praveen rai]

@arnold can u provide me the code that how u find robust subpixel circle recognition bcz i want to find the square

Answer 1

[Image Analyst]

I'd look at the average inter-row spacing as a function of row. It should be uniform. If there's a bias, you can determine that and use it to correct your real, non-calibration images. If you scan the calibration target multiple times and find that the inter-row distance is not deterministic and basically random, then there's not much you can do.

[arnold]

I'm aware of the problem regarding the inter-row distance of the linescan method and this is actually part of the final goal to get basically two matrices:

1. deterministic, constant distortion map

2. nondeterministic but predictable error-map. Might as well be that it's a constant jitter for all lines/columns that are being recorded by one linescan.

I'm confident I know how to do the statistics for multiple measuremnents and error estimation later but my question remains to be around this problem of determining the distortion for a single image. I just don't know how to approach this. :)

[arnold]

here is an example of the knnsearch result. The red lines indicate a distance of less than 400µm, the blue ones indicate a distance > 400µm. Looking at the histogram one could conclude a compression along the y-axis and an elongation along the x-axis.

but again, I don't know how to continue to get a 'distortion vectorfield' for each pixel of such an image. I feel like I'm just missing one clear thought on the problem but then again I'm certain this is a well known technique, just not to plain physicist like me :)

[Image Analyst]

You can't determine the "distortion" for a single image unless you have some known landmarks in there. For example if you just put a slide with some amorphous biological cells on it into your system, you won't be able to tell whether it's distorted or not. Will all your samples have some known landmarks on them?

[arnold]

ah, I see why you might be confused. Maybe I'm just not describing the problem clearly, let's try once more:

1. we like to calibrate this (and other) camera- and linear stage- systems using highly accurate calibration patterns that have well defined spacings (photolithographic masks i.e.). These can fill the entire field of view. Fiducial recognition is no problem, we know how to do that so we get the image coordinates of these teststructures. Our goal now is to get a distortion map / vector field for such an image from both inputs: real world distances/positions and image coordinates.

2. we'll repeat this process in order to get some information about the error/repeatability of such a 'camera calibration'. Naturally, one would translate/rotate the known calibration pattern. My first guess would be that one would determine a mean distortion as a result of multiple calibration runs and also a possibly location/vector based error estimation. It is to be expected, that the error along the scanning line is much smaller than along the moving stage, so be it.

3. We then want to measure samples that are non calibrated, like workpieces from a production. There are well defined fiducials/landmarks on these workpieces too. Again, we can do the recognition of these landmarks, it's already finished. The goal now is to determine the (relative) location of these points of interest in real world coordinates. .... basically, use the camera as a measurement device.

[Image Analyst]

That's what I thought, though I wasn't sure if you had fiducials/landmarks in your samples. So, what's the problem? You can find the circles and can do so for lots of calibration images so you can know any systematic and random bias/jitter from line to line. And I'm assuming you can then just apply this transform, at least the systematic part, to the "real" sample. Now sure where the stumbling block is.

[arnold]

hehe, the problem must be in my head, I'm aware of that :)

So, where am I at:

I can extract the list with XY coordinates (image coordinates) of the test pattern. What do I do with that now? I'm lacking the knowledge of the proper algorithm to process this XY list together with the info, of constant distance. I would assume, there is a known algorithm, possibly implemented many times, to formulate a warping map, a vector fields with local offsets or something, which can then be interpolated and inversely applied in order to correct this and other images accordingly.

... but I don't have that knowledge nor can I find it. It might be that I'm confused by the fact that I only have relative coordinates for the real-life test pattern structures as to absolute image coordinates?

How would you do it? An algorithm (in plain words) or sample implementations would probably help me do it. I'm not lazy, I'll code it, but I don't know the approach and again, I'm almost certain this has to have been implemented somehow many times by now.

[arnold]

here's a set of coordinates, maybe that's the easiest for you to understand my apparent mind blockade.

The real life distance is 'exactly' 400. I don't know how to procede :)

[Image Analyst]

Is the slant of your grid at a known, fairly constant angle? If so, you can use a template to determine which of the found circles are in which row. Then find out the distance of each circle to the closest circle in the adjacent row. Now you have a N by 2 array of row spacing and row. You can do a regression to see if the row spacing depends on the line. Do a scatterplot first to see if regressing a line through them makes sense. If it does, like it looks like a quadratic or something, then use polyfit().

[arnold]

I can keep the angle more or less constant (+-5°) but this seems to be a dirty workaround though. I tried to create a point mesh with distance 400 and then use some incarnation of 'iterative closest point' to shift this template towards the best match with the found points but this wasn't robust at all. I mean, ICP shifts the measured point cloud so it has the least square error versus the template grid - that's bound to be a non-robust approach.

Attached you see heatmaps of column and row distances. The lines of the line-camera are horizontal, the stage is travelling vertically. That's why you see a clear error correlation of the VERTICAL distance heatmap - as expected.

I want a 2d calibration, not just test the line distance in one dimension. Here is the delta-vector field that I get using this ICP approach with a template of world coordinates.

My final goal is to end up with a matrix with columns x_image, y_image, dx, dy and I want to use a proven/good algorithm to allocate measured landmark-coordinates with world-coordinates....Only the distance between the landmarks is fixed, there is no DOMINANT landmark to create a coordinate system with and define an origin.

So the question might be how to create the coordinate system of the world coordinates properly (!) and align it with the image coordinate system?

[arnold]

here is an illustration of this iterative closest points approach.

1. I create a grid of points with the known world distance of the marks: 400 (µm or a.u.)

2. I overlay this synthetic grid with the found image coordinates of the marks (see upper left diagram)

3. ICP runs and tries to rigidly translate and rotate the measured point cloud to match the synthetic one (see upper right).

4. I then calculate the dx/dy from each measured point to its closest synthetic counterpart.

The first problem that comes to mind: For large samples dimensions (you only see about 1% of the image field of view), cumulated distances could end up way further than the dot distance so ICQ will fail. ...and that's just the first problem out of many I see.

dx dy vectorfield based on ICQ alignment of synthetic to measured points.

This diagram now shows the vector field created by using rigid ICP to align the measured dots with artificial template dots as shown above. So obviously, for a mostly symmetrical distortion, the errors around the center are almost zero and the peripheral ones are large. I'd be ok with that but the approach using ICP doesn't work as argued above. If the cumulative errors become larger than the point distance, i.e.

[arnold]

maybe I need to rephrase the question and re-post it under a new title? I think I have most of the pieces with the one piece missing being the allocation image points to world points

[Walter Roberson]

I recommend posting your rephrased question here as a comment on your original question.

[arnold]

done, see above.