Image processing with sub pixel accuracy

Question

I'm writing a user interface program which controls industrial machinery.

The machine is an automatic inline screen printer used in the electronic production industry. The exact purpose of the machine is to print solder paste onto bare PCBs. The system includes 2 cameras which form a vision system used to locate fiducials and automate the process that the machinery is involved in. The issue I have is that I need to be able to locate shapes and objects within images taken from the cameras, which is straight forward enough, but - in order to get the required accuracy I need to be able to do this with sub pixel accuracy.

The two cameras form one twin vision camera, with one camera will be looking up at a stencil, and the other looking down at a PCB. The twin vision camera is on a moving carriage that moves in X and Y. The PCB is clamped onto a table than can move in X, Y and theta. The vision system is used to detect the difference between PCB position and stencil position, and the table is then adjusted to align the two together.

EDIT:

This is a partial screen shot taken from an existing UI program:

The stencil image is the top image, the PCB is the bottom, the images here are grayscale images - in the new system I am looking at using colour but will use grayscale is required.

NOTE: As you're reading this you're probably thinking "well why don't you just use whatever you use in the existing UI?" The existing UI is written in VB6, something I want to move away from, and the vision side of the program is an .ocx written by a third party company who no longer exists.

I have played with AForge.NET a little bit and found it was really easy to use and I was able to use it to locate various different shapes and find the center of them which was great, but it doesn't have the sub pixel accuracy. However I could possibly use this as a starting point then apply a sub pixel algorithm to the individual center pixel or even the whole shape.

EDIT:

Here is a sample image taken from the test program I've written using AForge:

The red outline and crosshair is what I've added as a visual aid/ experiment. This was taken with the camera that I'm likely to use, which has a resolution of 1280 x 1024, however the lens is not the actual lens I'll use, which is why the image is slightly 'fisheyed'. Also the objects of interest will be better light than this.

In the actual system the lens will look at a 10mm x 8mm sqaure on both stencil and PCB, meaning each pixel will represent 7.8125um^2, however I can move the XYY table by increments of 1.25um at a time, which is essentially useless if I can't see those movements with the camera. I require 1um (sub) pixel accuracy.

Does anyone know of anything I can use to do this? I have been searching for quite a while now but all I ever seem to find is information on rendering images with sub pixel accuracy.

Or, better still, does anyone know how I could write something to do this myself? I wouldn't even know where to begin!

Any feedback will be greatly appreciated.

Answer 1

new info tells a lot... I think you should be more focused on the paste application accuracy then the carriage position precision. My bet is that the printer has far bigger dot size and error then 1 um and also the desired accuracy is dependent on the PCB usage (wiring and gaps widths). Anyway I would do this:

1. linearize image geometry

   you need to de-fisheye image. As you camera/optics setup will be fixed (especially focus and distance to PCB) you should for each PCB thickness make an image of a chess board grid. then create mapping that will linearize the chessboard to real rectangles so you discard deviations before the next process.

2. normalize lighting conditions

   • see Enhancing dynamic range and normalizing illumination

3. sub pixel accuracy

   each pixel of the image is integration of all the stuff in its area. So if we know the 2 colors (c0,c1) of any boundary (background/foreground) then we can estimate their sub-pixel position. Let start with axis aligned rectangles. I see it like this:

   

   Each square of the grid represents a pixel area. c0 is the gray and c1 is the green color. In the camera image you got the final color as combination of all the colors inside each pixel:

   • c = s0*c0 + s1*c1

   where c is the final pixel color and s0,s1 are area coresponding to c0,c1 colors in range <0,1> where s0+s1=1.0. Now we want to compute the s0,s1 to obtain sub-pixel precision. So first detect the pixel position on the boundary as one of these:

   • horizontal edge
   • vertical edge
   • corner

   this can be done by inspecting the neighboring pixels. c0,c1 can be obtained from pixels with saturated color (all neighbors have the same color) These are in inside areas pixels. I would ignore the corner pixels as their position can be obtained from nearest H/V edge pixels (it is not possible to get both x,y coordinates from the equation above). So now for each H,V edge just solve system:

   I.  c = s0*c0 + s1*c1
   II. s0 + s1 = 1.0;
   

   compute the s0,s1 and the edge position for vertical edges is one of these:

   x=x0 + pixel_size*s0 // if c0 is on the left
   x=x0 + pixel_size*s1 // if c1 is on the left
   

   horizontal edges are like this:

   y=y0 + pixel_size*s0 // if c0 is on the top
   y=y0 + pixel_size*s1 // if c1 is on the bottom
   

   where x0,y0 is pixels top left position in pixels and the coordinate system is x+ is going to right and y+ is going down. If you got different setup just change the equations ...

   Now if you got non axis aligned edges then you need to compute the slope (how many pixels it takes in one axis to change in the other dy/dx. and handle the areas accordingly:

   

   So the only thing what changes is the conversion from computed s0,s1 to actual edge position. now you need to compute the left/right side or up/down. if you use the equation from the axis aligned example then you obtain the edge position in the middle of pixel. So you just shift it by the slope on both sides by x +/- 0.5*dx/dy or y +/- 0.5*dy/dx where dx,dy is the edge slope.

   

   To obtain dx,dy just search along the edge for fully saturated pixels and if found then (dx,dy) is the distance between 2 closest such pixels...

[Notes]

you can do this on booth grayscale and RGB. Hope this helps a bit.