Accurately correcting the offset between two objects with the same shape from two different images where segmentation is not possible

Question

I have two images of an object with the same shape at different positions. My goal is to modify one of the images so that the position of the object aligns with the object of the reference image.

See below reference image:

And image to modify:

To deal with this problem, I followed these steps: (1) identification of an analogous point, (2) calculation of the offset and (3) modification of image. The approach works but it is not very accurate (I need a nearly perfect alignment) nor efficient because identification of the point is manual with the mouse.

Here is my attempt for the record:

Find analogue coordinates of the object in images

import cv2
import numpy as np

# list of coordinates
left_clicks = []

# define a function to display the coordinates of the points clicked on the image
def click_event(event, x, y, flags, params):
   if event == cv2.EVENT_LBUTTONDOWN:
      print(f'({x},{y})')
      
      # put coordinates as text on the image
      cv2.putText(img, f'({x},{y})',(x,y),
      cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 0, 255), 2)
      
      # draw point on the image
      cv2.circle(img, (x,y), 3, (0,255,255), -1)

      # store coordinates
      left_clicks.append((x,y))
 
# read the input image
img = cv2.imread('red_object.png')

# create a window
cv2.namedWindow('Point Coordinates', cv2.WINDOW_NORMAL)

# bind the callback function to window
cv2.setMouseCallback('Point Coordinates', click_event)

# display the image
while True:
   cv2.imshow('Point Coordinates',img)
   k = cv2.waitKey(1) & 0xFF
   if k == 27:
      break
cv2.destroyAllWindows()

# define a function to display the coordinates of the points clicked on the image
def click_event2(event, x, y, flags, params):
   if event == cv2.EVENT_LBUTTONDOWN:
      print(f'({x},{y})')
      
      # put coordinates as text on the image
      cv2.putText(img2, f'({x},{y})',(x,y),
      cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 0, 255), 2)
      
      # draw point on the image
      cv2.circle(img2, (x,y), 3, (0,255,255), -1)

      # store coordinates
      left_clicks.append((x,y))

# read the input image
img2 = cv2.imread('blue_object.png')

# create a window
cv2.namedWindow('Point Coordinates2', cv2.WINDOW_NORMAL)

# bind the callback function to window
cv2.setMouseCallback('Point Coordinates2', click_event2)

# display the image
while True:
   cv2.imshow('Point Coordinates2',img2)
   k = cv2.waitKey(1) & 0xFF
   if k == 27:
      break
cv2.destroyAllWindows()
print(left_clicks)

Find offset

coordinates_ref = left_clicks[0]
coordinates_offset = left_clicks[1]

offset_x = coordinates_ref[0] - coordinates_offset[0]
offset_y = coordinates_ref[1] - coordinates_offset[1]

print(offset_x, offset_y)

Modify image to align objects

values = img2[0,0]
h, w, nch = img2.shape

margin_side = np.ones((h, abs(offset_x), 3), dtype=int) * values
margin_top_bottom = np.ones((abs(offset_y), w, 3), dtype=int) * values

if offset_x < 0:
    img2_new = np.concatenate((img2[:,abs(offset_x):], margin_side), axis=1)

if offset_y > 0:
    img2_new = np.concatenate((margin_top_bottom, img2_new[:(h-abs(offset_y)),:]), axis=0)

cv2.imwrite('modified_blue.png', img2_new)

Which produces:

And I can visually see that the output image approximately aligns with the reference image but is far from exact. An improvement would be to use findContours and find the centroid of both objects to calculate the offset. However, these are two simplified images for explaining the problem–the real images don't have contrast enough with the background for proper segmentation.

The desired answer would include a more accurate or/and efficient method that doesn't rely on segmentation nor is done manually.

EDIT

The real data consist in many z-scans at different focal depths. Then, I fuse the scans to obtain a multi-focused image using SESF-Fuse. However, the instrument that I use is not perfectly aligned along the z axis and the images are slightly shifted:

This one slightly shifted towards the left:

My goal is to correct this shift so that I can fuse the images. Since the real images are too large to post here, I can share a folder with the about 40,000 x 40,000 px scans if needed.

Answer 1

If we use ECC (assume affine) warping in Python/OpenCV, I get the following:

Script:

import cv2
import numpy as np
import math
import sys

# Get the image files from the command line arguments
# These are full paths to the images
# image2 will be warped to match image1
# argv[0] is name of script
image1 = sys.argv[1]
image2 = sys.argv[2]
outfile = sys.argv[3]

# Read the images to be aligned
# im2 is to be warped to match im1
im1 =  cv2.imread(image1);
im2 =  cv2.imread(image2);

# Convert images to grayscale for computing the rotation via ECC method
im1_gray = cv2.cvtColor(im1,cv2.COLOR_BGR2GRAY)
im2_gray = cv2.cvtColor(im2,cv2.COLOR_BGR2GRAY)

# Find size of image1
sz = im1.shape

# Define the motion model - euclidean is rigid (SRT)
warp_mode = cv2.MOTION_EUCLIDEAN

# Define 2x3 matrix and initialize the matrix to identity matrix I (eye)
warp_matrix = np.eye(2, 3, dtype=np.float32)

# Specify the number of iterations.
number_of_iterations = 5000;

# Specify the threshold of the increment
# in the correlation coefficient between two iterations
termination_eps = 1e-3;

# Define termination criteria
criteria = (cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, number_of_iterations,  termination_eps)

# Run the ECC algorithm. The results are stored in warp_matrix.
(cc, warp_matrix) = cv2.findTransformECC (im1_gray, im2_gray, warp_matrix, warp_mode, criteria, None, 1)

# Warp im2 using affine
im2_aligned = cv2.warpAffine(im2, warp_matrix, (sz[1],sz[0]), flags=cv2.INTER_LINEAR + cv2.WARP_INVERSE_MAP);

warp_diff = cv2.absdiff(im2_aligned, im1)

# write output
cv2.imwrite(outfile, im2_aligned)
cv2.imwrite('warp_diff.png', warp_diff)

print(warp_matrix)

# Print rotation angle
row1_col0 = warp_matrix[0,1]
angle = math.degrees(math.asin(row1_col0))
print(angle)

Command:

python python_ecc_warp.py img1.png img2.png x.png

This warps img2.png to match img1.png producing x.png

Result:

The warp matrix is:

[[ 1.0000000e+00 -1.4341946e-04 -1.2956342e+00]
 [ 1.4341946e-04  1.0000000e+00 -6.1281472e-02]]

which show an x shift of about 1.3 pixel and a y shift of about 0.06 pixels.

And the rotation angle is about:

-0.008 deg

Now just compare the difference between x.png and img1.png via cv2.absdiff().