how make panoramic view of serie of pictures using python, opencv, orb descriptors and ransac from opencv

Question

i need to make a panoramic view from serie of pictures (3 pictures). After that i created orb, detected and computed keypoints and descriptors for the three picture, i matched the most likely similars keypoints between:

1. image 1 and image 2

2. image 2 and image 3

Then i know to compute and find the panoramic view between only 2 images , and i do it between img1 and img2, and between img2 and img3. But Then for the last step , i want to find the panoramic view of theses 3 pictures using affine tranformation with Ransac algorithm from opencv. And i don't know to do that ( panoramic view for 3 pictures. So, I have to of course choose image 2 as the center of the panorama

I didn't find a good explanation or an enough good explanation for me in order to compute and fine the panoramic view of theses 3 pictures. Someone can help me please to implement what i need please ?

here is my code where i print panorama between img1 and img2, and between img2 and img3:

import numpy as np
import cv2 as cv
from matplotlib import pyplot as plt
from matplotlib.pyplot import imshow, show, subplot, title, axis


# draw matches
def draw_matches(img1, kpt1, img2, kpt2, matches):
    h1, w1 = img1.shape[:2]
    h2, w2 = img2.shape[:2]

    # Create a blank image with the size of the first image + second image
    new_img = np.zeros((max([h1, h2]), w1 + w2, 3), dtype='uint8')
    new_img[:h1, :w1, :] = np.dstack([img1, img1, img1])
    new_img[:h2, w1:w1 + w2, :] = np.dstack([img2, img2, img2])

    # extract the match keypoints
    for m in matches:
        (x1, y1) = kpt1[m.queryIdx].pt
        (x2, y2) = kpt2[m.trainIdx].pt

        # Draw circles on the keypoints
        cv.circle(new_img, (int(x1), int(y1)), 4, (0, 255, 0), 1)
        cv.circle(new_img, (int(x2) + w1, int(y2)), 4, (0, 255, 0), 1)

        # Connect the same keypoints
        cv.line(new_img, (int(x1), int(y1)), (int(x2) + w1, int(y2)), (255, 0, 255), 1)

return new_img


def warpImages(img1, img2, M):
    # get the corner coordinates of the "query" and "train" image
    h1, w1 = img1.shape[:2]
    h2, w2 = img2.shape[:2]
    pts_corners_src = np.float32([[0, 0], [0, h1], [w1, h1], [w1, 0]]).reshape(-1, 1, 2)
    pts_corners_temp = np.float32([[0, 0], [0, h2], [w2, h2], [w2, 0]]).reshape(-1, 1, 2)

    # When we have established a homography we need to warp perspective
    # perform perspective tranform using previously calculated matrix and the corners of  
    "query" image#
    # Change field of view
    pts_corners_dst = cv.perspectiveTransform(pts_corners_temp, M)

    ListOfPoints = np.concatenate((pts_corners_src, pts_corners_dst), axis=0)

    [x_min, y_min] = np.int32(ListOfPoints.min(axis=0).ravel() - 0.5)
    [x_max, y_max] = np.int32(ListOfPoints.max(axis=0).ravel() + 0.5)

    translation_dist = [-x_min, -y_min]

    H_translation = np.array([[1, 0, translation_dist[0]], [0, 1, translation_dist[1]], [0, 0, 
1]])

    new_img = cv.warpPerspective(img2, H_translation.dot(M), (x_max - x_min, y_max - y_min))
    new_img[translation_dist[1]:h1 + translation_dist[1], translation_dist[0]:w1 + translation_dist[0]] = img1

return new_img


def find_homography(kpt1, kpt2, matches):
    # Find an Homography matrix between two pictures
    # Transforming keypoints to list of points
    src_pts = np.float32([kpt1[m.queryIdx].pt for m in matches]).reshape(-1, 1, 2)
    dst_pts = np.float32([kpt2[m.trainIdx].pt for m in matches]).reshape(-1, 1, 2)

    # Compute a rigid transformation (without depth, only scale + rotation + translation) /affine transformation
    transformation_rigid_matrix, rigid_mask = cv.estimateAffinePartial2D(src_pts, dst_pts)
    affine_row = [0, 0, 1]
    transformation_rigid_matrix = np.vstack((transformation_rigid_matrix, affine_row))

return transformation_rigid_matrix


# Read images
img1 = cv.imread('1.jpg', 1)
img2 = cv.imread('2.jpg', 1)
img3 = cv.imread('3.jpg', 1)
img1 = cv.cvtColor(img1, cv.COLOR_BGR2GRAY)
img2 = cv.cvtColor(img2, cv.COLOR_BGR2GRAY)
img3 = cv.cvtColor(img3, cv.COLOR_BGR2GRAY)

# Initiate ORB detector, that it will be our detector object.
orb = cv.ORB_create()

# find the keypoints and compute the descriptors with ORB for images
kpts1, des1 = orb.detectAndCompute(img1, None)
kpts2, des2 = orb.detectAndCompute(img2, None)
kpts3, des3 = orb.detectAndCompute(img3, None)

# Create a BFMatcher object.
bf = cv.BFMatcher_create(cv.NORM_HAMMING)

# match descriptor
matches1to2 = bf.knnMatch(des1, des2, k=2)
matches2to3 = bf.knnMatch(des2, des3, k=2)

# draw matches
good1to2 = []
for m, n in matches1to2:
    if m.distance < 0.6 * n.distance:
        good1to2.append(m)

# draw matches
good2to3 = []
for m, n in matches2to3:
    if m.distance < 0.6 * n.distance:
        good2to3.append(m)

# find  affine transformation and panoramic view between 1 to 2
trans_affine_matrix1to2 = find_homography(kpts2, kpts1, good1to2)
img1to2 = warpImages(img1, img2, trans_affine_matrix1to2)

# find homography matrix and transformation 2 to 3
trans_affine_matrix2to3 = find_homography(kpts2, kpts3, good2to3)
img2to3 = warpImages(img3, img2, trans_affine_matrix2to3)

title1 = "panoramic between img1 and img2"
title2 = "panoramic between img2 and img3"
subplot(1, 2, 1)
imshow(img1to2)
axis('off')
title(title1)

subplot(1, 2, 2)
imshow(img2to3)
axis('off')
title(title2)
show()

Thank you all for your help. Here are my pictures:

Answer 1

Instead of creating panoramas from image 1 & 2 and image 2 & 3 and then combining it, try doing it sequentially. So kind of like this:

1. Take image 1 and 2, compute their matching faetures.
2. Compute a Homography relating image 2 to image 1.
3. Warp image 2 using this Homography to stitch it with image 1. This gives you an intermediate result 1.
4. Take this intermediate result 1 as first image and repeat step 1-3 for next image in the sequence.

A good blog post about the same to get started is here: https://kushalvyas.github.io/stitching.html

To compare your algorithm performance, you can see the result from opencv-stitcher class: