Search code examples
image-processingdeep-learningvideo-processingsimilaritysemantic-segmentation

Aligning two extremely similar but slightly different videos

I have two videos of the Super Smash Brothers video game. In one video, the characters exist. In the other video, the characters do not exist. Everything else about the videos are the exact same except the characters being invisible in one of them.

When I output the two videos, I have to manually align them in a video editor. Once they are aligned, they stay in sync! However, the videos have a random amount of start time which is the problem.

What's a good way to automatically align these two different but extremely similar videos? Here are example frames.

enter image description here

without characters

Current ideas: Take a random frame halfway through the video, compare to the other video at the same location. Use Mean Squared Error between the pixels. Move forward 5 seconds and back 5 seconds. Take the frame with the smallest MSE difference and use that as the matching frame remove the offset from the longer video at the beginning. This seems extremely brittle and slow.

Solution

  • Your current idea is good, but it doesn't need to be slow at all. Since the different part of the images are only the fighters! and we can assume those fighters are always at the middle of the image, so you just need to match a little part of the images, like the rectangle I drew:

    enter image description here

    Besides you can use other fast matching methods too like ORB features.