Find a good homography from different point of view of objects?

Question

I am doing object detection using feature extraction (sift,orb).

I want to extract ORB feature from different point of view of the object (train images) and then matching all of them with a query image.

The problem I am facing is: how can I create a good homography from keypoint coming from different point of view of the image that have of course different sizes?

Edit

I was thinking to create an homography for each train images that got say 3-4 matches and then calculate some "mean" homography...

The probleam arise when you have for example say just 1-2 matches from each train image, at that point you cannot create not even 1 homography

Code for create homography

  //> For each train images with at least some good matches ??
  H = findHomography( train, scene, CV_RANSAC );
  perspectiveTransform( trainCorners, sceneCorners, H);

Answer 1

I think there is no point on doing that as a pair of images A and B has nothing to do with a pair of images B and C when you talk about homography. You will get different sets of good matches and different homographies, but homographies will be unrelated and no error minimization would have a point.

All minimization has to be within matches, keypoints and descriptors considering just the pair of images.

There is an idea similar to what you ask in FREAK descriptor. You can train the selected pairs with a set of images. That means that FREAK will decide the best pattern for extracting descriptors basing on a set of images. After this training you are supposed to find more robust mathces that will give you a better homography.