Is it possible to create your own bucket with images where you can add extra information for each image. Then use Google Cloud Vision to search like they do now but as an extra also search the image in your bucket?
Reason: I have some images that, when I search them with Google Cloud Vision, return almost no text. For this reason I would add these type of images to a bucket and manually add more information. The next time a user takes a photo of the same thing, it needs to search inside this bucket and if found return the extra information about this image.
You can try out the product search api on vision api: https://cloud.google.com/vision/product-search/docs/how-to
It allows you to build your own image catalog and each image can be annotated with image metadata.
And then you can use image similarity search of the API to achieve what you want.
See a video tutorial here as well https://www.youtube.com/watch?v=BOdUYTO1jJY&t=539s
To manage image catalog they provides a bunch of APIs and also you can use a the batch processing API