Suppose we have finite data set {x_i, y_i}.
I am looking for an efficient data structure for the data set, such that given a,b it will be possible to find efficiently x,y such that x > a, y > b and x*y is minimal.
Can it be done using a red black tree ?
Can we do it in complexity O(log n)?
Well, without a preprocessing step, of course you can't do it in O(log n) with any data structure, because that's not enough time to look at all the data. So I'll assume you mean "after preprocessing".
Red-black trees are intended for single-dimension sorting, and so are unlikely to be of much help here. I would expect a kD-tree to work well here, using a nearest-neighbor-esque query: You perform a depth-first traversal of the tree, skipping branches whose bounding rectangle either violates the x and y conditions or cannot contain a lower product than the lowest admissible product already found. To speed things up further, each node in the kD-tree could additionally hold the lowest product among any of its descendants; intuitively I would expect that to have a practical benefit, but not to improve the worst-case time complexity.
Incidentally, red-black trees aren't generally useful for preprocessed data. They're designed for efficient dynamic updates, which of course you won't be doing on a per-query basis. They offer the same asymptotic depth guarantees as other sorted trees, but with a higher constant factor.