Is it possible to have the same in-app product(managed) in multiple apps? Such that if a user buys a premium upgrade in one of the apps, he gets premium upgrade in all of them.
Note that I'm not planning to maintain a user database currently and trying to find a solution using Google's in-app billing API alone if it's possible.
No, it's not possible, unless you build your own database. Entitlements are always related to an app (package name), which means if a user buys a SKU of App A, the Play Billing library can only pull entitlements for App A from its code.