I am using the following lines of code in my VF page to open the URL & it's opening in the same tab but I need to open a new Tab as soon as clicking the custom URL Button?
<apex:commandButton action="{!URLFOR($Action.My_Obj__c.My_custom_link)}" value="My custom button"/>
I don't think I have ever seen a button that opens something as new tab/window? Link yes, but not button. Think if that's really what you want, user might be surprised with unexpected behaviour.
Try using <apex:commandLink action="{!URLFOR($Action.My_Obj__c.My_custom_link)} target="_blank">. You could style it to look like button by looking at class names (both in Classic and Lightning Experience).
If that doesn't work - worst case you can always capture onclick and do window.open or whatever from there...