I've been added in a couple of organizations on Google Play Store Developer Console but I can't get in touch with the admins, how can I leave those organizations?
It is now possible.
Go to Users and permissions then click on Manage on the right side of the row showing your email address. Then on the next page click on Remove user.