How can I use AMA custom authentication with OAuth service provider?
Perhaps, the AMA doc of Implementing Bluemix Single Sign On as a custom in identity provider. in Getting started could help this. But this doc has deleted.
Unfortunately, we do not support Single Sign On.
Using Advanced Mobile Acces(AMA) custom authentication involves 2 steps.
In the first step, you have to configure the authentication mechanism to access the backend services and this is done in the AMA dashboard of the backend application. Please see the sub-section Configuring an authentication mechanism for back end access in the document Advanced Mobile Access Configuration page section Registering your Advanced Mobile Access Client App for details on configuring the custom authentication mechanism. Basically, you will be providing the url of the custom identity provider page that will handle authentication challenge.
In the second step, you have to configure the client app to use custom authentication and please see the document section Using a custom identity provider for procedure to do it.