From the Google Doc, I see below are the options to fetch an access token for the directory API
Reference Doc: https://developers.google.com/identity/protocols/oauth2
Is there any API to Fetch an access token from SAML bearer assertion for the directory API and if yes then what information does this API needs and is there any Doc to do so
Example Directory API for which I am looking to connect - Directory API used for user creation: https://www.googleapis.com/admin/directory/v1/users
Unfortunately, at current Google doesn't support client authorization and authentication for their APIs via SAML 2.0 Profiles.
As you've already pointed out, the only supported methods of fetching an access token at the present time are those mentioned in the documentation page titled "Using OAuth 2.0 to Access Google APIs".
You can however let Google know that this is a feature that is important for access to their APIs, and that you would like to request they implement it.
Google's Issue Tracker is a place for developers to report issues and make feature requests for their development services, I'd urge you to make a feature request there. While there is no component for general OAuth2 or Authentication methods, given your requirements I would suggest filing it under the Admin SDK component which you can find here