TL;DR How can we automate the creation of per-user credentials to access COS buckets using the S3 compatibility layer?
The only documented way AFAIK is the CLI; but embedding command-line executions inside a web service is not very appealing!
The IBM online documentation for COS (Cloud Object Storage) shows 2 ways to create a "service key with HMAC credentials" for a given bucket and a given role [reference]
ibmcloud resource service-key-create ****** --parameters '{"HMAC":true}'The point of these "HMAC creds" is to access the bucket as if it was stored in S3 (or any other appliance compatible with the S3 API).
Now, I assume the UI and the CLI both use some kind of API under the covers. Maybe they have to juggle with the IAM API and the COS API to expose that COS-specific feature as a "generic" security feature.
But I couldn't find anything relevant in the online docs for IAM API -- the only API call that specifies a "resource" and an "authorization policy" is about creating a temporary token, not a persistent key [reference]
I couldn't find anything relevant in the COS API either. Nothing about security there.
And I would try to dig into the source code of the CLI if it was Open Source but, alas, it is not.
I guess my only hope is to get "inside information" from the IBMers who actually developed the CLI... hence the tags below.
So when you create a Service Credential with HMAC keys, what is essentially happening there is that those persistent keys are being generated by COS (not IAM) and are bound to an IAM Service ID (ie a non-human user). When you send an S3 API request by creating a signature with those keys, COS figures out the associated Service ID and checks with IAM to see if that identity has sufficient privileges for the operation being requested. This allows you to use the persistent HMAC keys at the expense of needing to waste some CPU calculating all the HMAC signature stuff for each request. So while IAM is involved behind the scenes at the policy enforcement level, authentication is handled by COS directly.
Now, if you want to programmatically create Service Credentials, there's a REST API for that, but no published libraries, (although I suppose you could try and generate one from the OpenAPI spec). Essentially you just call that API and provide the {"HMAC":true} bit inside the "Parameters" object. Inside the response you'll find the credential with the HMAC keys.
All that being said, you don't want to use IAM tokens or HMAC signatures for your application's end users. Think about the Service ID that you've bound to the HMAC credentials as the executor of the operations being performed. Try and keep the user authentication piece isolated, and then when some end user is allowed to do something, then have the Service ID do the operation. Obviously all applications are going to be different and have their own challenges. One pattern that we've seen used (and I keep meaning to put together a tutorial for) is to have the application generate a pre-signed URL, and pass that to the client, avoiding any need to pass credentials outside of the application space.
Basically, IAM is intended for administrators, developers, and other cloud services to authenticate and authorize actions in the IBM Cloud (and thus aren't intended to scale past a few thousand access policies), but you should be looking to something like App ID for the end-user authz pieces.
What language/framework are you using? Are you using the COS SDKs? They abstract away the token management stuff for you if you'd rather use API keys.