STS credentials for calling Sagemaker InvokeEndpoint do not work despite the fact that IAM credentials generated from the same policy do work

Question

I'm trying to make a call to a Sagemaker endpoint using credentials generated through STS, but am receiving a permission error:

botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the InvokeEndpoint operation: User: arn:aws:sts::{REDACTED_AWS_ACCOUNT_ID}:federated-user/{REDACTED_USER_SUFFIX} is not authorized to perform: sagemaker:InvokeEndpoint on resource: arn:aws:sagemaker:us-west-2:{REDACTED_AWS_ACCOUNT_ID}:endpoint/{REDACTED_ENDPOINT} because no identity-based policy allows the sagemaker:InvokeEndpoint action

The strange part is that if use the same policy to generate IAM credentials (rather than STS credentials), I am able to call the Sagemaker endpoint without issue.

I've double checked the policy and it looks correct (and does in fact work for the IAM case, as mentioned above). Here's the policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "sagemaker:InvokeEndpoint"
      ],
      "Resource": [
        "arn:${ .aws_partition }:sagemaker:${ .aws_region }:${ .aws_account_id }:endpoint/${ .endpoint }",
      ]
    }
  ]
}

I've also checked the trust relationship on the role that is being assumed during the STS credential generation, and it also appears correct. One thought I had is that we are using Vault to proxy STS credential generation requests, and that perhaps the trust relationship needs to specify Vault? But I would be surprised because I don't think I've had to do that in the past when generating STS credentials for other AWS services. Here is the trust relationship:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "Service": "sagemaker.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

What could possibly be causing the STS credentials to fail while the IAM credentials work?

Answer 1

The answer may depend if you're using AssumedRole or Federation Token modes of STS, but the root of the issue is that the IAM user which is used to generate STS credentials does not have a policy with sagemaker:* permissions either it can assume (Assumed Role) or attached to the role (Federation Token).

See https://developer.hashicorp.com/vault/docs/secrets/aws#sts-federation-tokens for more detail.