InvalidClientTokenId when using Terraform apply with two different providers configuration

Question

I am getting this error as I am providing an EKS cluster in a VPC with Terraform. The key point is that I am providing the VPC with an IAM user and the EKS with another created within the same Terraform session. Right or wrong doesn't matter, at this point I am only trying to understand the issue. I have done some research here and on the web and tried various solutions, but it seems to me that none of the scenarios apply to my original configuration with two distinct AWS providers.

This is the set of instructions. A note: the providers are in a separate providers.tf only reported here in a single list for simplicity.

provider "aws" {
  alias = "aws-milano"
  region = local.region
  profile = local.terraform_profile
}

module "aws-jupyterhub-vpc" {
  source = "./modules/aws-jupyterhub-vpc"
  # parameters....
  providers = {
    aws = aws.aws-milano
  }
}

module "aws-jupyterhub-iam" {
  depends_on = [ module.aws-jupyterhub-vpc ]
  source = "./modules/aws-jupytherhub-iam"

  providers = {
    aws = aws.aws-milano
  }
}

provider "aws" {
  alias = "aws-milano-eks"
  region = local.region

  # exposed from module
  access_key = module.aws-jupyterhub-iam.cluster_profile_key
  secret_key = module.aws-jupyterhub-iam.cluster_profile_secret
}

module "aws-jupyterhub-eks" {
  source = "./modules/aws-jupyterhub-eks"
  depends_on = [ module.aws-jupyterhub-vpc, module.aws-jupyterhub-iam]
  # parameters...
  providers = {
    aws = aws.aws-milano-eks
  }  
}

But then I am having this error with terraform apply whenever it starts the EKS module:

│ Error: validating provider credentials: retrieving caller identity from STS: operation error STS: GetCallerIdentity, https response error StatusCode: 403, RequestID: xxxxxxxxxx, api error InvalidClientTokenId: The security token included in the request is invalid.

However if I terraform apply a second time, with Terraform only provisioning the rest of the infrastructure, it works without problems and the infrastructure is fully provisioned.

My guess is that Terraform inherit some environment variables that come into the picture of building the secret access token. In fact if I run again from scratch, it works.

Answer 1

After some investigation, I found this fragment on the provider configuration page on Hashicorp:

You can use expressions in the values of these configuration arguments, but can only reference values that are known before the configuration is applied. This means you can safely reference input variables, but not attributes exported by resources (with an exception for resource arguments that are specified directly in the configuration).

So, as I was interested in what was going on this is the reason why.

The provider is instantiated at runtime but it's unable to retrieve the value that derives from the resource block and they are maybe populated with null values:

module "aws-jupyterhub-iam" {
  depends_on = [ module.aws-jupyterhub-vpc ]
  source = "./modules/aws-jupytherhub-iam"

  providers = {
    aws = aws.aws-milano
  }
}

provider "aws" {
  alias = "aws-milano-eks"
  region = local.region

  # exposed from module
  access_key = module.aws-jupyterhub-iam.cluster_profile_key
  secret_key = module.aws-jupyterhub-iam.cluster_profile_secret
}

and this is clearly in violation of the spec.

Then, why does at second attempt everything run fine ? This is because the module "aws-jupyterhub-iam" has already finished its job since long and has already exported the variables which are now in terraform state where the provider can safely find them.