I have a HCP terraform workspace set to "remote" execution mode. I figured from this GitHub issue that I need to set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY env vars in the HCP terraform workspace as the terraform commands are running from a remote container/VM somewhere that does not have access to the env vars available in my local environment.
But I also notice that when I run terraform commands with TF_LOG env variable set to some value in my local system, it actually honours that env var and starts printing trace logs.
Question: How does terraform know that TF_LOG env var is set if it's not sending the env variable to the machine where it's actually executing? Is there a list if env vars that TF sends to remote machine? Any documentation explaining this behaviour?
When you start a HCP Terraform remote operation from Terraform CLI, there are two different Terraform CLI processes running: the one on your local computer, and the one running in the HCP Terraform remote execution environment.
The local Terraform CLI can only "see" the environment variables you've set on your local computer, while the remote Terraform CLI can only "see" environment variables specified in the workspace settings.
The local Terraform CLI does interpret some environment variables on your local system and translate them into something that HCP Terraform can make use of. For example, if you have set TF_VAR_... environment variables to specify values for input variables in your root module then local Terraform CLI is the one responsible for deciding which input variable values to send to the HCP Terraform API, and so the environment variables on your local system can be honored. However, remote Terraform CLI obtains those variables via HCP Terraform's automatically-generated terraform.tfvars file in the remote execution environment, not by interpreting the environment variables directly itself.
But the AWS provider is the one that interprets AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. The AWS provider is only running beneath the remote Terraform CLI, so it has no awareness of any environment variables from your local Terraform CLI process.
On your question about TF_LOG in particular: I'm not 100% sure, but I suspect when you are turning that on you are receiving only the subset of log information generated by your local Terraform CLI -- the information about Terraform CLI starting up and then calling the HCP Terraform API to start the remote run -- whereas the remote Terraform CLI process does not know that variable is set on your local system and so any log messages generated by the remote Terraform CLI, or any provider running beneath it, would not be included.