How to run python code inside terraform enterprise cloud?

I'm attempting to run python code like so in terraform cloud. However, the build environment my script is attempting to run from doesn't have python installed...

data "external" "example" {
  program = ["python", "${path.module}/example-data-source.py"]

  query = {
    # arbitrary map from strings to strings, passed
    # to the external program as the data query.
    id = "abc123"
  }
}

I get this error message...

│ Program: /bin/bash
│ Error Message: ./modules/kinesis/build-glue-schema.sh: line 14: python:
│ command not found
│
│ State: exit status 127

Is there any way to install python into the terraform cloud build environment? I can see there is a python provider here but I want to be able to use data.external block.

Does anyone have any advice as to how to get this working?

Solution

You can read about the context where Terraform runs in Terraform Cloud under Terraform Cloud's Run Environment.

In particular for your question The Terraform Worker VMs is relevant:

Terraform Cloud performs Terraform runs in single-use Linux virtual machines, running on an x86_64 architecture.

The operating system and other software installed on the worker VMs is an internal implementation detail of Terraform Cloud. It is not part of a stable public interface, and is subject to change at any time.

Before Terraform is executed, the worker VM's shell environment is populated with environment variables from the workspace, the selected version of Terraform is installed, and the run's Terraform configuration version is made available.

Changes made to the worker during a run are not persisted to subsequent runs, since the VM is destroyed after the run is completed. Notably, this requires some additional care when installing additional software with a local-exec provisioner; see Installing Additional Tools for more details.

Therefore you cannot rely on any software other than Terraform itself -- including Python -- to be available in the remote operations environment.

The best option for custom behavior in the Terraform Cloud remote operations environment is to develop a Terraform provider plugin which performs the action you wish to perform, and then publish that plugin either in the public Terraform Registry or in Terraform Cloud's private registry.

Alternatively, you can disable remote operations and run Terraform CLI on your own infrastructure where you can control what software is installed, while still using Terraform Cloud for other functions such as state storage.

Note that the joaqquin89/python you found is really just a wrapper around running the python executable, just like you are doing with the external data source in your example: https://github.com/joaqquin89/terraform-provider-python/blob/639caf60f9c28c9254669499d639309da17e287d/python/resource_python.go#L47-L65

Therefore this provider is not sufficient to get the result you want. A provider for running Python would need to somehow embed the Python interpreter in its own distribution package, either by linking it into the provider plugin executable itself or by including it as a separate file alongside the plugin executable.