I have a Terraform step that calls out to a bash script and creates a directory as part of a pipeline, inserts some files in that directory, and successfully uploads that material to AWS but only the first time.
#!/bin/bash
echo "Executing create_pkg.sh ..."
cd $path_cwd
dir_name=python_dependencies_folder/python
mkdir -p $dir_name
# Put some files in this directory
It is called from this section of Terraform:
resource "null_resource" "install_python_dependencies" {
provisioner "local-exec" {
command = "cd scripts && chmod +x ./create_pkg.sh && ./create_pkg.sh"
interpreter = ["bash", "-c"]
working_dir = path.module
environment = {
path_module = path.module
path_cwd = path.cwd
}
}
}
It is zipped up through this data section:
data "archive_file" "zip_creation" {
depends_on = [null_resource.install_python_dependencies]
type = "zip"
source_dir = "${path.cwd}/python_dependencies_folder"
output_path = var.output_path
}
The first time this process runs, everything executes correctly. However, if the same process is executed a second time, I see the following in the job logs:
module.aws_lambda_function.null_resource.python_dependencies_folder[0]: Refreshing state... [id=63624526464556] module.data.archive_file.zip_creation[0]: Reading ...
Error: Archive creation error
with module.data.archive_file.zip_creation[0] on .terraform/modules/aws_lambda_function/main.tf line 98, in data "archive_file" "zip_creation": 98: data "archive_file" "zip_creation" {
error creating archive: error archiving directory: could not archive missing directory: .terraform/modules/aws_lambda_function/python_dependencies_folder
As I stated, when the same pipeline runs a second time, it is almost like TF is skipping the execution of "null_resource" due to some entry in the tfstate file.
Thoughts on how I indicate to Terraform that the "null_resource" must be run each time and ignore what it finds in tfstate?
null_resource operates the standard resource lifecycle. as documented in the provider
The null_resource resource implements the standard resource lifecycle but takes no further action. The triggers argument allows specifying an arbitrary set of values that, when changed, will cause the resource to be replaced.
That is to say when your first run TF the resource will be executed and its resource state persisted. So when you run a second time the null resource won't be executed because as far as it knows nothing has changed with the resource so it won't run it again.
Your archive_file data object will be run on every execution. However on your second execution it fails since your null resource didn't run again. I am making an assumption here that you are running this each time in either Docker or some sort of ephemeral environment where the artifact produced by your null resource is not persisted and available on each run of your Terraform.
You can add a triggers argument to your null resource to tell terraform when it needs to rerun this null resource. In this case if you need the null resource to run every time you can add a triggers tag using the timestamp which will be different on every run so will trigger the resource every run. However personally that feels a bit wrong to me and will result in your TF output always saying it will destroy and add the null resource every time.
resource "null_resource" "install_python_dependencies" {
provisioner "local-exec" {
command = "cd scripts && chmod +x ./create_pkg.sh && ./create_pkg.sh"
interpreter = ["bash", "-c"]
working_dir = path.module
environment = {
path_module = path.module
path_cwd = path.cwd
}
}
triggers = {
ts = timestamp()
}
}
In reality it sounds like your zip file should be created as an artifact and persisted and pulled from an artifact repository our created outside of TF its self. However I feel this pain, as I have experienced this exact same issue when packaging a Python lambda for AWS and uploading via TF and trying to do it all in TF. In reality TF is not the solution to everything. I would really recommend making the zip file an artifact outside of the TF process.