I have a terraform script to deploy an AWS Glue Job with it's sourcecode (ET script). When I run terraform apply
it's updating the infrastructure. But not the source code. What should I do to update the code as well??
Folder structure.
--glue-script-tf.py
--main.tf
Below is my terraform script(main.tf).
provider "aws" {
region = "us-east-1"
}
variable "job-language" {
default = "python"
}
variable "bucket-name" {
default = "glue-poc"
}
variable "glue-role-arn" {
default = "arn:aws:iam::#####:role/dp-da-glue-poc-role-dev"
}
variable "job-name" {
default = "glue-job-from-terraform"
}
variable "file-name" {
default = "glue-script-tf.py"
}
#===========================================
resource "aws_s3_bucket_object" "upload-glue-script" {
bucket = "${var.bucket-name}"
key = "scripts/${var.file-name}"
source = "${var.file-name}"
}
resource "aws_glue_job" "glue-job" {
name = "${var.job-name}"
role_arn = "${var.glue-role-arn}"
description = "This is script to create large files from small files "
max_retries = "1"
timeout = 2880
command {
script_location = "s3://${var.bucket-name}/scripts/${var.file-name}"
python_version = "3"
}
execution_property {
max_concurrent_runs = 2
}
glue_version = "4.0"
}
What you could do (depending on the terraform version you are using) is to use the filemd5
built-in function with the etag
argument:
(Optional) Triggers updates when the value changes. The only meaningful value is
filemd5("path/to/file")
(Terraform 0.11.12 or later) or${md5(file("path/to/file"))}
(Terraform 0.11.11 or earlier).
So, if you are using newer terraform versions, you could do the following:
resource "aws_s3_bucket_object" "upload-glue-script" {
bucket = var.bucket-name
key = "scripts/${var.file-name}"
source = var.file-name
etag = filemd5("${var.file-name}") # or ${md5(file("${var.file-name}"))}
}
Note also that if the file is bigger than 16MB, you should default to using source_hash
argument (it also helps if you enable encryption with SSE-KMS).