Search code examples
amazon-web-servicesterraformaws-glueterraform-provider-aws

How to update the Source Code as well with terraform apply?


I have a terraform script to deploy an AWS Glue Job with it's sourcecode (ET script). When I run terraform apply it's updating the infrastructure. But not the source code. What should I do to update the code as well??

Folder structure.

--glue-script-tf.py
--main.tf

Below is my terraform script(main.tf).

provider "aws" {
  region     = "us-east-1"
}

variable "job-language" {
  default = "python"
}
    
variable "bucket-name" {
  default = "glue-poc"
}

variable "glue-role-arn" {
  default = "arn:aws:iam::#####:role/dp-da-glue-poc-role-dev"
}

variable "job-name" {
  default = "glue-job-from-terraform"
}

variable "file-name" {
  default = "glue-script-tf.py"
}

#===========================================
    
resource "aws_s3_bucket_object" "upload-glue-script" {
  bucket = "${var.bucket-name}"
  key    = "scripts/${var.file-name}"
  source = "${var.file-name}"
}
    
resource "aws_glue_job" "glue-job" {
  name        = "${var.job-name}"
  role_arn    = "${var.glue-role-arn}"
  description = "This is script to create large files from small files "
  max_retries = "1"
  timeout     = 2880
  command {
    script_location = "s3://${var.bucket-name}/scripts/${var.file-name}"
    python_version = "3"
  }
  execution_property {
    max_concurrent_runs = 2
  }
  glue_version = "4.0"
}

Solution

  • What you could do (depending on the terraform version you are using) is to use the filemd5 built-in function with the etag argument:

    (Optional) Triggers updates when the value changes. The only meaningful value is filemd5("path/to/file") (Terraform 0.11.12 or later) or ${md5(file("path/to/file"))} (Terraform 0.11.11 or earlier).

    So, if you are using newer terraform versions, you could do the following:

    resource "aws_s3_bucket_object" "upload-glue-script" {
      bucket = var.bucket-name
      key    = "scripts/${var.file-name}"
      source = var.file-name
      etag   = filemd5("${var.file-name}") # or ${md5(file("${var.file-name}"))}
    }
    

    Note also that if the file is bigger than 16MB, you should default to using source_hash argument (it also helps if you enable encryption with SSE-KMS).