Instance not refreshing after launch configuration update terraform

Question

I am trying to update launch configuration user-data. But after apply, launch config is getting created and updated ASG. but running instance are still with old user-data. why so ?

Below are the launch config and ASG blocks.

resource "aws_launch_configuration" "BackEndWebLaunchConfig" {
        name_prefix             = "${var.component_name}-BackEndWebLaunchConfig"
        user_data               = file("user_data/${terraform.workspace}/vision-be-user-data.sh")
        image_id                = var.ASLCWEBAPPSAMI
        instance_type           = var.ASGWebAppsInstanceType
        key_name                = var.ssh_key_name
        security_groups         = [module.vpc.sgssh, aws_security_group.vision_backend_EC2SG.id]
        root_block_device  {
            volume_size         = var.EC2_EBS_SIZE
            volume_type         = "standard"
            encrypted           = true
        }
        #iam_instance_profile    = var.EC2_instance_profile
        associate_public_ip_address = false 
        lifecycle       { 
            create_before_destroy = true 
            }
        
}

resource "aws_autoscaling_group" "vision_asg" {
        name                      = "${var.component_name}-BackEnd-ASG-TF"
        max_size                  = var.ASGWEBAPPSMaxSize
        min_size                  = var.ASGWEBAPPSMinSize
        health_check_grace_period = 300
        force_delete              = true
        health_check_type         = "ELB"
        desired_capacity          = var.ASGWEBAPPSDesiredSize
        launch_configuration      = aws_launch_configuration.BackEndWebLaunchConfig.name
        target_group_arns         = [module.loadbalancer.visionalb_ext_tg_arn]  
        vpc_zone_identifier       = [module.vpc.PrivateSubnet0, module.vpc.PrivateSubnet1]
        termination_policies      = ["OldestInstance"]
        lifecycle       { 
            create_before_destroy = true 
            }
        tags                      = [
            {
                key                 = "Name"
                value               = "${var.component_name}-BackEndWebASG-TF"
                propagate_at_launch =  true
            },
            {
                key                 = "component"
                value               = var.component_name
                propagate_at_launch =  true
            },
                        {
                key                 = "tier"
                value               = "web"
                propagate_at_launch =  true
            }
            ]
        depends_on = [
          aws_sns_topic.BackEndSNSTopic, aws_launch_configuration.BackEndWebLaunchConfig
        ]

}

After apply, new launch config is creating but ec2 machines are not refreshing.

aws_autoscaling_group.vision_asg: Refreshing state... [id=BackEnd-ASG-TF]
aws_autoscaling_policy.BEWebScaleUpPolicy: Refreshing state... [id=BEWebScaleUpPolicy]
aws_autoscaling_notification.vision_asg_notification: Refreshing state... [id=arn:aws:sns:us-east-1:193676128801:BackEndApplication-TF]
aws_autoscaling_policy.BEWebScaleDownPolicy: Refreshing state... [id=BEWebScaleDownPolicy]
aws_cloudwatch_metric_alarm.BEScaleDownNotifyAlarm: Refreshing state... [id=BEScaleDownNotifyAlarm]
aws_cloudwatch_metric_alarm.ScaleUPNotifyAlarm: Refreshing state... [id=ScaleUPNotifyAlarm]
aws_launch_configuration.BackEndWebLaunchConfig: Creating...
aws_launch_configuration.BackEndWebLaunchConfig: Creation complete after 8s [id=BackEndWebLaunchConfig20210508105416185400000001]
aws_autoscaling_group.vision_asg: Modifying... [id=BackEnd-ASG-TF]
aws_autoscaling_group.vision_asg: Modifications complete after 4s [id=BackEnd-ASG-TF]
aws_launch_configuration.BackEndWebLaunchConfig (2530c36e): Destroying... [id=BackEndWebLaunchConfig20210508103324724600000001]
aws_launch_configuration.BackEndWebLaunchConfig: Destruction complete after 2s

Apply complete! Resources: 1 added, 1 changed, 1 destroyed.

Outputs:

Please let me know, if i am doing something wrong.

Answer 1

This question is answered in the AWS documentation on Changing the launch configuration for an Auto Scaling group, which says:

After you change the launch configuration for an Auto Scaling group, any new instances are launched using the new configuration options, but existing instances are not affected. To update the existing instances, terminate them so that they are replaced by your Auto Scaling group, or allow automatic scaling to gradually replace older instances with newer instances based on your termination policies.

To perform the instance refresh during terraform apply, you could do the following:

1. Upgrade your Terraform AWS Provider to at least 3.22.0.
2. Add an instance_refresh block to your aws_autoscaling_group resource. Here is an example, provided from their docs:

 instance_refresh {
    strategy = "Rolling"
    preferences {
      // You probably want more than 50% healthy depending on how much headroom you have
      min_healthy_percentage = 50
    }
    // Depending the triggers you wish to configure, you may not want to include this
    triggers = ["tag"]
  }

The associated documentation states:

A refresh will always be triggered by a change in any of launch_configuration, launch_template, or mixed_instances_policy.

One final note to think about. Depending on your use case, you may prefer to control the cadence of instance restarts separately from terraform apply. In some projects, we have avoided this so that we can run terraform apply in CI/CD with less apprehension about swapping the machines running our production workloads at an inopportune time. Be aware that terraform can currently trigger the instance refresh, but it will not monitor the instance refresh for success or failure.