I have deployed AWS EKS cluster with Spot instances using Terraform. I am using EKS managed Node Groups, as for spot instances, I want to specify multiple instance types per AZ for HA. As I can see, ASG created by EKS for us has it's launch template created, which we should not edit(as they mentioned in aws doc). I wanted to ask, is there any way I can update this "multiple instances per AZ" in my cluster. Or do I need to configure custom ASG for it? Should I rather create ASG per AZ with multiple instance types for better availability of space capacity?
I am new to EKS Spot, please let me know if any more input is needed.
Please refer CAS configuration :
resource "helm_release" "cluster-autoscaler" {
name = local.app
namespace = var.namespace
repository = local.cluster_autoscaler_helm_repository
chart = var.cluster_autoscaler_helm_chart
version = var.cluster_autoscaler_helm_version
values = [
yamlencode({
autoDiscovery = {
clusterName = var.cluster_name
}
awsRegion = var.region
extraArgs = {
scan-interval = var.scan_interval
expander = "least-waste"
skip-nodes-with-local-storage = false
skip-nodes-with-system-pods = false
}
extraVolumes = [
{
name = "ssl-certs"
hostPath = {
path = "/etc/ssl/certs/ca-bundle.crt"
}
}
]
extraVolumeMounts = [
{
name = "ssl-certs"
readOnly = true
mountPath = "/etc/ssl/certs/ca-certificates.crt"
}
]
image = {
repository = local.cluster_autoscaler_image
tag = var.cluster_autoscaler_image_version
}
podAnnotations = {
"cluster-autoscaler.kubernetes.io/safe-to-evict" = "false"
}
podLabels = {
app = local.app
}
rbac = {
serviceAccount = {
annotations = {
"eks.amazonaws.com/role-arn": "arn:aws:iam::${var.account_number}:role/cluster-autoscaler"
}
}
}
replicaCount = var.replica_count
# resources -- Pod resource requests and limits.
resources = {
limits = {
cpu = var.resources_limit_cpu
memory = var.resources_limit_memory
}
requests = {
cpu = var.requests_limit_cpu
memory = var.requests_limit_memory
}
}
})]
}
For configuring spot allocation startegy, I found in this document that, for EKS Managed Node Groups, if I am using capacity_type = "SPOT"
then by default spotAllocationStrategy="capacity-optimized"
is assigned. So basically, my purpose is served here.