Azure AKS - oms agent AND diagnostic settings possible together?

Question

I'm deploying an AKS cluster via Terraform.

I set an oms_agent block within my aks resource block:

resource "azurerm_kubernetes_cluster" "tfdemo-cluster" {
  resource_group_name               = var.resourcegroup_name
  location                          = var.location
  name                              = "${var.projectname}-aks"
  node_resource_group               = "${var.resourcegroup_name}-node"
  ... omitted to shorten ...
  
  oms_agent {
    log_analytics_workspace_id = var.log_analytics_workspace_id
  }

Like this it works as aspected.

But when I add an additional resource of type diagnostic_settings like so

resource "azurerm_monitor_diagnostic_setting" "aks-diagnostics" {
  name = "aks-logs"
  storage_account_id = var.storage_account_id
  target_resource_id = azurerm_kubernetes_cluster.tfdemo-cluster.id

  log {
    category = "kube-audit"
    enabled  = true
  }

  metric {
    category = "AllMetrics"
    retention_policy {
      days    = 30
      enabled = true
    }
  }
}

I run into an error that says:

"diagnosticsettings.DiagnosticSettingsClient#CreateOrUpdate: Failure sending request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=nil nil"

When I tried to google that error messages I found issues related to other Azure services where the sku of that service wasn't matching a specified feature or capacity but I'm don't see that here.

Why I want log analytics workspace AND logs dumped into a storage account: My thinking was just that a log anal. ws is really expensive compared to storage in a storage account. So I thought I send say the audit data for long time retention to the cheap storage account (my settings in the given example might not 100% represent that but it's not the point here I'd say) and still have the "expensive" log analytics service to dig into the cluster performance.

Thanks a lot for any input!

Answer 1

I Tried to reproduce the same in my environment to Create an Azure AKS cluster with OMS Agent and Diagnostic Setting using Terraform:

Sending long-term data retention logs to a Azure Storage Account can be more cost-effective than keeping them in a Azure Log Analytics workspace. However, the Azure Log Analytics workspace can still be useful for real-time analysis and performance monitoring.

    provider "azurerm" {
  features {}
}
resource "azurerm_resource_group" "aksgroup" {
  name     = "aks-rg"
  location = "East US"
}

resource "azurerm_log_analytics_workspace" "oms" {
  name                = "oms-workspace"
  location            = azurerm_resource_group.aksgroup.location
  resource_group_name = azurerm_resource_group.aksgroup.name
  sku                 = "PerGB2018"
}

resource "azurerm_kubernetes_cluster" "aks" {
  name                = "cluster-aks1"
  location            = azurerm_resource_group.aksgroup.location
  resource_group_name = azurerm_resource_group.aksgroup.name
  dns_prefix          = "aks1"

  default_node_pool {
    name       = "default"
    node_count = 1
    vm_size    = "standard_a2_v2"
  }

  identity {
    type = "SystemAssigned"
  }

  tags = {
    Environment = "Production"
  }
  addon_profile {
      oms_agent {
        enabled                    = true
        log_analytics_workspace_id = azurerm_log_analytics_workspace.oms.id
      }
    }
}

output "client_certificate" {
  value     = azurerm_kubernetes_cluster.aks.kube_config.0.client_certificate
  sensitive = true
}

output "kube_config" {
  value = azurerm_kubernetes_cluster.aks.kube_config_raw

  sensitive = true
}

resource "azurerm_monitor_diagnostic_setting" "aks" {
  name                 = "aks-diagnostic-setting"
  target_resource_id   = azurerm_kubernetes_cluster.aks.id
  storage_account_id   = azurerm_storage_account.aks.id
  log_analytics_workspace_id = azurerm_log_analytics_workspace.oms.id
  log {
    category = "kube-audit"
    enabled  = true
  }
  metric {
    category = "AllMetrics"
    retention_policy {
      days    = 30
      enabled = true
    }
  }
}

resource "azurerm_storage_account" "aks" {
  name                = "aksdiagnostic"
  resource_group_name = azurerm_resource_group.aksgroup.name
  location            = azurerm_resource_group.aksgroup.location
  account_tier        = "Standard"
  account_replication_type = "LRS"
}

Terraform Apply:

Once ran the code resources are created, like below.

Azure AKS Diagnostic settings created with Log Analytics settings.

Log Analytics settings- created.