Azure AKS - oms agent AND diagnostic settings possible together?

I'm deploying an AKS cluster via Terraform.

I set an oms_agent block within my aks resource block:

resource "azurerm_kubernetes_cluster" "tfdemo-cluster" {
  resource_group_name               = var.resourcegroup_name
  location                          = var.location
  name                              = "${var.projectname}-aks"
  node_resource_group               = "${var.resourcegroup_name}-node"
  ... omitted to shorten ...
  
  oms_agent {
    log_analytics_workspace_id = var.log_analytics_workspace_id
  }

Like this it works as aspected.

But when I add an additional resource of type diagnostic_settings like so

resource "azurerm_monitor_diagnostic_setting" "aks-diagnostics" {
  name = "aks-logs"
  storage_account_id = var.storage_account_id
  target_resource_id = azurerm_kubernetes_cluster.tfdemo-cluster.id

  log {
    category = "kube-audit"
    enabled  = true
  }

  metric {
    category = "AllMetrics"
    retention_policy {
      days    = 30
      enabled = true
    }
  }
}

I run into an error that says:

"diagnosticsettings.DiagnosticSettingsClient#CreateOrUpdate: Failure sending request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=nil nil"

When I tried to google that error messages I found issues related to other Azure services where the sku of that service wasn't matching a specified feature or capacity but I'm don't see that here.

Why I want log analytics workspace AND logs dumped into a storage account: My thinking was just that a log anal. ws is really expensive compared to storage in a storage account. So I thought I send say the audit data for long time retention to the cheap storage account (my settings in the given example might not 100% represent that but it's not the point here I'd say) and still have the "expensive" log analytics service to dig into the cluster performance.

Thanks a lot for any input!

Solution

I Tried to reproduce the same in my environment to Create an Azure AKS cluster with OMS Agent and Diagnostic Setting using Terraform:

Sending long-term data retention logs to a Azure Storage Account can be more cost-effective than keeping them in a Azure Log Analytics workspace. However, the Azure Log Analytics workspace can still be useful for real-time analysis and performance monitoring.

    provider "azurerm" {
  features {}
}
resource "azurerm_resource_group" "aksgroup" {
  name     = "aks-rg"
  location = "East US"
}

resource "azurerm_log_analytics_workspace" "oms" {
  name                = "oms-workspace"
  location            = azurerm_resource_group.aksgroup.location
  resource_group_name = azurerm_resource_group.aksgroup.name
  sku                 = "PerGB2018"
}

resource "azurerm_kubernetes_cluster" "aks" {
  name                = "cluster-aks1"
  location            = azurerm_resource_group.aksgroup.location
  resource_group_name = azurerm_resource_group.aksgroup.name
  dns_prefix          = "aks1"

  default_node_pool {
    name       = "default"
    node_count = 1
    vm_size    = "standard_a2_v2"
  }

  identity {
    type = "SystemAssigned"
  }

  tags = {
    Environment = "Production"
  }
  addon_profile {
      oms_agent {
        enabled                    = true
        log_analytics_workspace_id = azurerm_log_analytics_workspace.oms.id
      }
    }
}

output "client_certificate" {
  value     = azurerm_kubernetes_cluster.aks.kube_config.0.client_certificate
  sensitive = true
}

output "kube_config" {
  value = azurerm_kubernetes_cluster.aks.kube_config_raw

  sensitive = true
}

resource "azurerm_monitor_diagnostic_setting" "aks" {
  name                 = "aks-diagnostic-setting"
  target_resource_id   = azurerm_kubernetes_cluster.aks.id
  storage_account_id   = azurerm_storage_account.aks.id
  log_analytics_workspace_id = azurerm_log_analytics_workspace.oms.id
  log {
    category = "kube-audit"
    enabled  = true
  }
  metric {
    category = "AllMetrics"
    retention_policy {
      days    = 30
      enabled = true
    }
  }
}

resource "azurerm_storage_account" "aks" {
  name                = "aksdiagnostic"
  resource_group_name = azurerm_resource_group.aksgroup.name
  location            = azurerm_resource_group.aksgroup.location
  account_tier        = "Standard"
  account_replication_type = "LRS"
}

Terraform Apply:

enter image description here

Once ran the code resources are created, like below.

enter image description here

Azure AKS Diagnostic settings created with Log Analytics settings.

enter image description here

Log Analytics settings- created.

enter image description here