Terraform deployment of Fluent Bit which should direct specific namespaces' logs to separate Log Analytics workspaces

Question

The overall idea should generate Fluent Bit's output blocks for each namespace. This will allow the logs of each namespace to be directed to a designated Log Analytics workspace, within a table, named as the namespace.

So... I have two maps with data - one is a simple key-value map (fluentbit_output_mappings), the other is a map of objects which contain map with more variables (log_analytics_workspaces). The values of the first map equal to the object names of the second map (and to second map's log_analytics_name parameters values also).

With this relation, I would like to be able to pull each of the second map's objects resource_group_name parameter's value.

Once then, using data source azurerm_log_analytics_workspace, I would like to get workspace_id and primary_shared_key of the Log Analytics workspaces.

Afterward, I would like to be enriching a template which should contain the key of the simple key-value map (fluentbit_output_mappings), and the workspace_id and primary_shared_key of the Log Analytics workspaces. The template should then be ready for an output, which I can use later.

Notes:

• The map of Log Analytics workspaces objects cannot be adjusted as it is used elsewhere
• Additional enhancements are also included like making sure the correct Log Analytics workspaces names are added as values in the fluentbit_output_mappings map; dependencies to avoid false runs

variable "log_analytics_workspaces" {
  type = map(
    object({
        "LA1" = {
        resource_group_name = "LA1_RG"
        log_analytics_name  = "LA1"
        log_analytics_sku   = "PerGB2018"
        },
        "LA2" = {
        resource_group_name = "LA2_RG"
        log_analytics_name  = "LA2"
        log_analytics_sku   = "PerGB2018"
        },
        "LA3" = {
        resource_group_name = "LA3_RG"
        log_analytics_name  = "LA3"
        log_analytics_sku   = "PerGB2018"
        },
        "LA4" = {
        resource_group_name = "LA4_RG"
        log_analytics_name  = "LA4"
        log_analytics_sku   = "PerGB2018"
        }
    })
  )
}

variable "fluentbit_output_mappings" {
  description = "Map of K8s namespaces to Log Analytics workspace names"
  type        = map(string)
  default     = {
    # Add more mappings as needed. Syntax: K8s_namespace = Log_Analytics_workspace
    "ns1" = "LA1"
    "ns2" = "LA2"
    "ns3" = "LA3"
    "ns4" = "LA1"
    "ns5" = "LA2"
  }
}

data "log_analytics_workspace" "mappings" {
  for namespace, log_analytics_name in var.fluentbit_output_mappings : namespace => {
    namespace = namespace
    log_analytics_workspace_name = log_analytics_name == var.log_analytics_workspaces[log_analytics_name].log_analytics_name ? log_analytics_name : "falseValue"
    log_analytics_resource_group_name = var.log_analytics_workspaces[log_analytics_name].resource_group_name

    # Assert condition to stop processing if log_analytics_workspace_name is "falseValue"
    assert(log_analytics_workspace_name != "falseValue", "Error: log_analytics_workspace_name is invalud. Check terragrunt.hcl for correct mapping in fluentbit_output_mappings")
  }

  depends_on = [
    var.fluentbit_output_mappings,
    var.log_analytics_workspaces,
  ]
}

data "azurerm_log_analytics_workspace" "log_analytics" {
  for_each = { for namespace, log_analytics_name in var.fluentbit_output_mappings : namespace => log_analytics_name }
  
  name                = data.log_analytics_workspace.mappings[var.fluentbit_output_mappings[count.index]].log_analytics_workspace_name
  resource_group_name = data.log_analytics_workspace.mappings[var.fluentbit_output_mappings[count.index]].log_analytics_resource_group_name

  depends_on = [
    var.fluentbit_output_mappings,
    data.log_analytics_workspace.mappings,
  ]
}

data "template_file" "example" {
  count = length(var.workspace_mappings)

  template = <<YAML
[OUTPUT]
    Name azure
    Match kube.*${var.fluentbit_output_mappings.key}*
    Customer_ID ${data.azurerm_log_analytics_workspace.log_analytics[data.log_analytics_workspace.mappings[var.fluentbit_output_mappings[count.index]]].workspace_id}
    Shared_Key ${data.azurerm_log_analytics_workspace.log_analytics[data.log_analytics_workspace.mappings[var.fluentbit_output_mappings[count.index]]].primary_shared_key}
    Log_Type ${var.fluentbit_output_mappings.key}
  YAML

  depends_on = [
    var.fluentbit_output_mappings,
    data.azurerm_log_analytics_workspace.workspace,
  ]
}

output "rendered" {
  value = join("\n", data.template_file.example[*].rendered)
  sensitive = true
}

However, upon run, I receive the following error where the , (comma) between "namespace" and "log_analytics_name" on line 42* (in data.log_analytics_workspace.mappings):

• It is line 17 for me as log_analytics_workspaces is used from variables.tf and fluentbit_output_mappings is declared in the above .tf file

╷
│ Error: Invalid block definition
│
│   on fluentbit.tf line 42, in data "log_analytics_workspace" "mappings":
│   42:   for namespace, log_analytics_name in var.fluentbit_output_mappings : namespace => {
│
│ Either a quoted string block label or an opening brace ("{") is expected
│ here.
╵
time=2024-02-09T15:49:47Z level=error msg=1 error occurred:
        * exit status 1

Any ideas on how to work this out will be highly appreciated. Thanks in advance!

Answer 1

So as no answers and I was stuck, I decided to modify my map so that it includes the Log Analytics' resource group names. Once then, I had to only loop through a single loop and didn't have to bind values and variables from different map objects. That then gave me a smooth way of operation to loop in order to generate the output.

I know there is a better way, but I will seek for it upon optimization as I do not have time for it now.