How do I extract distinct values from a CSV column in Terraform?

I have a CSV with multiple Network Security rules in it. I am trying to create NSGs using this list. However, there are duplicates in the list causing it to fail on creation. How do I extract distinct values from a CSV column in Terraform?

locals {
  csv_data = <<-CSV
  id,nsgname,rgname,rule,priority
  1,nsg-one,rg-test,rule1,100
  2,nsg-one,rg-test,rule2,110
  3,nsg-one,rg-test,rule3,120
  4,nsg-one,rg-test,rule4,130
  5,nsg-one,rg-test,rule5,100
  6,nsg-one,rg-test,rule6,110
  7,nsg-one,rg-test,rule7,120
  8,nsg-one,rg-test,rule8,130
  9,nsg-two,rg-test,rule9,2300
  10,nsg-two,rg-test,rule10,2300
  11,nsg-two,rg-test,rule11,2140
  12,nsg-two,rg-test,rule12,2140
  13,nsg-three,rg-test,rule13,2100
  14,nsg-three,rg-test,rule14,2110
  15,nsg-three,rg-test,rule15,2120
  16,nsg-three,rg-test,rule16,2130
  17,nsg-three,rg-test,rule17,2100
  18,nsg-three,rg-test,rule18,2130
  19,nsg-three,rg-test,rule19,2140
  20,nsg-three,rg-test,rule20,2140
  21,nsg-four,rg-test,rule21,2300
  22,nsg-four,rg-test,rule22,2140
  23,nsg-four,rg-test,rule23,2140
  CSV

  nsgss = csvdecode(local.csv_data)
}

resource "azurerm_network_security_group" "tf_network_security_groups" {
  for_each = { for nsg in local.nsgss : nsg.id => nsg }

  name                = format("nsg-%s", each.value.nsgname)
  location            = "uksouth"
  resource_group_name = "rg-test"
}

I am assuming that the resource group rg-test exists for this exercise. If you want to test it then you will need to create it by hand or by using the below config.

resource "azurerm_resource_group" "tf_resource_group" {
  name     = "rg-test"
  location = "uksouth"
}

Solution

A column can be extracted from a CSV by using the splat syntax

dist = distinct(local.nsgss.*.nsgname)

So my previous example needs to be changed to the following

csv_data = <<-CSV
    id,nsgname,rgname,rule,priority
    1,nsg-one,rg-test,rule1,100
    2,nsg-one,rg-test,rule2,110
    3,nsg-one,rg-test,rule3,120
    4,nsg-one,rg-test,rule4,130
    5,nsg-one,rg-test,rule5,100
    6,nsg-one,rg-test,rule6,110
    7,nsg-one,rg-test,rule7,120
    8,nsg-one,rg-test,rule8,130
    9,nsg-two,rg-test,rule9,2300
    10,nsg-two,rg-test,rule10,2300
    11,nsg-two,rg-test,rule11,2140
    12,nsg-two,rg-test,rule12,2140
    13,nsg-three,rg-test,rule13,2100
    14,nsg-three,rg-test,rule14,2110
    15,nsg-three,rg-test,rule15,2120
    16,nsg-three,rg-test,rule16,2130
    17,nsg-three,rg-test,rule17,2100
    18,nsg-three,rg-test,rule18,2130
    19,nsg-three,rg-test,rule19,2140
    20,nsg-three,rg-test,rule20,2140
    21,nsg-four,rg-test,rule21,2300
    22,nsg-four,rg-test,rule22,2140
    23,nsg-four,rg-test,rule23,2140
  CSV

  nsgss = csvdecode(local.csv_data)

  dist = distinct(local.nsgss.*.nsgname)
}

resource "azurerm_network_security_group" "tf_network_security_groups" {
  for_each = toset(local.dist)

  name                = each.key
  location            = "uksouth"
  resource_group_name = "rg-test"
}