golang: unmarshal various toml config using go-toml/v2

Question

How to unmarshal toml config file with various subsection name using go-toml/v2?

I have a toml config file like:

url     = "https://grafana.home/api/annotations"
api_key = "glsa_secret_2b3226ac"
cluster = "Sandbox"

[spikes]
dashboard = "84357279284502"
panel     = 23

[logs]
dashboard = "84357279284503"
panel     = 24

[cpu]
dashboard = "84357279284504"
panel     = 25

The part with url, api_key, and cluster is easy to unmarshal using structure. The parts with spikes,logs, and cpu is also easy to unmarshal by the structure, but [name] of each part can be different in different environments.

Answer 1

Your TOML document corresponds to the following JSON structure:

{
  "url": "https://grafana.home/api/annotations",
  "api_key": "glsa_secret_2b3226ac",
  "cluster": "Sandbox",
  "spikes": {
    "dashboard": "84357279284502",
    "panel": 23
  },
  "logs": {
    "dashboard": "84357279284503",
    "panel": 24
  },
  "cpu": {
    "dashboard": "84357279284504",
    "panel": 25
  }
}

That's tricky to parse into Go structures because the top level contains both known and unknown keys, and maps the keys to different data types (strings in one case and maps in the other).

If you have control over the format of this file, you would be better off moving the dynamic keys under a specific attribute.

If you don't have control over the file format, you'll need to unmarshal the data into a map[string]interface{}, and then inspect the key name or value type at runtime to determine what to do.

Here's one example:

package main

import (
  "fmt"
  "os"

  toml "github.com/pelletier/go-toml/v2"
  "k8s.io/utils/strings/slices"
)

func main() {
  data := make(map[string]interface{})

  content, err := os.ReadFile("data.toml")
  if err != nil {
    panic(err)
  }

  if err := toml.Unmarshal(content, &data); err != nil {
    panic(err)
  }

  for k, v := range data {
    if slices.Contains([]string{"url", "api_key", "cluster"}, k) {
      // it's a known string value
      fmt.Printf("Got %s = %s\n", k, v)
    } else {
      // it's an unknown key so value is a map
      fmt.Printf("Got dashboard %s:\n", k)
      x := v.(map[string]interface{})
      for kk, vv := range x {
        fmt.Printf("  %s: %v\n", kk, vv)
      }

      // we can also ask for specific items if we know the keys
      fmt.Printf("dashboard id is %s\n", v.(map[string]interface{})["dashboard"])
      fmt.Printf("panel id is %d\n", v.(map[string]interface{})["panel"])
    }
  }
}

That produces as output:

Got url = https://grafana.home/api/annotations
Got api_key = glsa_secret_2b3226ac
Got cluster = Sandbox
Got dashboard spikes:
  dashboard: 84357279284502
  panel: 23
dashboard id is 84357279284502
panel id is 23
Got dashboard logs:
  dashboard: 84357279284503
  panel: 24
dashboard id is 84357279284503
panel id is 24
Got dashboard cpu:
  dashboard: 84357279284504
  panel: 25
dashboard id is 84357279284504
panel id is 25

---

We can be sneaky and unmarshal the values of the unknown keys back to TOML and then into a struct if we're not particularly worried about performance:

package main

import (
  "fmt"
  "os"

  toml "github.com/pelletier/go-toml/v2"
  "k8s.io/utils/strings/slices"
)

type (
  Dashboard struct {
    Dashboard string
    Panel     int
  }
)

func main() {
  data := make(map[string]interface{})

  content, err := os.ReadFile("data.toml")
  if err != nil {
    panic(err)
  }

  if err := toml.Unmarshal(content, &data); err != nil {
    panic(err)
  }

  for k, v := range data {
    if slices.Contains([]string{"url", "api_key", "cluster"}, k) {
      // it's a known string value
      fmt.Printf("Got %s = %s\n", k, v)
    } else {
      // it's an unknown key so value is a map
      dashboard := Dashboard{}
      tomlStr, err := toml.Marshal(v)
      if err != nil {
        panic(err)
      }
      if err := toml.Unmarshal(tomlStr, &dashboard); err != nil {
        panic(err)
      }

      fmt.Printf("Got dashboard %s: %+v\n", k, dashboard)
    }
  }
}

This produces:

Got dashboard spikes: {Dashboard:84357279284502 Panel:23}
Got dashboard logs: {Dashboard:84357279284503 Panel:24}
Got dashboard cpu: {Dashboard:84357279284504 Panel:25}
Got url = https://grafana.home/api/annotations
Got api_key = glsa_secret_2b3226ac
Got cluster = Sandbox