Why does for loop with goroutines result in missing data

Question

Ok, so I have two bits of code. First off is a simple for loop that works great

package main

import (
    "context"
    "encoding/json"
    "fmt"
    "log"
    "os"

    elasticsearch "github.com/elastic/go-elasticsearch/v7"
    "github.com/elastic/go-elasticsearch/v7/esapi"
    "github.com/mitchellh/mapstructure"
)

type Esindices struct {
    Health       string `json:"health"`
    Status       string `json:"status"`
    Index        string `json:"index"`
    Uuid         string `json:"uuid"`
    Pri          string `json:"pri"`
    Rep          string `json:"rep"`
    DocsCount    string `json:"docs.count"`
    DocsDeleted  string `json:"docs.deleted"`
    StoreSize    string `json:"store.size"`
    PriStoreSize string `json:"pri.store.size"`
}

func main() {

    var r []map[string]interface{}

    es, err := elasticsearch.NewDefaultClient()
    if err != nil {
        log.Fatalf("Error creating client: %s", err)
    }

    req := esapi.CatIndicesRequest{
        Format: "json",
        Pretty: false,
    }

    res, err := req.Do(context.Background(), es)
    if err != nil {
        log.Fatalf("Error getting response: %s", err)
    }

    defer res.Body.Close()

    if err := json.NewDecoder(res.Body).Decode(&r); err != nil {
        log.Printf("Error parsing the response body: %s", err)
    }

    indexSlice := make([]*Esindices, len(r))

    for i, element := range r {
        result := &Esindices{}
        cfg := &mapstructure.DecoderConfig{
            Metadata: nil,
            Result:   &result,
            TagName:  "json",
        }
        decoder, _ := mapstructure.NewDecoder(cfg)
        decoder.Decode(element)
        indexSlice[i] = result
    }

    thisisjson, err := json.MarshalIndent(indexSlice, "", " ")
    if err != nil {
      log.Fatal("Can't encode to JSON", err)
    }


    fmt.Fprintf(os.Stdout, "%s", thisisjson)

Most of this is pretty self-explanatory, but just to clarify I am using the Elasticsearch client and the api.cat.indices API to get a list of all the indices in a local Elasticsearch install and then store them as an array of map[string]interface{} and then loop over this to add them to a slice of a struct of the results. This is fine, actually, but I want to be mindful of performance, and while I can't improve the latency of the request itself I can certainly improve the performance of the loop, at least I think I should be able to.

So when I try the below instead I get weird results.

var wg sync.WaitGroup
defer wg.Wait()
for i, element := range r {
    wg.Add(1)
    go func(i int, element map[string]interface{}) {
        defer wg.Done()
        result := Esindices{}
        cfg := &mapstructure.DecoderConfig{
            Metadata: nil,
            Result:   &result,
            TagName:  "json",
        }
        decoder, _ := mapstructure.NewDecoder(cfg)
        decoder.Decode(element)
        indexSlice[i] = result
    }(i, element)
}

The issue is, specifically, the some of the values of the keys of the elements in the slice are empty. This makes me think the code is trying to add to the slice, but it's passing even if it's not done.

Thoughts?

Answer 1

Instead of defer wg.Wait, use wg.Wait at the end of the for-loop. You are using the data constructed by the goroutines in the for-loop right after for-loop completes, and you're not waiting for all the goroutines to complete before you use that data.

When you use defer wg.Wait, waiting happens at the end of the function. The for-loop using the data operates on incomplete data because the goroutines are still running.

When you use wg.Wait at the end of the for-loop, you first wait for all the goroutines to end, and then use the data generated by them.