Search code examples
jsongodecode

json.Unmarshal file data works but json.NewDecoder().Decode() does not

The following correctly unmarshals the struct:

func foo() {
  d, err := os.ReadFile("file.json")

  var t T
  if err := json.Unmarshal(d, &t); err != nil {
    panic(err)
  }
}

but this doesn't work and throws a bunch of a classic JSON parsing errors i.e. EOF, unexpected token 't', etc.

func foo() {
  f, err := os.Open("file.json")
  
  var t T
  if err := json.NewDecoder(f).Decode(&t); err != nil {
    panic(err)
  }
}

Any idea why? The os.File or []byte is used in two goroutines at once, and the JSON is of the following structure (with some fields omitted):

{
  "data": [
    {
      "field": "stuff",
      "num": 123,
    },
    ...
  ]
}
Solution

  • The os.File or []byte is used in two goroutines at once...

    That's the issue. os.File has an internal file pointer, a position where the next read happens. If 2 unrelated entities keep reading from it, they will likely not read overlapping data. Bytes read by the first entity will not be repeated for the second.

    Also, os.File is not safe for concurrent use (the documentation doesn't state explicitly that it's safe): calling its methods from multiple, concurrent goroutines may result in a data race.

    When you pass a []byte to multiple functions / goroutines which read "from it", there's no shared pointer or index variable. Each function / goroutine will maintain its index, separately, and reading a variable from multiple goroutines is OK (which in this case would be the (fields of the) slice header and the slice elements).