Handling streamed HTTP ressponses

Question

I have the following example that connects to an HTTP service which streams responses back in a stream of chunks to create a JSON structure. For each chunk my code appends a byte rb array with the individual lines. However, my problem is trying to work out when the rb is complete so I can then decode it.

Am I missing something obvious here ?

package main

import (
    "bufio"
    "bytes"
    "fmt"
    "io"
    "net/http"
)

func main() {

    body := []byte("test")

    resp, err := http.Post("http://localhost:8281/tap", "application/json", bytes.NewReader(body))
    if err != nil {
        fmt.Printf("%v\n", err)
        return
    }
    defer resp.Body.Close()
    fmt.Printf("Status: [%s]\n", resp.Status)
    fmt.Println()
    //var rb []byte
    reader := bufio.NewReader(resp.Body)
    var rb []byte

    for {

        line, err := reader.ReadBytes('\n')
        if err != nil {
            if err == io.EOF {
                break
            }
            fmt.Printf("Error reading streamed bytes %v", err)
        }
        rb = append(rb, line...)

        fmt.Println(rb)

    }
}

Answer 1

Ignoring the bugs in the program, rb is complete after the loop breaks.

The program does have bugs:

• The program only breaks out of the loop on EOF. The program will spin forever on any other kind of error.
• The program does not handle the case where ReadBytes returns data and an error. An example of where this can happen is when the response does not end with the delimiter.

It looks like your goal is to slurp up the entire response to rb. Use io.ReadAll do do that:

resp, err := http.Post("http://localhost:8281/tap", "application/json", bytes.NewReader(body))
if err != nil {
    fmt.Printf("%v\n", err)
    return
}
defer resp.Body.Close()
rb, err := io.ReadAll(resp.Body)
if err != nil {
    // handle error
}
var data SomeType
err = json.Unmarshal(rb, &data)
if err != nil {
     // handle error
}

If you want to decode the response body as JSON, then a better approach is to let the JSON decoder read the response body:

resp, err := http.Post("http://localhost:8281/tap", "application/json", bytes.NewReader(body))
if err != nil {
    fmt.Printf("%v\n", err)
    return
}
defer resp.Body.Close()
var data SomeType
err := json.NewDecoder(resp.Body).Decode(&data)