downloading files with goroutines?

Question

I'm new to Go and I'm learning how to work with goroutines.

I have a function that downloads images:

func imageDownloader(uri string, filename string) {
    fmt.Println("starting download for ", uri)

    outFile, err := os.Create(filename)
    defer outFile.Close()
    if err != nil {
        os.Exit(1)
    }

    client := &http.Client{}

    req, err := http.NewRequest("GET", uri, nil)

    resp, err := client.Do(req)
    defer resp.Body.Close()

    if err != nil {
        panic(err)
    }

    header := resp.ContentLength
    bar := pb.New(int(header))
    rd := bar.NewProxyReader(resp.Body)
    // and copy from reader
    io.Copy(outFile, rd)
}

When I call by itself as part of another function, it downloads images completely and there is no truncated data.

However, when I try to modify it to make it a goroutine, images are often truncated or zero length files.

func imageDownloader(uri string, filename string, wg *sync.WaitGroup) {
    ...
    io.Copy(outFile, rd)
    wg.Done()
}

func main() {
var wg sync.WaitGroup
wg.Add(1)
go imageDownloader(url, file, &wg)
wg.Wait()
}

Am I using WaitGroups incorrectly? What could cause this and how can I fix it?

Update:

Solved it. I had placed the wg.add() function outside of a loop. :(

Answer 1

While I'm not sure exactly what's causing your issue, here's two options for how to get it back into working order.

First, looking to the example of how to use waitgroups from the sync library, try calling defer wg.Done() at the beginning of your function to ensure that even if the goroutine ends unexpectedly, that the waitgroup is properly decremented.

Second, io.Copy returns an error that you're not checking. That's not great practice anyway, but in your particular case it's preventing you from seeing if there is indeed an error in the copying routine. Check it and deal with it appropriately. It also returns the number of bytes written, which might help you as well.