How to read input from one channel and write output to another channel in golang go routine?

Question

I am learning golang and I was wondering if it is possible to have a goroutine read the input from one channel and write output to another channel while it is bounded by waitgroup?

See the code below

func downloaderInputUrlFromChannel(id int, urlChan chan string, responseChan chan int, wg *sync.WaitGroup) {

    for url := range urlChan {
        resp, _ := http.Get(url)
        fmt.Println("STARTED worker ", id, " For URL ", url, "With Response Code", resp.StatusCode)
        responseChan <- resp.StatusCode
        fmt.Println("FINISHED worker ", id, " For URL ", url, "With Response Code", resp.StatusCode)

    }
    defer wg.Done()
}

func main() {
    urls := []string{....}  // Containing 25 urls
    MaxGoroutineCount := 5
    var wg sync.WaitGroup
    urlChan := make(chan string)
    responseChan := make(chan int)

    // Start Workers
    for i := 0; i < MaxGoroutineCount; i++ {
        wg.Add(1)
        go downloaderInputUrlFromChannel(i, urlChan, responseChan, &wg)
    }

    // Fill in the queues
    for _, url := range urls {
        urlChan <- url
    }

    close(urlChan)
    for i := 0; i < len(urls); i++ {
        fmt.Println(<-responseChan)
    }
    close(responseChan)
    wg.Wait()

I do not see the output with line FINISHED with the output being

STARTED worker  0  For URL  https://httpbin.org?q=1,l=2,p=1 With Response Code 200
STARTED worker  4  For URL  https://httpbin.org/ With Response Code 200
STARTED worker  2  For URL  https://httpbin.org?q=1,r=2,p=1 With Response Code 200
STARTED worker  1  For URL  https://httpbin.org?q=1 With Response Code 200
STARTED worker  3  For URL  https://httpbin.org?a=1 With Response Code 200

So responseChan channel never gets filled, Even though I am consuming message from that channel in the main the program is stuck at that point. If I remove responseChan altogether, the program works as expected. And faster than sequential run.

I need the output from downloaderInputUrlFromChannel so I need to have some channel. Can anyone help me what am I doung wrong ?

Answer 1

Quick fix: create the responseChan with make(chan int, len(urls)).

Let's begin with how you are handling the channels. For both of them, the capacity is missing (in the make built-in function), therefore the channels are unbuffered[1].

Channel: The channel's buffer is initialized with the specified buffer capacity. If zero, or the size is omitted, the channel is unbuffered.

That means the execution is blocked if there is no receiver ready prior to a send. So, as you know there is a maximum amount of urls, it is possible to create the channel as follow: make(chan int, len(urls)). That doesn't apply to the urlChan because there are 5 goroutines receiving.

As @Gimby mentioned, it makes sense to close the channels after you are completely done with them. In your case, you could just defer the close calls right after creating the channels. Although, it will still work regardless of that because you are sending all the urls before closing the channel, and the actual shutdown won't happen until the last sent value is received[2].