How to limit go routines based on number of file descriptors

Question

I have a program that executes queries to an HTTP server, each request gets a goroutine. I quickly found out it was too much for MacOS, as there is a file descriptor limit of 250.

I was wondering if I could limit the number of goroutines, or perhaps block until there are available goroutines, instead of failing.

Maybe a worker pool with 250 goroutines, and queue the rest of the requests?

What are your thoughts

Answer 1

package main

import "fmt"

const ROUTINE_LIMIT = 250

func main() {

    channelCounter := make(chan bool, ROUTINE_LIMIT)

    for {
        select {
        //will add till 250 after that it will block
        case channelCounter <- true:
            fmt.Println("added channel")
            go startRoutine(channelCounter)
        }
    }
}

func startRoutine(channelCounter chan bool) {
    /*
        do your stuff
    */

    //free the channel
    <-channelCounter
}

You can limit your goruotine count using channels. A channel to keep count of the number of go routines running .. And once the job is done you can read the channel value to reduce the count.

The above program is like a rough sample code.. (i think it covers the general idea)