Is Go offering an equivalent to Threadpool

Question

I'm coming from Java/Scala and started using Go recently. In Java/Scala threadpools are pretty common and they'll be used for at least 4 different reasons.

1. Reuse worker already instantiated
2. Resource management. When we have multiple threadpools we can ensure that if there is a burst in one part of the system, it doesn't stop the other parts from running.
3. Customize the type of scheduling we'd want (fork/join, classic, scheduled, etc ...)
4. Customize rejection policy.

Since Goroutines are so light 1 is not needed, and even though it would be nice to provide one, we could create some kind of worker pool without too much trouble to address 2.

However, I feel that in Go we cannot address 3 and 4.

Is it because it's not needed or is it just a missing functionalty?

Answer 1

As you've guessed, due to (1) being mostly a non-issue in Go, thread/worker pools are much less needed. (2) is also possible to address with techniques like rate limiting and resource semaphores. As for (3), the Go runtime does the goroutine scheduling, so customizing this isn't easy, or idiomatic.

That said, thread/worker pools are extremely easy to implement in Go. Here's a simple example from gobyexample:

// In this example we'll look at how to implement
// a _worker pool_ using goroutines and channels.

package main

import "fmt"
import "time"

// Here's the worker, of which we'll run several
// concurrent instances. These workers will receive
// work on the `jobs` channel and send the corresponding
// results on `results`. We'll sleep a second per job to
// simulate an expensive task.
func worker(id int, jobs <-chan int, results chan<- int) {
    for j := range jobs {
        fmt.Println("worker", id, "started  job", j)
        time.Sleep(time.Second)
        fmt.Println("worker", id, "finished job", j)
        results <- j * 2
    }
}

func main() {

    // In order to use our pool of workers we need to send
    // them work and collect their results. We make 2
    // channels for this.
    jobs := make(chan int, 100)
    results := make(chan int, 100)

    // This starts up 3 workers, initially blocked
    // because there are no jobs yet.
    for w := 1; w <= 3; w++ {
        go worker(w, jobs, results)
    }

    // Here we send 5 `jobs` and then `close` that
    // channel to indicate that's all the work we have.
    for j := 1; j <= 5; j++ {
        jobs <- j
    }
    close(jobs)

    // Finally we collect all the results of the work.
    // This also ensures that the worker goroutines have
    // finished. An alternative way to wait for multiple
    // goroutines is to use a [WaitGroup](waitgroups).
    for a := 1; a <= 5; a++ {
        <-results
    }
}

And if you google a bit, you'll find a bunch of 3rd-party packages implementing this pattern for various uses. As you could guess, there's no single best way to do this, so you'll pick whatever is important for your specific use case.