Go, tcp too many open files debug

Question

Here's a straightforward Go http (tcp) connection test script

func main() {
    ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        fmt.Fprintln(w, "Hello, client")
    }))
    defer ts.Close()
    var wg sync.WaitGroup
    for i := 0; i < 2000; i++ {
        wg.Add(1)
        go func(i int) {
            defer wg.Done()
            resp, err := http.Get(ts.URL)
            if err != nil {
                panic(err)
            }
            greeting, err := ioutil.ReadAll(resp.Body)
            resp.Body.Close()
            if err != nil {
                panic(err)
            }
            fmt.Printf("%s", i, greeting)
        }(i)
    }
    wg.Wait()
}

And If I run this in Ubuntu I get:

panic: Get http://127.0.0.1:33202: dial tcp 127.0.0.1:33202: too many open files

Other posts say to make sure Close the connection, which I am doing it all here. And others say to increase the limit of maximum connection with ulimit or try sudo sysctl -w fs.inotify.max_user_watches=100000 but still does not work.

How do I run millions of tcp connection goroutines in a single server? It crashes only with 2,000 connections.

Thanks,

Answer 1

I think you need to change your max file descriptors. I have run into the same problem on one of my development VMs before and needed to change the file descriptors max, not anything with inotify settings.

FWIW, your program runs fine on my VM.

·> ulimit -n
120000

But after I run

·> ulimit -n 500
·> ulimit -n
500

I get:

panic: Get http://127.0.0.1:51227: dial tcp 127.0.0.1:51227: socket: too many open files

** Don't fall into the trap that Praveen did **

Note ulimit != ulimit -n.

➜  cmd git:(wip-poop) ✗ ulimit -a
-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8192
-c: core file size (blocks)         0
-v: address space (kbytes)          unlimited
-l: locked-in-memory size (kbytes)  unlimited
-u: processes                       1418
-n: file descriptors                4864