sending millions of short messages over tcp socket in golang

Question

I'm writing two services in golang that need to send to each other about 2 million messages per second. Each message is about 50 bytes, so throughput should only be about 100MB/s. I want to use tcp for this. However, results are very slow. I configured SetNoDelay(false) to make sure that data is buffered before sending, but that didn't make any difference.

I can only send about 50k messages per second, and message size doesn't matter too much, so I assume the code is blocking somewhere. Here's my test code:

package main

import "net"
import "fmt"
import "bufio"
import (
    //"strings"
    "time"
)

func startserver() {
    fmt.Println("Launching server...")
    ln, _ := net.Listen("tcp", ":8081")
    conn, _ := ln.Accept()

    for {
        bufio.NewReader(conn).ReadString('\n')
        //fmt.Println(message)
        //newmessage := strings.ToUpper(message)
        //conn.Write([]byte(newmessage + "\n"))
    }
}

func startclient() {
    time.Sleep(time.Second) // so that server has time to start
    servAddr := "127.0.0.1:8081"
    tcpAddr, _ := net.ResolveTCPAddr("tcp", servAddr)
    conn, _ := net.DialTCP("tcp", nil, tcpAddr)
    conn.SetNoDelay(false)
    conn.SetWriteBuffer(10000)
    msg := "abc\n"
    start := time.Now()
    for i := 0; i < 1000000; i++ {
        conn.Write([]byte(msg))
        //bufio.NewReader(conn).ReadString('\n')
        //fmt.Print("Message from server: ", response)
    }
    fmt.Println("took:", time.Since(start))
}

func main() {
    go startserver()
    startclient()
}

Any suggestion?

Answer 1

Syscalls are expensive, and writing data over loopback is very fast, so this is essentially just timing how fast you can make 1 million calls to write.

Making lots of small reads and writes is the primary use case for using buffered io. Wrapping the normal net.TCPConn in a bufio.Writer will provide the expected performance increase.