Cannot detect that a tcp service in Kubernetes has no pods with golang app

Question

I have a statsd client that pushes to a kubernetes-hosted statsd service in tcp.

When I redeploy the service and update its pods on the server, the data stops flowing but I have no error on the statsd client.

Based on this thread, I have added a read detection to my golang library to see if I receive a message, but nothing.

I start the connection, send statsd packets from the client, then after I while, I scale the statsd Service down to 0 pods, but the client continues to successfully read and write from the tcp channel without errors!

When I scale the deployment back to 1, the old client does not reconnect and the client and the server continue to remain non-communicating forever.

How can I receive an error from the client when the server replicas go to 0 in the Kubernetes service?

The statsd client is a fork of https://github.com/alexcesaro/statsd

Note that we are aware that statsd is used mostly in UDP but this is a custom flavor and for several reason we need one instance of this in TCP (already have several UDP ones and they don't have this problem)

...
    c.w, err = net.DialTimeout(c.network, c.addr, 5*time.Second)
...

func (c *conn) checkTCPConnectionOpen() error {
    if c.network[:3] == "tcp" {
        one := []byte{}
        _, err := c.w.Read(one)
        return err
    }
    return nil
}

func (c *conn) flush(n int) {
    if len(c.buf) == 0 {
        return
    }
    if n == 0 {
        n = len(c.buf)
    }

    err := c.checkTCPConnectionOpen()
    c.handleError(err)

    _, err = c.w.Write(c.buf[:n])
    c.handleError(err)
    if n < len(c.buf) {
        copy(c.buf, c.buf[n:])
    }
    c.buf = c.buf[:len(c.buf)-n]
}

Answer 1

Ok so this is not a Kubernetes problem, but a golang one.

This is basically a duplicate of How to know TCP connection is closed in Golang net package?