I'm a Go novice coming from the Node world and I'm building a consumer using the official Bitly Go client. I am using AddConcurrentHandlers to spawn 50 goroutines to handle the fire hose of messages. The issue is that my consumer falls behind leaving an exponential of unprocessed/received messages on nsq. Has anyone else encountered this?
I built the same thing in Node to see if there was a server or NSQ configuration issue and its able to process all messages as quickly as they come in.
GO CODE:
q, _ := nsq.NewConsumer("chat", "golangbetches", config)
q.AddConcurrentHandlers(nsq.HandlerFunc(func(message *nsq.Message) error {
l.Debug("Got a message: %v", message)
message.Finish()
return nil
}), 50)
err := q.ConnectToNSQLookupd("<address here>")
cfg.MaxInFlight handles the, "maximum number of messages this comsumer instance will allow in-flight..." More details are available in the consumer source
Set cfg.MaxInFlight to something reasonable, as it defaults to 1
An example configuration is available in the documentation where it is set to 1000. This may or may not be suitable for your application; and, you'd do well to monitor it, as a misconfiguration may result in truncated messages.