Search code examples
linuxgohttpsrate-limitingtrafficshaping

How can I externally rate-limit a golang http upload?


I have a golang utility that uploads a lot of data via https to OpenStack Object Storage. I'm running it on Ubuntu Linux, and I would like to ensure that its maximum upload bandwidth does not exceed 2,500KB/s, preferably without affecting other users on the same system (i.e. slowing down the ethernet interface).

How can I do this, preferably without altering my source code? I've tried a few approaches so far:

  1. Traffic Shaping: I've been unable to figure out the appropriate network shaping techniques to affect a single process, though I know that should be possible with iptables and tc. Any guidance on how to do this would be appreciated. My networking background isn't very extensive.
  2. Flowrate Limiting: I tried using the flowrate package in go to rate-limit my writes to the upload. This didn't have any discernable effect. I think this doesn't work because the github.com/ncw/swift.ObjectCreateFile.Write() method doesn't upload the data as it is recieved, but rather when it is closed. I could be mistaken about this though.
  3. trickle: I tried the trickle command, but it isn't compatible with golang executables (see this question)

Solution

  • It turns out that you can wrap the Golang process in a Docker container and then rate-limit the Docker container's network interface (from inside of the container) with the tc utility. See this answer for an example Dockerfile to do just that.