Search code examples
pythonpython-2.7gopublish-subscribe

Simplest pub-sub for golang <--> python communication, possibly across machines?

I'm working on a web application written in Golang that needs to call a Python program/module to do some heavy work. Since that is very memory/CPU intensive, it may be on a separate machine. Since Golang and Python can't talk directly, there are 3 ways to achieve this:

  1. Just execute the python program as an OS process from Go (if on same machine) (or RPC?)
  2. Wrap Python process in a service and expose it for it to be called from Go (may be a simple CRUD like service - A Bottle/flask restful service)
  3. Have a simple pub-sub system in place to achieve this (Redis or some MQ system) - Adding Redis based caching is on the radar so maybe a good reason to go this way. Not sure.

The main thing is that the python process that takes really long to finish must "inform" the web application that it has finished. The data could either be in a file/DB or 'returned' by the process.

What could be the simplest way to achieve this in a pub/sub like environment?

UPDATE REST seems like one way but would incur the cost of implementing server side push which may or may not be easily doable with existing micro web frameworks. The pub/sub would add an additional layer of complexity for maintainability and a learning curve nevertheless. I'm not sure if an RPC like invocation could be achieved across machines. What would be a good choice in this regard?

Solution

  • Use ZeroMQ

    Python has nice pyzmq

    GO bindngs exists too.

    ZeroMQ code is typically short and effective.

    You can even use local socket on Unix.

    There are even other messaging patterns apart from pub/sub, you can use req/resp or push/pull.

    Not knowing much about your applications, I can offer sample of integrating lock manager using zeromq in Python

    Using ZeroRPC

    There is experimental Golang lib being able communicating with Python and Node.js code via ZeroMQ as shown (for Python and Node.js) zerorpc.dotclould.com

    Writing functions in Python being callable over ZeroRPC is extremely simple - in fact you do not have to write a line using zeromq (there is command line tool, which allows integrating functions with properly shaped signatures).