How to do log tailing in realtime efficiently and in an unblocking manner in tornado python

Question

I have a little webapp in a tornado based python project and i need to implement realtime log tailing (one of the few things i am stuck at). The behaviour shoudl be similar to unix's tail -f. It woudl be good if it worked on all platforms but for starters just unix is good enough.

I searched a lot of ways to do it on stackoverflow and else where but didnt find what i am looking for. Pythonic solutions are not good in keeping track of rotating logs or file being inaccessible sometimes.

Therefore i went the python subprocess way. However i could not use the tornado's Subprocess class due to lack of examples. So i have tried normal subprocess with run_in_executor approach. i am not sure if this is a good way to do it or if i will have future issues.

def tail(self, file):
        self.__process = subprocess.Popen(["tail", "-n", "1", "-f", file], stdout=subprocess.PIPE)

        while 1:
            line = self.__process.stdout.readline()
            print(line)
            # call a sync marked method and push line data
            asyncio.sleep(.0001)
            if not line:
                asyncio.sleep(.5)


    async def start_tail(self):
        # start here
        tornado.ioloop.IOLoop.current().run_in_executor(self.executor, self.tail, self.__log_path)
        pass

the trouble here is that i need to push line to a queue. And that queue is in a function marked async. To call a async method it says calling method should also be async. In that case i end up with an error : coroutines cannot be used with run_in_executor. so i am confused on how to get this done.

I would want to the log tailing to work as it does with standard linux tail -f command. It should not block my Tornado loop from other stuff that is going on (such as web requests, websocket messaging etc etc). I should be abel to send line data to any sync or async method in my codebase.

Answer 1

Use tornado.process.Subprocess instead of subprocess.Popen (and its STREAM option instead of PIPE). This lets you read from the subprocess asynchronously:

async def tail(self, file):
    self.__process = Subprocess(["tail", "-n", "1", "-f", file], stdout=Subprocess.STREAM)
    while True:
        line = await self.__process.stdout.read_until(b"\n")
        do_something(line)

def start_tail(self):
    IOLoop.current().spawn_callback(self.tail, self.__log_path)