I have a weird problem to read from STDIN in a python script.
Here is my use case. I have rsyslog configured with an output module so rsyslog can pipe log messages to my Python script.
My Python script is really trivial :
#! /usr/bin/env python
# -*- coding: utf-8 -*-
import sys
fd = open('/tmp/testrsyslogomoutput.txt', 'a')
fd.write("Receiving log message : \n%s\n" % ('-'.join(sys.stdin.readlines())))
fd.close()
If I run echo "foo" | mypythonscript.py I can get "foo" in the target file /tmp/testrsyslogomoutput.txt. However when I run it within rsyslog, messages seems to be sent only when I stop/restart rsyslog (I believe some buffer is flushed at some point).
I first thought it was a problem with Rsyslog. So I replaced my python program with a shell one, without changing anything to the rsyslog configuration. The shell script works perfectly with rsyslog and as you can see in the code below, the script is really trivial:
#! /bin/sh
cat /dev/stdin >> /tmp/testrsyslogomoutput.txt
Since my shell script works but my Python one does not, I believe I made a mistake somewhere in my Python code but I can not find where. If you could point me to my mistake(s) that would be great.
Thanks in advance :)
I'd also suspect the reason is that rsyslog does not terminate. readlines() should not return until it reaches a real EOF. But why would the shell script act differently? Perhaps the use of /dev/stdin is the reason. Try this version and see if it still runs without hanging:
#!/bin/sh
cat >> /tmp/testrsyslogomoutput.txt
If this makes a difference, you'll also have a fix: open and read /dev/stdin from python, instead of sys.stdin.
Edit: So cat somehow reads whatever is waiting at stdin and returns, but python blocks and waits until stdin is exhausted. Strange. You can also try replacing readlines() with a single read() followed by split("\n"), but at this point I doubt that will help.
So, forget the diagnosis and let's try a work-around: Force stdin to do non-blocking i/o. The following is supposed to do that:
import fcntl, os
# Add O_NONBLOCK to the stdin descriptor flags
flags = fcntl.fcntl(0, fcntl.F_GETFL)
fcntl.fcntl(0, fcntl.F_SETFL, fl | os.O_NONBLOCK)
message = sys.stdin.read().split("\n") # Read what's waiting, in one go
fd = open('/tmp/testrsyslogomoutput.txt', 'a')
fd.write("Receiving log message : \n%s\n" % ('-'.join(message)))
fd.close()
You probably want to use that in combination with python -u. Hope it works!