How to compress Twitter streaming using LZO in Linux/Python/Tweepy environment?

Question

I'm receiving huge amounts of data streaming from Twitter using Tweepy (a Python Twitter API library). What I want to do is to compress the stream of received tweets and store them in file.

The compression must be LZO and I don't want to use Linux pipes for compression. I want to use LZO directly from the Python code. Using Linux pipes I could do:

Python downloader.py | lzop -c > output.json.lzo

But I don't want to use pipes and want to compress the stream within the Python script downloader.

I couldn't find any Python library or sample code to compress streaming data using LZO.

Answer 1

Two options:

1. use the library.

2. if for some reason you cannot use the library, the following code is an equivalent of the one you wrote:

   from subprocess import Popen, PIPE, STDOUT
   
   p = Popen(['lzop', '-c'], stdout=PIPE, stdin=PIPE, stderr=STDOUT)    
   result_stdout = p.communicate(input=json.dump(results))[0]