Search code examples
pythonscheduled-tasksjobs

Stopping and restarting RAM-hungry python scripts, while saving data (linux)

I have a python 2.7 script that takes various parameters. For any choice of parameters, a very large amount of RAM is needed, so that I can only run at most one instance of the script at a time.

I would like to have multiple instances with various sets of parameters taking turns every, say, 48 hours, and each time saving some data to the hard disk, so that they can resume later. In other words, let's say that I have two sets of parameters {x1, x2}, and {y1, y2} for which I want to take turns. I would like to achieve something like this:

while True:
    execute myscript.py x1, x2 
    sleep 48 hours
    tell myscript.py to stop and save a bunch of data to a file
    execute myscript.py y1, y2
    sleep 48 hours
    tell myscript.py to stop and save a bunch of data to a file

What should be the best approach here? I am using linux.

Solution

  • Stackless Python is the answer to this, as it can serialize a full python process at any given time & then restart it later.

    http://www.stackless.com/wiki/Pickling