how to abort python process on memory allocation failure

Question

Python may raise out-of-memory MemoryError exception but is there a way to just let the process terminate if malloc() return NULL?

I'm using some third-party python program do not want the program to handle MemoryError, just want the process to exit fast.

So what I'm looking for is some approach to let the python runtime terminate directly without issuing an memory-error exception.

Answer 1

A MemoryError commonly means that the interpreter has run out of memory to allocate for the program (function) executing. We both know that MemoryErrors are usually an indication that there is likely an issue within your code base.

I'm going to assume that you have a greedy piece of code that is robbing you of memory.

As mentioned in the comments and in Teddy's answer you can catch the MemoryError and exit the program.

import sys

try:
    # your greedy piece of code 

except MemoryError:
    sys.exit()

Another option would be to explicitly use garbage collection, which might help curtail the MemoryError. By using gc.collect you are forcing the garbage collector to release unreferenced memory.

import gc 

# your greedy piece of code 
gc.collect()

Another option would to look at limiting the CPU usage for your application, which might alleviate the MemoryError.

Here are the references:

• Python signal
• Python resource

And here is some sample code that can be modified.

import signal 
import resource 
import os 
 
# checking time limit exceed 
def time_exceeded(signo, frame): 
    print("Time's up !") 
    raise SystemExit(1) 
 
def set_max_runtime(seconds): 
    # setting up the resource limit 
    soft, hard = resource.getrlimit(resource.RLIMIT_CPU) 
    resource.setrlimit(resource.RLIMIT_CPU, (seconds, hard)) 
    signal.signal(signal.SIGXCPU, time_exceeded) 
 
# max run time of 15 millisecond 
if __name__ == '__main__': 
    set_max_runtime(15) 
    while True: 
        pass

Without knowing more about your code troubleshooting this problem isn't easy.

For example, maybe the issue is related to a function calling a dataset that is too large to be processed in a loop

or maybe adding some threading to the function would resolve the MemoryError

There are lots of unknowns...

Based on my comment about calling your problematic script from another script. I looked into this and discovered this answer by 0xPrateek

0xPrateek suggested this:

import os

result = os.system('python other_script.py')
if 0 == result:
    print(" Command executed successfully")
else:
    print(" Command didn't executed successfully")

and this:

# Importing subprocess 
import subprocess

# Your command 
cmd = "python other_script.py"

# Starting process
process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE.PIPE)

# Getting the output and errors of the program
stdout, stderr = process.communicate()

# Printing the errors 
print(stderr)

I'm unsure how his code will work in your environment, but it is worth a try.

I have been trying to reproduce a MemoryErrors issue on my Mac. When my example doesn't use the exception MemoryErrors it exits fine from the other script. If I use the exception things become hairy and makes my MAC unstable.

Is the third party script that you cannot modify public?