Search code examples
pythonmultithreadingcpupython-multithreading

Python - Is this threading / looping / waiting code bad - It 'pegs' cpu


I have a python application where thread1 calls an api to see 'what reports are ready to download' and sends that report_id to thread2 which 'downloads/processes those reports. these threads iterate over a dictionary and then wait 5 minutes. Even while code is 'doing nothing' it is pegging on CPU.

I am not sure if the CPU is being pegged in a thread or in main. In main I have some handlers to check for a stop signal so I posted most of that code. I have a few threads that do similar tasks with similar ways they wait at end of loop. Areas that I suspect that code relate to pegging of cpu a)In main - the while run: pass. b)In each of the threads is_killed = self._kill.wait(600) and that being a theading.Event()

Any idea what is pegging cpu.

if __name__ == '__main__':

t2 = ProcessReport()
t2.start()

t1 = RequestReport(t2)
t1.start()

t3 = report_test()
t3.start()

t4 = run_flask()
t4.start()

run = True
signal.signal(signal.SIGINT, handler_stop_signals)
signal.signal(signal.SIGTERM, handler_stop_signals)
signal.signal(signal.SIGHUP, handler_stop_signals)

while run:
    pass  # Stay here until kill

print("About to kill all threads in clean order")
t3.kill()
t3.join()  
t1.kill()
t1.join() 
t2.kill()
t2.join()
print("Clean Exit")
sys.exit()

Signal Handler

def handler_stop_signals(signum, frame):
    global run
    run = False

One of the threads

class report_test(Thread):
  def __init__(self):
    Thread.__init__(self)
    self._kill = threading.Event()

def kill(self):
    self._kill.set()

def run(self):

    while True:
        cursor.execute("SELECT * FROM tbl_rpt_log where cron='1'")
        reports_to_run = cursor.fetchall()

        for row in reports_to_run:
            report_on_row(row)

        is_killed = self._kill.wait(600)
        if is_killed:
            print("Killing - ReportCheckTable")
            break

Solution

  • The problem (the main one, at least) is:

    while run:
        pass  # Stay here until kill
    

    This is because the only operation here is evaluating loop's condition and the CPU "can't catch a break".

    I didn't look through the whole code, to understand it high level, but the quickest way to work around it is:

    import time
    
    # ...
    
    while run:
        time.sleep(0.1)  # Stay here until kill