quit 1000 of QThread - pyqt5

Question

I have python gui app using Qt

I'm using pyqt5

the app should creating 1000 or more Qthreads, each thread will use pycurl to open external URL

I'm using this code to open the threads:

self.__threads = []
# workerClass thread
for i in range(1000):
    workerInstance =  workerClass()
    workerInstance.sig.connect(self.ui.log.append)
    thread = QThread()
    self.__threads.append((thread, workerInstance))
    workerInstance.moveToThread(thread)
    thread.started.connect(workerInstance.worker_func)
    thread.start()

WorkerClass will use a simple pycurl to visit an external URL and emit signal to add some info on the log, here is the code:

class workerClass(QObject):
    x = 1
    sig = pyqtSignal(str)

    def __init__(self, linksInstance, timeOut):
        super().__init__()
        self.linksInstance = linksInstance
        self.timeOut = timeOut

    @pyqtSlot()
    def worker_func(self):
        while self.x:
            # run the pycurl code
            self.sig.emit('success!') ## emit success message

Now the problem is with stopping all this, I used the below code for stop button:

def stop_func(self):
    workerClass.x = 0
    if self.__threads:
        for thread, worker in self.__threads:
            thread.quit()
            thread.wait()

changing the workerClass.x to 0 will stop the while self.x in the class workerClass and quit and wait should close all threads

all this is working properly, BUT only in the low amount of threads, 10 or may be 100 but if I rung 1000 threads, it takes much time, I wait for 10 minutes but it didn't stopped (mean threads not terminated), however the pycurl time out is only 15 seconds

I also used: self.thread.exit() self.thread.quit() self.thread.terminate() but it gives no difference

if any one have any idea about this, this will be really great :)

Answer 1

after some tries, I removed the pycurl code and test to start 10k QThreads and stopped it, and this working perfect without any errors. so I define that the error is with the pycurl code.

as per the advice of ekhumoro in the question comments, I may try to use pycurl multi

but for now I added this line to the pycurl code:

c.setopt(c.TIMEOUT, self.timeOut)

I was using only this:

c.setopt (c.CONNECTTIMEOUT, self.timeOut)

so now with the both 2 parameters all is properly working, starting and stopping bulk qthreads