QProgressBar causing bad performance in QT5?

Question

I'm developping a program which parses a file (365000 lines) in which I try to match some keywords after reading each line. This computation along with the update of my QProgressBar are made in another thread using QThread. Everything works fine except for the performance especially when I update the QProgressBar. I use a timer for the parsing and the result is just STUNNING. When I emit a signal to update the QProgressBar the program takes around 45 seconds but when I do not emit the signal for the QProgressBar update then the program takes around 0.40 sec =/

from PyQt5 import QtCore, QtWidgets, QtGui
import sys
import time

liste = ["failed", "exception"]

class ParseFileAsync(QtCore.QThread):

    match = QtCore.pyqtSignal(str)
    PBupdate = QtCore.pyqtSignal(int)
    PBMax = QtCore.pyqtSignal(int)

    def run(self):        
        cpt = 0
        with open("test.txt", "r") as fichier:
            fileLines = fichier.readlines()  
            lineNumber = len(fileLines)    
            self.PBMax.emit(lineNumber)

            t0 = time.time()
            for line in fileLines:
                cpt+=1
                self.PBupdate.emit(cpt)   
                for element in liste:
                    if element in line:
                        self.match.emit(line)

        finalTime = time.time() - t0
        print("over :", finalTime)

    class Ui_MainWindow(QtWidgets.QMainWindow):

        def __init__(self):
            super().__init__()       
            self.setupUi(self)
            self.thread = ParseFileAsync()

            self.thread.match.connect(self.printError)
            self.thread.PBupdate.connect(self.updateProgressBar)
            self.thread.PBMax.connect(self.setMaximumProgressBar)

            self.pushButton_GO.clicked.connect(self.startThread)

    def printError(self, line):
        self.textEdit.append(line)

    def updateProgressBar(self, value):
        self.progressBar.setValue(value)

    def setMaximumProgressBar(self, value):
        self.progressBar.setMaximum(value)

    def startThread(self):
        self.thread.start()

Console output:

over : 44.49321101765038  //QProgressBar updated
over : 0.3695987798147516 //QProgressBar not updated

Am I missing something or is that expected ?

EDIT :

I followed jpo38 and Matteo very good advices. I update the QProgressBar less frequently. The progression is still smooth and the performance is very good (around one second with this implementation). PSB :

class ParseFileAsync(QtCore.QThread):

match = QtCore.pyqtSignal(str)
PBupdate = QtCore.pyqtSignal(int)
PBMax = QtCore.pyqtSignal(int)

def run(self):        
    with open("test_long.log", "r") as fichier:
        fileLines = fichier.readlines()  
        self.lineNumber = len(fileLines)
        self.PBMax.emit(self.lineNumber)

        if (self.lineNumber < 30):
            self.parseFile(fileLines, False)
        else:
            self.parseFile(fileLines, True)

def parseFile(self, fileLines, isBig):                
        cpt = 0

        if(isBig):
            for line in fileLines:
                cpt+=1             
                if(cpt % (int(self.lineNumber/30)) == 0):
                    self.PBupdate.emit(cpt)        
                for element in liste:
                    if element in line:
                        self.match.emit(line)

            self.PBupdate.emit(self.lineNumber) #To avoid QProgressBar stopping at 99%
        else:         
            for line in fileLines:
                cpt+=1                
                self.PBupdate.emit(cpt)                                  
                for element in liste:
                    if element in line:
                        self.match.emit(line)

Answer 1

Updating a QProgressBar too often will definitely lead to performance issues. You should update the progress bar less often. You don't want/need to do that for every iteration...365000 times. When you read one line out of 365000, you progressed by 0.0002%, no need to update GUI for that...

Showing progression to user always has a cost...and we accept that because user prefers to wait a bit more and have progress information. However, showing progression must not multiply processing time by 100 as you experienced.

You can either emit the signal to update the progress bar only when progression significantly changed (for instance every time percentage value casted to a int changed, you can store a progression as int value to check that...or test if (line%(fileLines/100)==0) for instance...this will significantly decrease the cost of progress bar update).

Or you could start a QTimer to update the progress bar every 100ms for instance. Then you don't emit any signal from the for loop and just save the progression value to be used when timer times out.

If the file size is always 365000 lines, you can also decide to emit the signal every 1000 lines for instance (if line%1000==0). But the two earlier solutions are preferable because they will fix your performance issues whatever the file size is.