In Qt's QAbstractTableModel, if I set the row count to 71,582,788 the app performs well and scrolling in any direction works as desired. CPU usage is minimal, and memory usage is as expected.
If I add one row, for a total of 71,582,789 rows, the app hangs. This hang happens whether I set the row count during initialization (as shown below) or after.
I have tested this on Mac OS and windows, the same exact limit exists on both.
Column count appears to have no impact on this issue.
from PySide6.QtWidgets import QTableView, QApplication, QMainWindow, QVBoxLayout, QWidget
from PySide6.QtCore import Qt, QAbstractTableModel
import sys
import time
class TableModel(QAbstractTableModel):
def __init__(self, row_count, col_count, parent=None):
super(TableModel, self).__init__(parent)
self.row_count = row_count
self.col_count = col_count
def rowCount(self, parent=None):
return self.row_count
def columnCount(self, parent=None):
return self.col_count
def data(self, index, role=Qt.ItemDataRole.DisplayRole):
pass
class MainWindow(QMainWindow):
def __init__(self):
super().__init__()
self.setWindowTitle("Table Example")
row_count = 71582789
col_count = 300
model = TableModel(row_count, col_count, self)
table_view = QTableView(self)
table_view.setModel(model)
layout = QVBoxLayout()
layout.addWidget(table_view)
container = QWidget()
container.setLayout(layout)
self.setCentralWidget(container)
# Change the app's initial size
self.resize(800, 600)
if __name__ == "__main__":
app = QApplication(sys.argv)
timestart = time.time()
window = MainWindow()
print(time.time()-timestart)
window.show()
print(time.time()-timestart)
sys.exit(app.exec())
I expected performance degradation to happen gradually as row count increased, but the sudden hang at an arbitrary row count across different operating systems seems to be a limit set by the Qt internals.
This issue is a variant of the one covered by this question:
All internal calculations performed within the model/view architecture are made using signed ints, which means that on most platforms, the maximum value will equate to 2,147,383,647. This imposes a hard upper limit on the number of items that can be displayed at the same time, since an overflow error will occur for any values greater than this.
The difference with the current case is that the limit is restricted even further by the vertical header, which must make internal calculations that also include the row height. On my machine, the default row height is 30 pixels, which results in the following equation: 2147383647 // 30 == 71582788. Thus, a row count of 71582788 can be displayed without issue, but if you then attempt to manually resize any row a few pixels larger, the app will immediately hang. This is because some internal int variable has overflowed to a negative value, which will very likely create an infinite loop somewhere in the header-view calculations when the view is updated.
As a consequence of this, a simple fetch-more approach won't help, since the header would still eventually need to create the same number of sections. In addition, it would also be necessary to prevent row resizing and set a fixed section size (since the default depends on the style which will very likely calculate it from the current font-metrics). These sorts of considerations mean some sort of paging approach would be much more robust, since that would make it much easier to keep within the hard limits (and if the page-size is kept reasonably small, it would have the additional benefit of reducing the loading time and memory consumption). More generally speaking, it's always advisable to think carefully about how much data the user really needs to interact with at one time. There are usually other ways of presenting the data that don't require filling the entire view.
PS: it seems there may be some pending changes to Qt6 landing in the near future that could eliminate some of the above issues. For more details, see the following discussion: