In PyQt6 I have a custom QTableView using a model of about 18000 rows, which is passed through several chained custom QSortFilterProxy steps.
I would like to paginate the results and the only method I've found to do this is another QSortFilterProxy based on the row number in the source Model. This works fine with the unfiltered source model but once I put it at the end of the filter chain it starts behaving very badly. Only small portions of the intended results are showing up. My guess is it's filtering based on the sourceModel row number rather than the filtered version of said model.
How do I get it to paginate the filtered results properly? Or get the indexes of the visible rows? Or is there another way to do it that doesn't rely on yet another filter? The user should be able to scroll to first, prev, next, last, or jump to any specific page of their choice.
Sample proxy chain:
self.model = CustomTableModel(dataTable, self.labels)
self.proxyModelA = TextFilterProxy(0)
self.proxyModelA.setSourceModel(self.model)
self.proxyModelB = DictValueProxy(5, "Name")
self.proxyModelB.setSourceModel(self.proxyModelA)
self.paginateProxy = PaginationProxy(maxrows=30)
self.paginateProxy.setSourceModel(self.proxyModelB)
self.tableView.setModel(self.paginateProxy)
Current Filter:
class PaginationProxy(QtCore.QSortFilterProxyModel):
def __init__(self, maxrows=30, parent=None):
super(PaginationProxy, self).__init__(parent)
self.maxrows = maxrows
def filterAcceptsRow(self, row_num, parent):
regex = self.filterRegularExpression()
pattern = regex.pattern()
if pattern:
firstRow = (int(pattern) - 1) * self.maxrows
lastRow = firstRow + self.maxrows + 1
else:
firstRow = 0
lastRow = self.maxrows + 1
if row_num in range(firstRow, lastRow):
return True
return False
If you want to deal with logical values such as row numbers, using the filter of the proxy is wrong, because it is based on strings: it's completely pointless to use a string (even worse, a regex pattern) to represent a number that needs to be used for logical purposes such as this.
The above is also a terrible choice especially if implemented like in your example, because the conversion is done for each call to filterAcceptsRow(), meaning that you are doing all the following for every single row of the source model:
range() object;This is completely ineffective and terribly inefficient, especially for large models (which, coincidentally, is exactly your case).
Instead, you should create a custom function that sets the reference page using actual row numbers; once the reference page is chosen, you only need to create a single range() object just once, making it as an instance attribute, and finally invalidate the filter.
Then, filterAcceptsRow() will just return whether the row number is within that range (return row_number in self.someRange).
A simple solution, which can be used for a static source model, is to just create a QSortFilterProxyModel subclass that creates the range object for the selected page, and override filterAcceptsRow() as explained above. For example:
class SimplePaginationModel(QSortFilterProxyModel):
def __init__(self, maxRows=50, parent=None):
super().__init__(parent)
self.maxRows = max(1, maxRows)
# the model is empty, the row range is therefore invalid
self.rowRange = range(0)
def setCurrentPage(self, page):
if not self.sourceModel():
return
rowCount = self.sourceModel().rowCount()
if rowCount <= 0:
firstRow = rangeEnd = 0
else:
firstRow = page * self.maxRows
if firstRow >= rowCount:
firstRow = (rowCount - 1) // self.maxRows * self.maxRows
rangeEnd = firstRow + self.maxRows
if rangeEnd > rowCount:
rangeEnd = rowCount
# create the row range just once, as an instance attribute
self.rowRange = range(firstRow, rangeEnd)
self.invalidateFilter()
def filterAcceptsRow(self, row, parent):
# that's it
return row in self.rowRange
def setSourceModel(self, model):
super().setSourceModel(model)
if model:
self.setCurrentPage(0)
Unfortunately, this does not suffice, because the source model could reduce its shape (by filtering) without triggering the filter invalidation of the proxy: the result is that you get less rows than you should, because the row range has not been updated accordingly; despite the issues of your implementation, this is the cause of the problem you mentioned.
That situation is a symptom of a valid optimization.
When rows are removed from the source model, there is no point in going through the whole model in order to filter its contents again: all previous contents have been already filtered, so, if the source has removed some rows, it's obviously faster to remove the related rows of the proxy.
When that happens, the proxy just removes the related rows, without invalidating the filter, so filterAcceptsRow() is not being called and you get only the rows in the previously accepted range that "survived" the filtering of the source. This is obviously problematic for a pagination system, because the row range of the proxy is based on the row count of the source; for example, if a row that is shown in the current page is being removed, then a new row from the next page should probably be shown; if the removed row is before the current page, the last row in the previous page should be inserted, while the last row in the current page should be removed.
This means that a pagination model should always invalidate the filter (and cause filterAcceptsRow() calls) whenever the shape of the source model has been filtered, causing rows removal.
A possible workaround is to connect the rowsInserted and rowsRemoved signals of the source model to invalidateFilter().
class SimplePaginationModel(QSortFilterProxyModel):
...
def setSourceModel(self, model):
super().setSourceModel(model)
if model:
model.rowsInserted.connect(self.invalidateFilter)
model.rowsRemoved.connect(self.invalidateFilter)
self.setCurrentPage(0)
This would still be not sufficient, though: if the source is filtered (resulting in less pages), we could end up with an empty model if the previously chosen page was beyond the visible range.
Moreover, we probably want a model that always reflects the amount of available pages reliably, including the possibility to switch to next pages.
It's clear that the above pagination may be fine for basic situations, but it won't be enough to cover more complex models, especially when trying to "chain" proxy models, because there's no way to know how any intermediate model (including the actual source at the end of the chain) eventually alters its contents, so we must implement a mechanism that takes into account all changes to the source model of the proxy by connecting to the relevant signals: other than the two mentioned above, we should also consider the layoutChanged and modelReset signals, but we also should keep track of the previous state of the model before the change (to notify about changes to page count or current page number), meaning that we also need to connect to the "about to be" related signals:
rowsAboutToBeInsertedrowsAboutToBeRemovedlayoutAboutToBeChangedmodelAboutToBeResetThese 4 signals can be connected to a simple function that stores the current page count and number, while the related signals (emitted by the model after it has changed) are connected to a function that gets the new page count and possible current number (possibly decreased if less pages are available), compares them with the previous stored state and eventually emit related signals, but only after calling invalidateFilter(), so that any change in the proxy would properly reflect the current page count and number.
Once all this is done, the page navigation becomes quite easy, including going to the previous or next page, jumping to the first or last page, and even providing signals that notify when a page (or layout) change causes or prevents the possibility to go backward or forward.
The following is a complete implementation of the above, including an example that shows its behavior:
from random import choice
from PyQt5.QtCore import *
from PyQt5.QtGui import *
from PyQt5.QtWidgets import *
class PaginationModel(QSortFilterProxyModel):
_currentPage = -1
currentPageChanged = pyqtSignal(int)
pageCountChanged = pyqtSignal(int)
canGoBackChanged = pyqtSignal(bool)
canGoForwardChanged = pyqtSignal(bool)
def __init__(self, maxRows=50, parent=None):
super().__init__(parent)
self._maxRows = max(1, maxRows)
self.rowRange = range(0)
def _emitCanChange(self, oldBack, oldFwd):
newBack = self.canGoBack()
if oldBack != newBack:
self.canGoBackChanged.emit(newBack)
newFwd = self.canGoForward()
if oldFwd != newFwd:
self.canGoForwardChanged.emit(newFwd)
def _beginSourceChange(self):
self._preChangeData = (
self.pageCount(),
self._currentPage,
self.canGoBack(),
self.canGoForward(),
)
def _endSourceChange(self):
oldPageCount, oldPage, oldBack, oldFwd = self._preChangeData
newRowCount = self.sourceModel().rowCount()
if not newRowCount:
newPageCount = firstRow = rangeEnd = 0
self._currentPage = -1
else:
newPageCount = newRowCount // self._maxRows + 1
if oldPage < 0:
self._currentPage = firstRow = 0
else:
firstRow = oldPage * self._maxRows
if firstRow >= newRowCount:
firstRow = (newRowCount - 1) // self._maxRows * self._maxRows
rangeEnd = firstRow + self._maxRows
if rangeEnd > newRowCount:
rangeEnd = newRowCount
self._currentPage = newRowCount // self._maxRows
self.rowRange = range(firstRow, rangeEnd)
self.invalidateFilter()
if oldPageCount != newPageCount:
self.pageCountChanged.emit(newPageCount)
if oldPage != self._currentPage:
self.currentPageChanged.emit(self._currentPage)
self._emitCanChange(oldBack, oldFwd)
def _setCurrentPage(self, page, force=False):
if not self.sourceModel():
return
rowCount = self.sourceModel().rowCount()
if rowCount <= 0:
firstRow = rangeEnd = 0
else:
firstRow = page * self._maxRows
if firstRow >= rowCount:
firstRow = (rowCount - 1) // self._maxRows * self._maxRows
rangeEnd = firstRow + self._maxRows
if rangeEnd > rowCount:
rangeEnd = rowCount
self.rowRange = range(firstRow, rangeEnd)
newPage = firstRow // self._maxRows
if self._currentPage != newPage or force:
oldBack = self.canGoBack()
oldFwd = self.canGoForward()
self._currentPage = newPage
self.invalidateFilter()
self.currentPageChanged.emit(newPage)
self._emitCanChange(oldBack, oldFwd)
def maxRows(self):
return self._maxRows
def setMaxRows(self, maxRows):
if maxRows <= 0 or self._maxRows == maxRows:
return
if not self.sourceModel():
self._maxRows = maxRows
return
oldPageCount = self.pageCount()
oldBack = self.canGoBack()
oldFwd = self.canGoForward()
self._maxRows = maxRows
self.invalidateFilter()
self._setCurrentPage(max(0, self._currentPage), True)
newPageCount = self.pageCount()
if oldPageCount != newPageCount:
self.pageCountChanged.emit(newPageCount)
self._emitCanChange(oldBack, oldFwd)
def canGoBack(self):
return self._currentPage > 0
def canGoForward(self):
return self._currentPage < self.pageCount() - 1
def firstPage(self):
self.setCurrentPage(0)
def previousPage(self):
self.setCurrentPage(self._currentPage - 1)
def nextPage(self):
self.setCurrentPage(self._currentPage + 1)
def lastPage(self):
self.setCurrentPage(self.sourceModel().rowCount() // self._maxRows)
def pageCount(self):
if self.sourceModel():
pages, rest = divmod(self.sourceModel().rowCount(), self._maxRows)
if rest:
pages += 1
return pages
return 0
def currentPage(self):
return self._currentPage
def setCurrentPage(self, page):
if self.sourceModel() and page >= 0 and self._currentPage != page:
self._setCurrentPage(page)
def filterAcceptsRow(self, row, parent):
return row in self.rowRange
def setSourceModel(self, model):
old = self.sourceModel()
if old == model:
return
if old:
old.rowsAboutToBeRemoved.disconnect(self._beginSourceChange)
old.rowsAboutToBeInserted.disconnect(self._beginSourceChange)
old.layoutAboutToBeChanged.disconnect(self._beginSourceChange)
old.modelAboutToBeReset.disconnect(self._beginSourceChange)
old.rowsRemoved.disconnect(self._endSourceChange)
old.rowsInserted.disconnect(self._endSourceChange)
old.layoutChanged.disconnect(self._endSourceChange)
old.modelReset.disconnect(self._endSourceChange)
self._currentPage = -1
super().setSourceModel(model)
if not model:
self.rowRange = range(0)
else:
model.rowsAboutToBeRemoved.connect(self._beginSourceChange)
model.rowsAboutToBeInserted.connect(self._beginSourceChange)
model.layoutAboutToBeChanged.connect(self._beginSourceChange)
model.modelAboutToBeReset.connect(self._beginSourceChange)
model.rowsRemoved.connect(self._endSourceChange)
model.rowsInserted.connect(self._endSourceChange)
model.layoutChanged.connect(self._endSourceChange)
model.modelReset.connect(self._endSourceChange)
self._setCurrentPage(max(0, self._currentPage))
class Example(QWidget):
def __init__(self, rowCount=2000):
super().__init__()
self.sourceModel = QStandardItemModel(rowCount, 1)
letters = 'abcdefghijklmnopqrstuvwxyz'
for i in range(rowCount):
self.sourceModel.setItem(i, 0, QStandardItem(
''.join(choice(letters) for _ in range(8))))
table = QTableView()
table.horizontalHeader().setSectionResizeMode(QHeaderView.Stretch)
table.horizontalHeader().hide()
self.filterModel = QSortFilterProxyModel()
self.filterModel.setSourceModel(self.sourceModel)
self.pageModel = PaginationModel()
self.pageModel.setSourceModel(self.filterModel)
table.setModel(self.pageModel)
textFilterEdit = QLineEdit()
topLayout = QHBoxLayout()
topLayout.addWidget(QLabel('Filter:'))
topLayout.addWidget(textFilterEdit)
topLayout.addWidget(QLabel('Max rows:'))
maxRowSpin = QSpinBox(
minimum=1, maximum=self.sourceModel.rowCount(), singleStep=5)
maxRowSpin.setValue(self.pageModel.maxRows())
topLayout.addWidget(maxRowSpin)
self.firstBtn = QToolButton(text='First')
self.prevBtn = QToolButton(text='Prev')
self.pageSpin = QSpinBox(minimum=1)
self.nextBtn = QToolButton(text='Next')
self.lastBtn = QToolButton(text='Last')
pageLayout = QHBoxLayout()
pageLayout.addWidget(self.firstBtn)
pageLayout.addWidget(self.prevBtn)
pageLayout.addWidget(self.pageSpin)
pageLayout.addWidget(self.nextBtn)
pageLayout.addWidget(self.lastBtn)
layout = QVBoxLayout(self)
layout.addLayout(topLayout)
layout.addLayout(pageLayout)
layout.addWidget(table)
self.pageSpin.setMaximum(self.pageModel.pageCount())
self.pageSpin.setSuffix('/{}'.format(self.pageModel.pageCount()))
textFilterEdit.textEdited.connect(self.filterModel.setFilterWildcard)
maxRowSpin.valueChanged.connect(self.pageModel.setMaxRows)
self.firstBtn.clicked.connect(self.pageModel.firstPage)
self.prevBtn.clicked.connect(self.pageModel.previousPage)
self.nextBtn.clicked.connect(self.pageModel.nextPage)
self.lastBtn.clicked.connect(self.pageModel.lastPage)
self.pageSpin.valueChanged.connect(
lambda p: self.pageModel.setCurrentPage(p - 1))
self.pageModel.currentPageChanged.connect(self.currentPageChanged)
self.pageModel.pageCountChanged.connect(self.pageCountChanged)
self.pageModel.canGoBackChanged.connect(self.updateButtons)
self.pageModel.canGoForwardChanged.connect(self.updateButtons)
self.updateButtons()
def currentPageChanged(self, page):
with QSignalBlocker(self.pageSpin):
self.pageSpin.setValue(page + 1)
def pageCountChanged(self, count):
with QSignalBlocker(self.pageSpin):
self.pageSpin.setMaximum(count)
if count:
self.pageSpin.setMinimum(1)
self.pageSpin.setSuffix('/{}'.format(count))
def updateButtons(self):
canBack = self.pageModel.canGoBack()
self.firstBtn.setEnabled(canBack)
self.prevBtn.setEnabled(canBack)
canFwd = self.pageModel.canGoForward()
self.nextBtn.setEnabled(canFwd)
self.lastBtn.setEnabled(canFwd)
app = QApplication([])
main_window = Example()
main_window.show()
app.exec()
The above signal connections show how important to follow the appropriate implementation in its order: correctly call model functions and emit all their required signals, respecting the logical context of any change.
This is extremely important when implementing custom models, as a common mistake is to just emit the "final" (or wrong) signals whenever the contents of the model change.
For instance, many sources often suggest to just emit layoutChanged when rows or columns have been added or removed, but that is wrong in principle (and is often cause of crash when the custom model is used on a proxy).
Such changes must always be done in the proper order, with both the related signal pairs: for instance, a row insertion should always be preceded by a correct rowsAboutToBeInserted signal (before the actual insertion) and followed by rowsInserted one after all changes have been applied.
In reality, the existing functions of QAbstractItemModel should always be preferred in such cases; for example, beginInsertRows() and endInsertRows() for the case above.
As usual, going through the documentation is mandatory, starting from the official Qt Model/View Programming guide.