Using NatTable for large data which is slow to retrieve

Question

I got a question about the NatTable SWT datagrid widget which I want to use to display a huge log file.

As the log file can be several GB in size, I cannot load it in memory entirely. The application should use as little heap space as possible. Thus, my IDataProvider implements its getDataValue method to read a log file line on the fly, with the help of a little caching mechanism and a cache for raw line positions within the file. When opening a file, almost nothing is known about its contents - not even the amount of rows the table will eventually have to display.

This leads to several problems which currently freak me out:

1) I have to estimate the total row count (using average line length) until the file has been scanned entirely and a maximum line count is really known for sure. As soon this is the case my getRowCount method returns the actual, correct number of rows. This seems to freak out the NatTable widget, it loses its current position entirely and jumps to row # 1!

2) When doing something intuitive as double-clicking on the border of a column (to auto-size the column, just like in Excel), my application freezes completely as suddenly ALL rows cells contents is queried using my provider and the whole concept is abused! I did not want anything to trigger a full file read!

Does anybody have some hints for me? The documentation is so little and so bad... Any hint is greatly appreciated! How can I prevent my application from freezing??

Thanks!

Answer 1

1) Sounds like the selection isn't preserved when you change the row count - a row count change implies a backing data change hence it's probably playing safe and setting the selection to the first row. You could query the selected row before changing the row count and then reset the selection after you change the row count.

2) In order to figure out what the max-width of the column is, the table has to query each value for that cell in every row... Either you have to implement a paging mechanism such that you return something resulting in zero width when the row isn't visible, or you'll probably want to raise a bug against NatTable to request a feature for an auto-resize mode based on only the visible columns!