Search code examples
c#cubein-memorynreco

Nreco cube cahnge processing

NReco PivotData Cube processing

This component is new to me, and i am curious whether it supports frequent updates to the data source? Normal size of data will be 1000 rows containing 60 facts and 6 dimensions. Is it possible to update single rows of data? Will this trigger full reprocessing, and how long will it take?

Solution

  • NReco.PivotData library implements in-memory multidimensional data structure which holds pre-aggregated metrics for dimensions you want to use in your reports. There are 2 different ways how it can be used:

    • ROLAP-style way (mostly used): PivotData class instance is short-living, it is created and populated 'on-the-fly' at the moment when you need to render pivot table or prepare JSON data for the chart. Data source doesn't matter: this can be in-memory list of models that is processed with PivotData.ProcessData method or DB aggregation query results may be loaded with help of GroupedSourceReader class.
    • for real-time reports on append-only dataset, you can load PivotData class instance once on the application start, and then 'update' it with new records. Depending on your needs, PivotData may keep and pre-calculate totals/totals, or 'lazy' roll-ups may be performed only when total value is accessed (this is controlled with PivotData.LazyTotals flag).

    Regarding

    Normal size of data will be 1000 rows containing 60 facts and 6 dimensions.

    this is very small data, 1k rows is processed by ProcessData method very quickly (only dimensions/measures that are needed for the concrete report are calculated). If you use database as a data source, it should be able to execute aggregate (GROUP BY) queries fast enough; specialized analytical DBs can do that in seconds even for billions of rows.