Search code examples
eventsout-of-memorydereferenceesper

Esper is not defrencing objects and leading to oom for aggregated objects


We are Esper To aggregate i.e group by on certain set of events...but esper is not dereferencing that aggregate object.

Esper query:

  private static final String HOURLY_CONTEXT =
  "create context HourlyRollup start(0,*,*,*,*,0) end(59,*,*,*,*,59)";

This is our hourly context...

This bean in query is not getting de-referenced and we are getting gbs of these objects.

  private static final String HOURLY_STATEMENT =
      "context HourlyRollup "
          + "select count(*) as xcount, hourlyFloor(min(from_time)), a, b, c, d, e, f,"
          + "g,h,sum(h),sum(i),j,k,l,"
          + "m,n,y,o,p,q,r "
          + "from io.common.Bean where Dir in (-5,-3,0,1) "
          + "group by a,b,c,d,e,f,g,Direction,h,"
          + "i,j,k,l,m,l,n,o,p output all "
          + "when terminated order by a,b,c,Dir,d,e";
  private static final int HOURLY = RollupPeriod.HOURLY.ordinal();

Solution

  • When a select-clause selects properties of each event that are not in the group-by clause, that means that Esper cannot forget the event itself and keeps the events around until it is time to output and terminate.
    This type of query is http://esper.espertech.com/release-8.2.0/reference-esper/html_single/index.html#processingmodel_aggregation_batch_group_agg

    When a select-clause only has aggregated properties plus un-aggregated properties that appear in the group-by clause that mean Esper can forget the event and keep the aggregated value instead. This type of query is http://esper.espertech.com/release-8.2.0/reference-esper/html_single/index.html#processingmodel_aggregation_batch_full_agg

    So...check the expressions in the select-clause and make sure that event properties are either in the group-by clause, or are all aggregated for example "last(r)" or "first(r)" or similar.