Search code examples
datastaxdatastax-enterprisedatastax-enterprise-graph

Specify worker pool for Apache TinkerPop's Spark-Gremlin - DataStax-Enterprise Graph-Analytics

I need to designate a specific worker pool to run gremlin olap queries. When I run gremlin olap queries using gremlin console or datastax studio it runs under the default pool (which is not what I want). I want to run the gremlin olap queries under a specific worker pool e.g. gremlin_olap or be able to specify the memory and executors. I tried a few settings in dse.yaml (in the location resources/dse/conf) and olap.properties (in the location resources/graph/conf), I re-started the cluster still not able to force to use gremlin_olap worker pool.

olap.properties

spark.scheduler.pool=gremlin_olap
spark.executor.cores=2
spark.executor.memory=2g

dse.yaml

resource_manager_options:
    worker_options:
        cores_total: 0.7
        memory_total: 0.6

        workpools:
            - name: alwayson_sql
              cores: 0.25
              memory: 0.25
            - name: gremlin_olap
              cores: 0.25
              memory: 0.25

Gremlin console bin/dse gremlin-console

         \,,,/
         (o o)
-----oOOo-(3)-oOOo-----
plugin activated: tinkerpop.server
plugin activated: tinkerpop.tinkergraph
gremlin> :remote config alias g identity.a
==>g=identity.a
gremlin> g.V().groupCount().by(label)
==>{identity=50000}
gremlin>

Spark Master UI

Am I missing something?

Solution

  • These directions should help:

    https://docs.datastax.com/en/dse/6.8/dse-dev/datastax_enterprise/graph/graphAnalytics/graphAnalyticsSparkGraphComputer.html#SettingSparkpropertiesfromGremlin

    This doesn’t exactly create a Spark resource pool — but it does affect the resources that the Gremlin OLAP Spark application will use — and the way it works in DSE Graph is that there will only ever be one of these applications spun up, so it has the same effect as having a Spark resource pool.