Search code examples
amazon-emrpresto

presto configuration through emr launch config


I am trying to deploy presto om EMR through our EMR launch config JSON. I have decided the config properties as advised in this github issue of presto. I have added the following presto properties in the launch config

{
    "Classification": "presto-connector-hive",
    "Properties": {
        "hive.metastore.glue.datacatalog.enabled": "true",
        "hive.table-statistics-enabled": "true"
    },
    "Configurations": []
},
{
    "Classification": "presto-config",
    "Properties": {
        "query.max-memory": "150G",
        "query.max-memory-per-node": "20G",
        "query.max-total-memory-per-node": "30G",
        "memory.heap-headroom-per-node": "10G",
        "query.initial-hash-partitions": "15"
    },
    "Configurations": []
}   

EMR cluster has been created but presto is failing due to the following errors

1) Explicit bindings are required and com.facebook.presto.memory.LowMemoryKiller is not explicitly bound.
  while locating com.facebook.presto.memory.LowMemoryKiller
    for parameter 7 at com.facebook.presto.memory.ClusterMemoryManager.<init>(ClusterMemoryManager.java:123)
  at com.facebook.presto.server.CoordinatorModule.setup(CoordinatorModule.java:189) (via modules: com.facebook.presto.server.ServerMainModule -> com.facebook.presto.server.CoordinatorModule)

2) Error: Could not coerce value '150G' to io.airlift.units.DataSize (property 'query.max-memory') in order to call [public com.facebook.presto.memory.MemoryManagerConfig com.facebook.presto.memory.MemoryManagerConfig.setMaxQueryMemory(io.airlift.units.DataSize)]

3) Error: Could not coerce value '20G' to io.airlift.units.DataSize (property 'query.max-memory-per-node') in order to call [public com.facebook.presto.memory.NodeMemoryConfig com.facebook.presto.memory.NodeMemoryConfig.setMaxQueryMemoryPerNode(io.airlift.units.DataSize)]

4) Configuration property 'memory.heap-headroom-per-node' was not used
  at io.airlift.bootstrap.Bootstrap.lambda$initialize$2(Bootstrap.java:234)

5) Configuration property 'query.max-memory' was not used
  at io.airlift.bootstrap.Bootstrap.lambda$initialize$2(Bootstrap.java:234)

6) Configuration property 'query.max-memory-per-node' was not used
  at io.airlift.bootstrap.Bootstrap.lambda$initialize$2(Bootstrap.java:234)

7) Configuration property 'query.max-total-memory-per-node' was not used
  at io.airlift.bootstrap.Bootstrap.lambda$initialize$2(Bootstrap.java:234)

7 errors
    at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:466)
    at com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:155)
    at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:107)
    at com.google.inject.Guice.createInjector(Guice.java:96)
    at io.airlift.bootstrap.Bootstrap.initialize(Bootstrap.java:241)
    at com.facebook.presto.server.PrestoServer.run(PrestoServer.java:114)
    at com.facebook.presto.server.PrestoServer.main(PrestoServer.java:66)

My config.properties file

coordinator=true
node-scheduler.include-coordinator=false
discovery.uri=X.X.X.X:YYYY
http-server.threads.max=500
discovery-server.enabled=true
sink.max-buffer-size=1GB
query.max-memory=150G
query.max-memory-per-node=20G
query.max-history=40
query.min-expire-age=30m
http-server.http.port=8889
http-server.log.path=/var/log/presto/http-request.log
http-server.log.max-size=67108864B
http-server.log.max-history=5
log.max-size=268435456B
log.max-history=5
query.initial-hash-partitions = 15
memory.heap-headroom-per-node = 10G
query.max-total-memory-per-node = 30G

Solution

  • Setup fails because

    1. You need to use "GB" (not "G") as the unit when setting data size config properties
    2. Your version (0.194) doesn't support some properties that you're setting (memory.heap-headroom-per-node nor query.max-total-memory-per-node).