Search code examples
javajvmprestotrino

Encountered too many errors talking to a worker node with trino

Facing this issue using while running a single large query. Can we kill such queries before this error occurs? io.trino.operator.PageTransportTimeoutException: Encountered too many errors talking to a worker node. The node may have crashed or be under too much load. This is probably a transient issue, so please retry your query in a few minutes. (http://172.22.66.206:8889/v1/task/20230727_083615_00032_edi7s.0.0.0/results/0/0 - 30 failures, failure duration 302.86s, total failed request time 312.86s)

3 node cluster m6g.16xlarge (coordinator and 2 worker)

node-scheduler.include-coordinator=false
discovery.uri=http://ip-172-22-69-150.ec2.internal:8889
http-server.threads.max=500
sink.max-buffer-size=1GB
query.max-memory=3000GB
query.max-memory-per-node=60GB
query.max-history=40
query.min-expire-age=30m
query.client.timeout=30m
query.stage-count-warning-threshold=100
query.max-stage-count=150
http-server.http.port=8889
http-server.log.path=/var/log/trino/http-request.log
http-server.log.max-size=67108864B
http-server.log.max-history=5
log.max-size=268435456B
jmx.rmiregistry.port = 9080
jmx.rmiserver.port = 9081
node-scheduler.max-splits-per-node = 200
experimental.query-max-spill-per-node = 50GB
graceful-shutdown-timeout = 3600s
task.concurrency = 16
query.execution-policy = phased
experimental.max-spill-per-node = 100GB
query.max-concurrent-queries = 20
query.max-total-memory = 5000GB
Solution

  • I had the following flags in the jvm config. -XX:+HeapDumpOnOutOfMemoryError -XX:OnOutOfMemoryError=kill -9 %p

    due to this whenever there was an OOM the process would dump its heap onto disk and kill the process. The worker nodes disk utilization was too high (>95%) which was preventing the trino process to start.

    After looking around I figured the OOM issue was due to this issue :- https://bugs.openjdk.org/browse/JDK-8293861

    enter image description here

    To fix this I added the following jvm props -XX:+UnlockDiagnosticVMOptions -XX:-G1UsePreventiveGC

    This prevents the process from going into OOM due to GC