Search code examples
jdbchiveodbchadoop2amazon-emr

Hive LLAP Daemon Not available over ODBC/JDBC


I have Hive LLAP deployed on a cluster via Slider. The Hive prompt clearly submits jobs via llap. When I attempt to run the same query (any query, really) in beeline via odbc connection, I get an error:

0: jdbc:hive2://localhost:10000/default> select count(1) from testtable limit 1;
ERROR : Status: Failed
ERROR : Dag received [DAG_TERMINATE, SERVICE_PLUGIN_ERROR] in RUNNING state.
ERROR : Error reported by TaskScheduler [[2:LLAP]][SERVICE_UNAVAILABLE] No LLAP Daemons are running
ERROR : Vertex killed, vertexName=Reducer 2, vertexId=vertex_1538061470561_0009_2_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:1, Vertex vertex_1538061470561_0009_2_01 [Reducer 2] killed/failed due to:DAG_TERMINATED]
ERROR : Vertex killed, vertexName=Map 1, vertexId=vertex_1538061470561_0009_2_00, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:1, Vertex vertex_1538061470561_0009_2_00 [Map 1] killed/failed due to:DAG_TERMINATED]
ERROR : DAG did not succeed due to SERVICE_PLUGIN_ERROR. failedVertices:0 killedVertices:2
INFO  : org.apache.tez.common.counters.DAGCounter:
INFO  :    NUM_KILLED_TASKS: 1
INFO  :    AM_CPU_MILLISECONDS: 1380
INFO  :    AM_GC_TIME_MILLIS: 0
ERROR : FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Dag received [DAG_TERMINATE, SERVICE_PLUGIN_ERROR] in RUNNING state.Error reported by TaskScheduler [[2:LLAP]][SERVICE_UNAVAILABLE] No LLAP Daemons are runningVertex killed, vertexName=Reducer 2, vertexId=vertex_1538061470561_0009_2_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:1, Vertex vertex_1538061470561_0009_2_01 [Reducer 2] killed/failed due to:DAG_TERMINATED]Vertex killed, vertexName=Map 1, vertexId=vertex_1538061470561_0009_2_00, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:1, Vertex vertex_1538061470561_0009_2_00 [Map 1] killed/failed due to:DAG_TERMINATED]DAG did not succeed due to SERVICE_PLUGIN_ERROR. failedVertices:0 killedVertices:2
INFO  : Completed executing command(queryId=hive_20180927173233_63d898d4-c137-495d-91de-72482212ce43); Time taken: 60.516 seconds
Error: org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Dag received [DAG_TERMINATE, SERVICE_PLUGIN_ERROR] in RUNNING state.Error reported by TaskScheduler [[2:LLAP]][SERVICE_UNAVAILABLE] No LLAP Daemons are runningVertex killed, vertexName=Reducer 2, vertexId=vertex_1538061470561_0009_2_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:1, Vertex vertex_1538061470561_0009_2_01 [Reducer 2] killed/failed due to:DAG_TERMINATED]Vertex killed, vertexName=Map 1, vertexId=vertex_1538061470561_0009_2_00, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:1, Vertex vertex_1538061470561_0009_2_00 [Map 1] killed/failed due to:DAG_TERMINATED]DAG did not succeed due to SERVICE_PLUGIN_ERROR. failedVertices:0 killedVertices:2

Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Dag received [DAG_TERMINATE, SERVICE_PLUGIN_ERROR] in RUNNING state.Error reported by TaskScheduler [[2:LLAP]][SERVICE_UNAVAILABLE] No LLAP Daemons are runningVertex killed, vertexName=Reducer 2, vertexId=vertex_1538061470561_0009_2_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:1, Vertex vertex_1538061470561_0009_2_01 [Reducer 2] killed/failed due to:DAG_TERMINATED]Vertex killed, vertexName=Map 1, vertexId=vertex_1538061470561_0009_2_00, diagnostics=[Vertex received Kill while in RUNNING state.,Vertex did not succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:1, Vertex vertex_1538061470561_0009_2_00 [Map 1] killed/failed due to:DAG_TERMINATED]DAG did not succeed due to SERVICE_PLUGIN_ERROR. failedVertices:0 killedVertices:2

But I know the daemon is in fact running. Are there different configurations I should use in order to enable the session run via ODBC to access the llap daemon?


Solution

  • I learned that the JDBC/ODBC user runs as the hive unix user by default. In order to access my llap daemon, I need to run the slider application as the hive user. The following invocation does the trick:

    sudo -H -u hive bash -c "${SLIDER_APP}/run.sh"