I'm trying to insert to a table with a query in an EMR cluster on AWS. The table is creating correctly, and a colleague can run the exact same code that I'm using and it won't fail. However, when I try to run the code, I get failures in Map1 that make the entire job fail with the error below for the query below.
Can someone help me figure out why my job is failing when I run it, but my friend can run it without issue? I've been staring at this for the entire day and can't get past it.
----------------------------------------------------------------------------------------------
VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
----------------------------------------------------------------------------------------------
Map 1 container RUNNING 13 0 0 13 40 1
Map 3 .......... container SUCCEEDED 1 1 0 0 0 0
Map 5 .......... container SUCCEEDED 1 1 0 0 0 0
Map 7 .......... container SUCCEEDED 1 1 0 0 0 0
Map 8 .......... container SUCCEEDED 1 1 0 0 0 0
Reducer 2 container INITED 6 0 0 6 0 0
Reducer 4 ...... container SUCCEEDED 2 2 0 0 0 0
Reducer 6 ...... container SUCCEEDED 2 2 0 0 0 0
Reducer 9 ...... container SUCCEEDED 2 2 0 0 0 0
----------------------------------------------------------------------------------------------
VERTICES: 07/09 [========>>------------------] 34% ELAPSED TIME: 132.71 s
----------------------------------------------------------------------------------------------
Status: Failed
Vertex failed, vertexName=Map 1, vertexId=vertex_1544915203536_0453_2_07, diagnostics=[Task failed, taskId=task_1544915203536_0453_2_07_000009, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : attempt_1544915203536_0453_2_07_000009_0:java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: java.lang.IllegalArgumentException: [
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1840)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: java.lang.IllegalArgumentException: [VALUE] BINARY is not in the store:
at
So it turns out that the vectorization was the issue. These were the settings that would be activated at the beginning of the session.
set hive.vectorized.execution.enabled = true;
set hive.vectorized.execution.reduce.enabled = true;
By not activating this it was able to run slower but successfully. It seems hive does not like the timestamp value. At the bottom of the below wiki is a limitations piece. It definitely works without these options set.
https://cwiki.apache.org/confluence/display/Hive/Vectorized+Query+Execution
In summary, timestamps and vectorization don't like each other in hive.... But only sometimes...