Search code examples
javabashhiveenvironment-variablesprocessbuilder

NoSuchFieldError when executed from Java but not from command line

Is it possible to have java use the command line's environment instead of its own?

I have a java program calling a bash script with 

final ProcessBuilder pb = new ProcessBuilder("bash", "-x", exportScriptLoc, platformId, oldWorkFlowId,
            newWorkFlowId).inheritIO();
    pb.directory(new File("/opt/nis/ddziak-dev/nis-hadoop"));
    final Process p = pb.start();
    p.waitFor();
    final int exitValue = p.exitValue();
    p.destroy();
    if (exitValue == 0) {

When I execute the script on the command line it behaves as expected. When my java program executes the script it produces the following exception:

Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:347)
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1422)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
    at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2457)
    at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2469)
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:341)
    ... 7 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1420)
    ... 12 more
Caused by: java.lang.NoSuchFieldError: SASL_PROPS
    at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S.getHadoopSaslProperties(HadoopThriftAuthBridge20S.java:127)
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.getMetaStoreSaslProperties(MetaStoreUtils.java:1485)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:322)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:214)
    ... 17 more

The script is as follows

#!/bin/bash

. ~/.bash_profile
export HIVE_HOME=/usr/lib/hive
#export PATH=$PATH:$HIVE_HOME/bin

env
echo "hive -S -e \"show partitions nis.subscribers 
partition(destinationPlatformId='$1', build='$2');\"|awk '{if(NR>1)print}'" >> subscribers.out
PARTITIONS=`/usr/bin/hive -S -e "show partitions nis.subscribers 
partition(destinationPlatformId='$1', build='$2');"|awk '{if(NR>1)print}'`
echo "${PARTITIONS}"
if [ -z "$PARTITIONS" ]
then
  exit 2
fi


echo "${PARTITIONS}"

hqlOut=`/usr/bin/hive -S -hiveconf destPlatId=$1 -hiveconf newWorkFlowId=$3 -hiveconf oldWorkFlowId=$2 -f /opt/nis/ddziak-dev/nis-
hadoop/dapLib/updateSubscriberHiveTable.hql`

newPARTITIONS=`/usr/bin/hive -S -e "show partitions nis.subscribers partition(destinationPlatformId='$1');"|awk '{if(NR>1)print}'`
if [ -z "$newPARTITIONS" ]
then
  exit 3
fi
Solution

  • Is it possible to have java use the command line's environment instead of its own?

    It depends on what you mean by "the command line's environment". If you mean that in the narrow sense of a mapping from variable names to values, then

    • The java command by which your VM is launched gets its environment from the command that launches it. That more likely than not works out to "the command line's environment".

    • Unless you take action to produce a different effect (and the code presented does not) processes you launch with the help of a ProcessBuilder will inherit a copy of the VM's environment.

    But I think you're barking up the wrong tree. Although it is conceivable that an environment issue might contribute indirectly to a java.lang.NoSuchFieldError, such an Error is

    Thrown if an application tries to access or modify a specified field of an object, and that object no longer has that field.

    Normally, this error is caught by the compiler; this error can only occur at run time if the definition of a class has incompatibly changed.

    (API docs)

    This suggests that the proximal cause of the error is that you have mismatched classes in your runtime classpath. Field in this context means a member variable of a class, so that's what Java is looking for SASL_PROPS to be (though it's unclear to me which class is expected to have that field).