Search code examples
openshiftapache-drill

Running Apache Drill on OpenShift fails at startup

does anyway have experience on running apache drill on openshift?

I'm having a problem where I'm trying to run the standard docker-image in openshift https://hub.docker.com/r/apache/drill really just the embedded mode without trying to configure anything.

it works fine if I run it in my local docker desktop daemon and it's totally fine. but when I deploy it to openshift it becomes totally weird by not even starting up.

my deployment yaml is very straight forward:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: apache-drill
  labels:
    app.kubernetes.io/component: apache-drill-svc
spec:
  replicas: 1
  selector:
    matchLabels:
  template:
    metadata:
      labels:
        app.kubernetes.io/component: apache-drill-svc
    spec:
      containers:
        - name: apache-drill
          image: apache/drill
          imagePullPolicy: IfNotPresent
          resources:
            requests:
              cpu: 1
              memory: 1Gi
            limits:
              cpu: 4
              memory: 4Gi

But once it launches this is the feedback that it fails to start because of the log directories: pod events pod log

I believe it might be because of this guidelines regarding the security. https://docs.openshift.com/container-platform/3.11/creating_images/guidelines.html#openshift-specific-guidelines

so because of that I've tried to create my own image wrapping the official one: https://hub.docker.com/r/cstmgl/adrill/tags (probably there are better ways to do this) anyway I still get an error even thought the image also works final in my local docker daemon but if I try it on OpenShift I get this error:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: apache-drill
  labels:
    app.kubernetes.io/component: apache-drill-svc
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app.kubernetes.io/component: apache-drill-svc
    spec:
      containers:
        - name: apache-drill
          image: cstmgl/adrill:1.18.0
          imagePullPolicy: IfNotPresent
          resources:
            requests:
              cpu: 1
              memory: 1Gi
            limits:
              cpu: 4
              memory: 4Gi

Any idea what I'm doing wrong? I'll update this soon with the actual startup error it's rather basic but I believe it's a problem with the user running the application just not sure how to get around it.

second event logs

ERROR: Drill config file not readable: /opt/drill/conf/drill-override.conf - Wrong user?

Anyway I changed that image also to something like this in my dockerfile

FROM cstmgl/adrill:1.18.0

ARG APP_ROOT=/opt/drill

RUN chmod -R ugo+rw ${APP_ROOT}

RUN echo ${UID}

USER ${UID}

WORKDIR ${DRILL_HOME}

ENTRYPOINT [ "drill-embedded" ]

but still not working, this is the log:

Error: Failure in starting embedded Drillbit: org.apache.drill.exec.exception.DrillbitStartupException: Failed to login. (state=,code=0)
java.sql.SQLException: Failure in starting embedded Drillbit: org.apache.drill.exec.exception.DrillbitStartupException: Failed to login.
    at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:143)
    at org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:67)
    at org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:67)
    at org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
    at org.apache.drill.jdbc.Driver.connect(Driver.java:75)
    at sqlline.DatabaseConnection.connect(DatabaseConnection.java:135)
    at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:192)
    at sqlline.Commands.connect(Commands.java:1364)
    at sqlline.Commands.connect(Commands.java:1244)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
    at sqlline.SqlLine.dispatch(SqlLine.java:730)
    at sqlline.SqlLine.initArgs(SqlLine.java:410)
    at sqlline.SqlLine.begin(SqlLine.java:515)
    at sqlline.SqlLine.start(SqlLine.java:267)
    at sqlline.SqlLine.main(SqlLine.java:206)
Caused by: org.apache.drill.exec.exception.DrillbitStartupException: Failed to login.
    at org.apache.drill.exec.server.BootStrapContext.login(BootStrapContext.java:161)
    at org.apache.drill.exec.server.BootStrapContext.<init>(BootStrapContext.java:82)
    at org.apache.drill.exec.server.Drillbit.<init>(Drillbit.java:171)
    at org.apache.drill.exec.server.Drillbit.<init>(Drillbit.java:135)
    at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:133)
    ... 18 more
Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: javax.security.auth.login.LoginException: java.lang.NullPointerException: invalid null input: name
    at com.sun.security.auth.UnixPrincipal.<init>(UnixPrincipal.java:71)
    at com.sun.security.auth.module.UnixLoginModule.login(UnixLoginModule.java:133)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
    at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
    at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
    at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
    at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
    at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:1926)
    at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1837)
    at org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:710)
    at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:660)
    at org.apache.drill.exec.server.BootStrapContext.login(BootStrapContext.java:156)
    at org.apache.drill.exec.server.BootStrapContext.<init>(BootStrapContext.java:82)
    at org.apache.drill.exec.server.Drillbit.<init>(Drillbit.java:171)
    at org.apache.drill.exec.server.Drillbit.<init>(Drillbit.java:135)
    at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:133)
    at org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:67)
    at org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:67)
    at org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
    at org.apache.drill.jdbc.Driver.connect(Driver.java:75)
    at sqlline.DatabaseConnection.connect(DatabaseConnection.java:135)
    at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:192)
    at sqlline.Commands.connect(Commands.java:1364)
    at sqlline.Commands.connect(Commands.java:1244)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
    at sqlline.SqlLine.dispatch(SqlLine.java:730)
    at sqlline.SqlLine.initArgs(SqlLine.java:410)
    at sqlline.SqlLine.begin(SqlLine.java:515)
    at sqlline.SqlLine.start(SqlLine.java:267)
    at sqlline.SqlLine.main(SqlLine.java:206)

    at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1847)
    at org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:710)
    at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:660)
    at org.apache.drill.exec.server.BootStrapContext.login(BootStrapContext.java:156)
    ... 22 more
Caused by: javax.security.auth.login.LoginException: java.lang.NullPointerException: invalid null input: name
    at com.sun.security.auth.UnixPrincipal.<init>(UnixPrincipal.java:71)
    at com.sun.security.auth.module.UnixLoginModule.login(UnixLoginModule.java:133)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
    at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
    at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
    at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
    at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
    at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:1926)
    at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1837)
    at org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:710)
    at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:660)
    at org.apache.drill.exec.server.BootStrapContext.login(BootStrapContext.java:156)
    at org.apache.drill.exec.server.BootStrapContext.<init>(BootStrapContext.java:82)
    at org.apache.drill.exec.server.Drillbit.<init>(Drillbit.java:171)
    at org.apache.drill.exec.server.Drillbit.<init>(Drillbit.java:135)
    at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:133)
    at org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:67)
    at org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:67)
    at org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
    at org.apache.drill.jdbc.Driver.connect(Driver.java:75)
    at sqlline.DatabaseConnection.connect(DatabaseConnection.java:135)
    at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:192)
    at sqlline.Commands.connect(Commands.java:1364)
    at sqlline.Commands.connect(Commands.java:1244)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
    at sqlline.SqlLine.dispatch(SqlLine.java:730)
    at sqlline.SqlLine.initArgs(SqlLine.java:410)
    at sqlline.SqlLine.begin(SqlLine.java:515)
    at sqlline.SqlLine.start(SqlLine.java:267)
    at sqlline.SqlLine.main(SqlLine.java:206)

    at javax.security.auth.login.LoginContext.invoke(LoginContext.java:856)
    at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
    at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
    at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
    at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
    at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:1926)
    at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1837)
... 25 more
Apache Drill 1.18.0
"Two things are infinite: the universe and Drill; and I'm not sure about the universe."
Solution

  • I have come across this problem myself and after some hard googling, reading this post a few times, and some OpenShift documentation I stumbled upon the way to resolve this.

    The problem you are seeing in the logs indicates there is a problem with the name, if you look into sqlline.log you will see that user.name and user.home are empty - it looks something like this:

    INFO o.apache.drill.exec.server.Drillbit - Drillbit environment: user.name=?
INFO o.apache.drill.exec.server.Drillbit - Drillbit environment: user.home=?

    It seems not everyone knows this but basically OpenShift generates user IDs upon pod creation and the user does not have a home or a name, but in an older version of documentation from OpenShift they provide a way to generate 'default' for the username: https://docs.openshift.com/container-platform/3.10/creating_images/guidelines.html#openshift-specific-guidelines

    Specifically you want this uid_entrypoint script in your image that you can run to generate default for your 100xxxxx user: https://github.com/RHsyseng/container-rhel-examples/blob/master/starter-arbitrary-uid/bin/uid_entrypoint

    You will need to make sure that when building your image that you change the permission of /etc/passswd in order to be able to modify it when you start your image

    After I did this I was able to start Drill and you should see the user.name and user.home updated in the log file sqlline.log:

    INFO  o.apache.drill.exec.server.Drillbit - Drillbit environment: user.name=default
INFO  o.apache.drill.exec.server.Drillbit - Drillbit environment: user.home=/