Not sure what is happening since there are multiple moving parts here. We have a cloudera cluster for hdfs, hadoop, impala, hbase. We also have an F5 loadbalancer in front of all our impala servers. We are trying to secure the servers/cluster with Kerberos. My colleague has setup Kerberos using MIT KDC. This setup works fine when we query impala directly to the server but not when we go thru an F5 load balancer.
We've run kinit to get a ticket for a pre-created keytab file.
kinit -k -t /blah/keytabs/first.last.keytab first.last
When I run klist, it shows all these tickets:
$ klist
Ticket cache: FILE:/tmp/krb5cc_14377
Default principal: first.last@MADEUPNAME
Valid starting Expires Service principal
08/23/17 11:32:02 08/24/17 11:32:02 krbtgt/MADEUPNAME@MADEUPNAME
renew until 08/23/17 11:32:02
08/23/17 11:33:39 08/24/17 11:32:02 impala/hslave32101.company.com@MADEUPNAME
renew until 08/23/17 11:32:02
When I run my impala-shell command, it works fine:
$ impala-shell -i hslave32101.company.com:21000 -k -q "select 1"
Starting Impala Shell using Kerberos authentication
Using service name 'impala'
Connected to hslave32101.company.com:21000
Server version: impalad version 2.7.0-cdh5.9.2 RELEASE (build 2f7871169d894fab16f8a2fb99f2e34f0df8763d)
Query: select 1
Query submitted at: 2017-08-23 13:08:34 (Coordinator: http://hslave32101.company.com:25000)
Query progress can be monitored at: http://hslave32101.company.com:25000/query_plan?query_id=4940ca8ca2f267c5:5eeb29af00000000
+---+
| 1 |
+---+
| 1 |
+---+
Fetched 1 row(s) in 0.01s
However, when I run my command thru the F5 loadbalancer, it doesn't work because the ticket it's looking for doesn't match what's in klist because it replaced part of it for some reason.
impala-shell -i bdaudit.company.com:21000 -d bigdata -k -q "select 1"
Starting Impala Shell using Kerberos authentication
Using service name 'impala'
Error connecting: TTransportException, Could not start SASL: Error in sasl_client_start (-1) SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server krbtgt/COMPANY.COM@MADEUPNAME not found in Kerberos database)
Not connected to Impala, could not execute queries.
The problem is this line here
(Server krbtgt/COMPANY.COM@MADEUPNAME not found in Kerberos database)
Somehow when going thru the F5 VIP, it changes first.last@MADEUPNAME to COMPANY.COM@MADEUPNAME. Does anyone know why it replaced this part of the ticket?
Found the reason from Cloudera's instructions on how to setup Impala with an F5 here and here
Here's the snippet from the PDF:
In Cloudera Manager, navigate to the Impala service, select the Configuration pane, then search for “balancer” to find the Impala Daemons Load Balancer parameter. The load balancer should be specified in host:port format, where host is your virtual server’s FQDN and port. These values are used by Cloudera Manager and are also passed to Hue If the Impala Daemons Load Balancer parameter is specified and Kerberos is enabled, Cloudera Manager adds a principal for 'impala/<load_balancer_host>@<realm>' to the keytab for all Impala daemons. No additional configuration is required for Kerberos.