We are trying to configure IWA for SAS Data Loader for Hadoop (DLH). SAS Servers are running under Active Directory domain and SSO is successfully configured. We need to configure DLH to talk to Hortonworks Hadoop MIT Kerberos using client generated tickets. That functionality is not working.
So basically we have problem with AD (ABC.COM) and Hadoop MIT Kerberos (xyz - Hadoop realm name doesnt have any FQDN and it is all small letters) 2 way trust establishment. We have confiure the trust as per the following link (https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_security/content/kerb-config-realm-kdc.html) and everything worked fine but somehow using AD's HTTP ticket, we are not able to login to hadoop and we get the following error message:
com.sas.svcs.dm.hadoop.spi.exception.HadoopConfigurationException: Failed to find GSSCredential. Check Kerberos configuration
we have tried hell lot of troubleshooting. Now finally it has been finalized that we have issue with trust and to test following are the steps are asked to perform.
on SAS Server (linux)
kinit -f HTTP/xxx.abc.com@ABC.COM
klist -eaf
kvno hive/xyz@xyz
if all these above steps works, that means we have trust enabled
This is the error which we see
kvno: KDC returned error string: PROCESS_TGS while getting credentials for hive/xyz@xyz
and
kvno: Decrypt integrity check failed while getting credentials for hive/xyz@xyz
kinit -f HTTP/xxx.abc.com@ABC.COM (this works fine)
# kinit -k -t xxx.host.keytab HTTP/xxx.abc.com@ABC.COM
[65181] 1559895039.846538: Getting initial credentials for HTTP/xxx.abc.com@ABC.COM
[65181] 1559895039.846539: Looked up etypes in keytab: des-cbc-crc, des, des-cbc-crc, rc4-hmac, aes256-cts, aes128-cts
[65181] 1559895039.846541: Sending unauthenticated request
[65181] 1559895039.846542: Sending request (220 bytes) to ABC.COM
[65181] 1559895039.846543: Sending initial UDP request to dgram 10.68.5.219:88
[65181] 1559895039.846544: Received answer (819 bytes) from dgram 10.68.5.219:88
[65181] 1559895039.846545: Response was from master KDC
[65181] 1559895039.846546: Processing preauth types: PA-ETYPE-INFO2 (19)
[65181] 1559895039.846547: Selected etype info: etype aes256-cts, salt "ABC.COMHTTPxxx.abc.com", params ""
[65181] 1559895039.846548: Produced preauth for next request: (empty)
[65181] 1559895039.846549: Getting AS key, salt "ABC.COMHTTPxxx.abc.com", params ""
[65181] 1559895039.846550: Retrieving HTTP/xxx.abc.com@ABC.COM from FILE:xxx.host.keytab (vno 0, enctype aes256-cts) with result: 0/Success
[65181] 1559895039.846551: AS key obtained from gak_fct: aes256-cts/8AEB
[65181] 1559895039.846552: Decrypted AS reply; session key is: aes256-cts/E734
[65181] 1559895039.846553: FAST negotiation: unavailable
[65181] 1559895039.846554: Initializing FILE:/tmp/krb5cc_0 with default princ HTTP/xxx.abc.com@ABC.COM
[65181] 1559895039.846555: Storing HTTP/xxx.abc.com@ABC.COM -> krbtgt/ABC.COM@ABC.COM in FILE:/tmp/krb5cc_0
##########################################################
klist -e (this shows the ticket is generated)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# klist -e
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: HTTP/xxx.abc.com@ABC.COM
Valid starting Expires Service principal
06/07/2019 13:40:39 06/07/2019 13:50:39 krbtgt/ABC.COM@ABC.COM
Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
###########################################################
kvno hive/xyz@xyz (this command fails)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# kvno hive/xyz@xyz
[65247] 1559895064.242178: Getting credentials HTTP/xxx.abc.com@ABC.COM -> hive/xyz@xyz using ccache FILE:/tmp/krb5cc_0
[65247] 1559895064.242179: Retrieving HTTP/xxx.abc.com@ABC.COM -> hive/xyz@xyz from FILE:/tmp/krb5cc_0 with result: -1765328243/Matching credential not found (filename: /tmp/krb5cc_0)
[65247] 1559895064.242180: Retrieving HTTP/xxx.abc.com@ABC.COM -> krbtgt/xyz@xyz from FILE:/tmp/krb5cc_0 with result: -1765328243/Matching credential not found (filename: /tmp/krb5cc_0)
[65247] 1559895064.242181: Retrieving HTTP/xxx.abc.com@ABC.COM -> krbtgt/ABC.COM@ABC.COM from FILE:/tmp/krb5cc_0 with result: 0/Success
[65247] 1559895064.242182: Starting with TGT for client realm: HTTP/xxx.abc.com@ABC.COM -> krbtgt/ABC.COM@ABC.COM
[65247] 1559895064.242183: Retrieving HTTP/xxx.abc.com@ABC.COM -> krbtgt/xyz@xyz from FILE:/tmp/krb5cc_0 with result: -1765328243/Matching credential not found (filename: /tmp/krb5cc_0)
[65247] 1559895064.242184: Requesting TGT krbtgt/xyz@ABC.COM using TGT krbtgt/ABC.COM@ABC.COM
[65247] 1559895064.242185: Generated subkey for TGS request: aes256-cts/C142
[65247] 1559895064.242186: etypes requested in TGS request: aes256-cts, aes128-cts, aes256-sha2, aes128-sha2, des3-cbc-sha1, rc4-hmac, camellia128-cts, camellia256-cts, des-cbc-crc, des, des-cbc-md4
[65247] 1559895064.242188: Encoding request body and padata into FAST request
[65247] 1559895064.242189: Sending request (1001 bytes) to ABC.COM
[65247] 1559895064.242190: Sending initial UDP request to dgram 10.68.5.219:88
[65247] 1559895064.242191: Received answer (873 bytes) from dgram 10.68.5.219:88
[65247] 1559895064.242192: Response was from master KDC
[65247] 1559895064.242193: Decoding FAST response
[65247] 1559895064.242194: FAST reply key: aes256-cts/9192
[65247] 1559895064.242195: TGS reply is for HTTP/xxx.abc.com@ABC.COM -> krbtgt/xyz@ABC.COM with session key des-cbc-crc/330F
[65247] 1559895064.242196: TGS request result: 0/Success
[65247] 1559895064.242197: Storing HTTP/xxx.abc.com@ABC.COM -> krbtgt/xyz@ABC.COM in FILE:/tmp/krb5cc_0
[65247] 1559895064.242198: Received TGT for service realm: krbtgt/xyz@ABC.COM
[65247] 1559895064.242199: Requesting tickets for hive/xyz@xyz, referrals on
[65247] 1559895064.242200: Generated subkey for TGS request: des-cbc-crc/FB8F
[65247] 1559895064.242201: etypes requested in TGS request: aes256-cts, aes128-cts, aes256-sha2, aes128-sha2, des3-cbc-sha1, rc4-hmac, camellia128-cts, camellia256-cts, des-cbc-crc, des, des-cbc-md4
[65247] 1559895064.242203: Encoding request body and padata into FAST request
[65247] 1559895064.242204: Sending request (935 bytes) to xyz
[65247] 1559895064.242205: Resolving hostname xyz
[65247] 1559895064.242206: Sending initial UDP request to dgram 10.68.166.7:88
[65247] 1559895064.242207: Received answer (138 bytes) from dgram 10.68.166.7:88
[65247] 1559895064.242208: Response was not from master KDC
[65247] 1559895064.242209: TGS request result: -1765328324/KDC returned error string: PROCESS_TGS
[65247] 1559895064.242210: Requesting tickets for hive/xyz@xyz, referrals off
[65247] 1559895064.242211: Generated subkey for TGS request: des-cbc-crc/01C2
[65247] 1559895064.242212: etypes requested in TGS request: aes256-cts, aes128-cts, aes256-sha2, aes128-sha2, des3-cbc-sha1, rc4-hmac, camellia128-cts, camellia256-cts, des-cbc-crc, des, des-cbc-md4
[65247] 1559895064.242214: Encoding request body and padata into FAST request
[65247] 1559895064.242215: Sending request (935 bytes) to xyz
[65247] 1559895064.242216: Resolving hostname xyz
[65247] 1559895064.242217: Sending initial UDP request to dgram 10.68.166.7:88
[65247] 1559895064.242218: Received answer (138 bytes) from dgram 10.68.166.7:88
[65247] 1559895064.242219: Response was not from master KDC
[65247] 1559895064.242220: TGS request result: -1765328324/KDC returned error string: PROCESS_TGS
kvno: KDC returned error string: PROCESS_TGS while getting credentials for hive/xyz@xyz
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Also just for troubleshooting I add enctypes on my AD server using the following command:
ksetup /SetEncTypeAttr xyz DES-CBC-CRC DES-CBC-MD5 RC4-HMAC-MD5 AES128-CTS-HMAC-SHA1-96 AES256-CTS-HMAC-SHA1-96
So, after running the above command when I try to run the kvno command, my error message changes
:from
kvno: KDC returned error string: PROCESS_TGS while getting credentials for hive/xyz@xyz
:to
kvno: Decrypt integrity check failed while getting credentials for hive/xyz@xyz
full kvno cmmand trace is as below:
# kvno hive/xyz@xyz
[128763] 1559917554.849763: Getting credentials HTTP/xxx.abc.com@ABC.COM -> hive/xyz@xyz using ccache FILE:/tmp/krb5cc_0
[128763] 1559917554.849764: Retrieving HTTP/xxx.abc.com@ABC.COM -> hive/xyz@xyz from FILE:/tmp/krb5cc_0 with result: -1765328243/Matching credential not found (filename: /tmp/krb5cc_0)
[128763] 1559917554.849765: Retrieving HTTP/xxx.abc.com@ABC.COM -> krbtgt/xyz@xyz from FILE:/tmp/krb5cc_0 with result: -1765328243/Matching credential not found (filename: /tmp/krb5cc_0)
[128763] 1559917554.849766: Retrieving HTTP/xxx.abc.com@ABC.COM -> krbtgt/ABC.COM@ABC.COM from FILE:/tmp/krb5cc_0 with result: 0/Success
[128763] 1559917554.849767: Starting with TGT for client realm: HTTP/xxx.abc.com@ABC.COM -> krbtgt/ABC.COM@ABC.COM
[128763] 1559917554.849768: Retrieving HTTP/xxx.abc.com@ABC.COM -> krbtgt/xyz@xyz from FILE:/tmp/krb5cc_0 with result: -1765328243/Matching credential not found (filename: /tmp/krb5cc_0)
[128763] 1559917554.849769: Requesting TGT krbtgt/xyz@ABC.COM using TGT krbtgt/ABC.COM@ABC.COM
[128763] 1559917554.849770: Generated subkey for TGS request: aes256-cts/4F0F
[128763] 1559917554.849771: etypes requested in TGS request: aes256-cts, aes128-cts, aes256-sha2, aes128-sha2, des3-cbc-sha1, rc4-hmac, camellia128-cts, camellia256-cts, des-cbc-crc, des, des-cbc-md4
[128763] 1559917554.849773: Encoding request body and padata into FAST request
[128763] 1559917554.849774: Sending request (1022 bytes) to ABC.COM
[128763] 1559917554.849775: Sending initial UDP request to dgram 10.68.5.219:88
[128763] 1559917554.849776: Received answer (969 bytes) from dgram 10.68.5.219:88
[128763] 1559917554.849777: Response was from master KDC
[128763] 1559917554.849778: Decoding FAST response
[128763] 1559917554.849779: FAST reply key: aes256-cts/944C
[128763] 1559917554.849780: TGS reply is for HTTP/xxx.abc.com@ABC.COM -> krbtgt/xyz@ABC.COM with session key aes256-cts/B3D3
[128763] 1559917554.849781: TGS request result: 0/Success
[128763] 1559917554.849782: Storing HTTP/xxx.abc.com@ABC.COM -> krbtgt/xyz@ABC.COM in FILE:/tmp/krb5cc_0
[128763] 1559917554.849783: Received TGT for service realm: krbtgt/xyz@ABC.COM
[128763] 1559917554.849784: Requesting tickets for hive/xyz@xyz, referrals on
[128763] 1559917554.849785: Generated subkey for TGS request: aes256-cts/DF91
[128763] 1559917554.849786: etypes requested in TGS request: aes256-cts, aes128-cts, aes256-sha2, aes128-sha2, des3-cbc-sha1, rc4-hmac, camellia128-cts, camellia256-cts, des-cbc-crc, des, des-cbc-md4
[128763] 1559917554.849788: Encoding request body and padata into FAST request
[128763] 1559917554.849789: Sending request (1013 bytes) to xyz
[128763] 1559917554.849790: Resolving hostname xyz
[128763] 1559917554.849791: Sending initial UDP request to dgram 10.68.166.7:88
[128763] 1559917554.849792: Received answer (138 bytes) from dgram 10.68.166.7:88
[128763] 1559917554.849793: Response was not from master KDC
[128763] 1559917554.849794: TGS request result: -1765328353/Decrypt integrity check failed
[128763] 1559917554.849795: Requesting tickets for hive/xyz@xyz, referrals off
[128763] 1559917554.849796: Generated subkey for TGS request: aes256-cts/34D1
[128763] 1559917554.849797: etypes requested in TGS request: aes256-cts, aes128-cts, aes256-sha2, aes128-sha2, des3-cbc-sha1, rc4-hmac, camellia128-cts, camellia256-cts, des-cbc-crc, des, des-cbc-md4
[128763] 1559917554.849799: Encoding request body and padata into FAST request
[128763] 1559917554.849800: Sending request (1013 bytes) to xyz
[128763] 1559917554.849801: Resolving hostname xyz
[128763] 1559917554.849802: Sending initial UDP request to dgram 10.68.166.7:88
[128763] 1559917554.849803: Received answer (138 bytes) from dgram 10.68.166.7:88
[128763] 1559917554.849805: TGS request result: -1765328353/Decrypt integrity check failed
kvno: Decrypt integrity check failed while getting credentials for hive/xyz@xyz
The issue was with AD and Hadoop Trust wasnt working fine. So during troubleshooting I added the enctypes on my Hadoop Principal on AD. I found the following note on one of the site
“The principal (account) is created using the system-default enctype. When you change the enctype, you must also recreate the principal, or at least update the principal’s password.”
So, I did the reset of the password
netdom trust xyz /Domain:ABC.COM /reset /realm /passwordt:xxxxXXXxxxx
Also, KVNO no wasnt matching between AD and Hadoop, so I updated the kvno at Hadoop side
restarted the following services on Hadoop Server
/sbin/service krb5kdc restart /sbin/service kadmin restart
and Voila...I was able to run the kvno command.
[74264] 1561019777.500742: Storing HTTP/xxx.abc.com@ABC.COM -> krbtgt/ABC.COM@ABC.COM in FILE:/tmp/krb5cc_1001
Ticket cache: FILE:/tmp/krb5cc_1001 Default principal: HTTP/xxx.abc.com@ABC.COM
Valid starting Expires Service principal 06/20/2019 14:06:17 06/21/2019 00:06:17 krbtgt/ABC.COM@ABC.COM renew until 06/27/2019 14:06:17, Flags: FRI Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96 Addresses: (none)
[74362] 1561019789.592571: Received creds for desired service hive/xyz@xyz [74362] 1561019789.592572: Storing HTTP/xxx.abc.com@ABC.COM -> hive/xyz@xyz in FILE:/tmp/krb5cc_1001 hive/xyz@xyz: kvno = 1