I encrypt file with openssl then put it on HDFS, I used AES/ECB, 128 bits and salt option, and with some research I find out openssl uses PKCS5 padding as default which are all defaults in CryptoFileLoader class. Here is my encryption process:
# echo -n "password" > .pw
# openssl enc -aes-128-ecb -salt -in .pw -out .pw.enc
# hdfs dfs -put .pw.enc /user/user1/
Sqoop version is 1.4.6
Command:
sqoop import \
-Dorg.apache.sqoop.credentials.loader.class=org.apache.sqoop.util.password.CryptoFileLoader \
-Dorg.apache.sqoop.credentials.loader.crypto.passphrase=sqoop \
--connect jdbc:oracle:thin:@host/database \
--username user1 \
--password-file /user/user1/.pw.enc \
--table db.table1 \
--hive-import \
--hive-overwrite \
--hive-table hivedb.table1 \
--hive-drop-import-delims
which gives:
17/03/08 15:10:37 WARN tool.BaseSqoopTool: Failed to load password file
java.io.IOException: Can't decrypt the password
at org.apache.sqoop.util.password.CryptoFileLoader.loadPassword(CryptoFileLoader.java:151)
at org.apache.sqoop.util.CredentialsUtil.fetchPasswordFromLoader(CredentialsUtil.java:81)
at org.apache.sqoop.util.CredentialsUtil.fetchPassword(CredentialsUtil.java:66)
at org.apache.sqoop.tool.BaseSqoopTool.applyCredentialsOptions(BaseSqoopTool.java:1042)
at org.apache.sqoop.tool.BaseSqoopTool.applyCommonOptions(BaseSqoopTool.java:997)
at org.apache.sqoop.tool.ImportTool.applyOptions(ImportTool.java:875)
at org.apache.sqoop.tool.SqoopTool.parseArguments(SqoopTool.java:435)
at org.apache.sqoop.Sqoop.run(Sqoop.java:131)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
Caused by: javax.crypto.BadPaddingException: Given final block not properly padded
at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:966)
at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:824)
at com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:436)
at javax.crypto.Cipher.doFinal(Cipher.java:2165)
at org.apache.sqoop.util.password.CryptoFileLoader.loadPassword(CryptoFileLoader.java:149)
... 12 more
Error while loading password file: Can't decrypt the password
I tried manually giving the other CryptoFileLoader parameters too and also passing local file to the --password-file .
I can decrypt the file back successfully with openssl. I can't decrypt with Java program(?)
I saw there is an issue with padding but I didn't know what it is and how to encrypt the file with a certain padding method or whatever else to do, I'm not experienced with encryption.
There is also org.apache.sqoop.credentials.loader.crypto.iterations parameter in the class which indicates number of PBKDF2 iterations but I don't know if it changes anything.
Thanks for any help.
I am not expert with Sqoop and Hadoop but starting from your exception
CryptoFileLoader.loadPassword(CryptoFileLoader.java:151)
I gave a look at the source code of CryptoFileLoader.java
It seems to me that things are a bit different from what you do: the password is stored in an encrypted file using the PBKDF2 algorithm, which is not equivalent to apply AES-128-ECB. From wikipedia:
PBKDF2 applies a pseudorandom function, such as hash-based message authentication code (HMAC), to the input password or passphrase along with a salt value and repeats the process many times to produce a derived key, which can then be used as a cryptographic key in subsequent operations. The added computational work makes password cracking much more difficult, and is known as key stretching.
There is no way to do PBKDF2 from Openssl command line. I made a small test using Java, it could be an alternative
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.security.NoSuchAlgorithmException;
import javax.crypto.Cipher;
import javax.crypto.SecretKeyFactory;
import javax.crypto.spec.PBEKeySpec;
import javax.crypto.spec.SecretKeySpec;
public class Test {
/*Default is AES in electronic code book with padding.*/
private static String DEFAULT_ALG = "AES/ECB/PKCS5Padding";
/*Default salt is not much secure, use your own!*/
private static String DEFAULT_SALT = "SALT";
/*Iterate 10000 times by default.*/
private static int DEFAULT_ITERATIONS = 10000;
/*One of valid key sizes for default algorithm (AES).*/
private static int DEFAULT_KEY_LEN = 128;
public static void main(String[] args) throws IOException {
String inputFileName = "C:\\temp\\in.txt"; /*Enter your input (plain) file path */
String outputFileName = "C:\\temp\\out.bin"; /*Enter your output (encrypted) file path */
String passPhrase = "mypassphrase"; /*Enter your passphrase */
String salt = DEFAULT_SALT;
String alg = DEFAULT_ALG;
int iterations = DEFAULT_ITERATIONS;
int keyLen = DEFAULT_KEY_LEN;
SecretKeyFactory factory = null;
try {
factory = SecretKeyFactory.getInstance("PBKDF2WithHmacSHA1");
} catch (NoSuchAlgorithmException e) {
throw new IOException("Can't load SecretKeyFactory", e);
}
SecretKeySpec key = null;
try {
String algOnly = alg.split("/")[0];
key = new SecretKeySpec(
factory.generateSecret(
new PBEKeySpec(passPhrase.toCharArray(), salt.getBytes(), iterations, keyLen)).getEncoded(),
algOnly);
} catch (Exception e) {
throw new IOException("Can't generate secret key", e);
}
Cipher crypto = null;
try {
crypto = Cipher.getInstance(alg);
} catch (Exception e) {
throw new IOException("Can't initialize the decryptor", e);
}
Path inputFileLocation = Paths.get(inputFileName);
byte[] decrypted = Files.readAllBytes(inputFileLocation);
byte[] encrypted;
try {
crypto.init(Cipher.ENCRYPT_MODE, key);
encrypted = crypto.doFinal(decrypted);
} catch (Exception e) {
throw new IOException("Can't decrypt the password", e);
}
Path outputFileLocation = Paths.get(outputFileName);
Files.write(outputFileLocation, encrypted);
}
}