Decrypting an AES256 hex ciphertext with a hex key

I have generated an AES-256 ciphertext in hex (cipher.hex) that I am trying to decrypt:

53 9B 33 3B 39 70 6D 14 90 28 CF E1 D9 D4 A4 07

with a corresponding 256-bit key in hex (key.hex):

80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01

I started by attempting to decode both into base64 using the following commands:

base64 cipher.hex > input;
base64 key.hex > key;

and lastly, passed them into openssl as seen below:

openssl aes-256-cbc -d -a -pass file:key -in input -out out

at which point I get back "bad magic number" from openssl with no guidance on what went wrong.

Is there something wrong with the procedure I'm following? I have also tried converting the hex values into binary... really not sure what input to give openssl from hex, let alone if base64 decoding a newline-wrapped hexidecimal is valid.

Solution

You're trying to use CBC and password-based encryption on something that is encrypted with ECB and a raw key. You're also base64 encoding the hex characters, not the actual bytes. You're encoding <ASCII for 5><ASCII for 3><ASCII for space>... You want to encode 0x53, 0x9B, 0x33, etc. Base64 isn't needed here in any case.

First, convert your hex input to raw data (not base64). I like xxd for this:

echo "53 9B 33 3B 39 70 6D 14 90 28 CF E1 D9 D4 A4 07" | xxd -r -p > input

Next, you need your key in the hex format openssl likes, no spaces or newlines:

KEY=$(echo "80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01" | tr -d '[ \n]'

echo $KEY  
# prints 8000000000000000000000000000000000000000000000000000000000000001

And now, you can decrypt the data (aes-256-ecb with a hex key):

openssl aes-256-ecb -d -in input -K $KEY | xxd

This prints:

00000000: 8070 6050 4030 2010 0807 0605 0403 02    .p`P@0 ........

Which is too orderly to be wrong. (0x80, 0x70, 0x60, 0x50, ...)

If this were CBC, there would be an IV, and in most cases a padding block. That said, for a single block with no padding and a NULL IV (which you should never use), CBC is equivalent to ECB, so it is possible to decrypt this as CBC and get the same result:

openssl aes-256-cbc -iv 00000000000000000000000000000000 -d -in input -K $KEY | xxd

The "bad magic number" error is because you used the -pass option, which employs OpenSSL's password key derivation algorithm. Files that have been encrypted that way start with the ASCII sequence Salted__. That's missing, so the input is rejected. (It wouldn't work anyway, since this was encrypted with a raw key.)