C AES-GCM Decryption of ESP packet not working

I'm trying to decrypt my ESP packet encrypted with AES128-GCM12. I used c openssl library for decryption. But decrypted result is wrong!

I used two Linux 18.04 VM for simulating ESP packet with strongswan IPsec. I captured ESP packet and printed my whole variables in my decrypt function.

static void _aes_gcm_decrypt(uint8_t* payload, uint16_t payload_len, uint8_t* key, uint16_t key_len, uint8_t* iv, uint8_t iv_len, uint8_t* icv, uint8_t icv_len, uint8_t* aad, uint8_t aad_len) {
    EVP_CIPHER_CTX *ctx;
    int outlen;
    uint8_t outbuf[1024];
    int rv;
    const EVP_CIPHER *cipher;

    // Ignore salt
    key_len -= 4;
    switch(key_len) {
        case 16: cipher = EVP_aes_128_gcm(); break;
        case 24: cipher = EVP_aes_192_gcm(); break;
        case 32: cipher = EVP_aes_256_gcm(); break;
        default: break;
    }

    print_hex("original payload", payload, payload_len);
    print_hex("key", key, key_len);
    print_hex("iv", iv, iv_len);
    print_hex("icv", icv, icv_len);
    print_hex("aad", aad, aad_len);

    ctx = EVP_CIPHER_CTX_new();
    // Select cipher
    EVP_DecryptInit_ex(ctx, cipher, NULL, key, iv);
    // Set IV length, omit for 96 bits
    EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_SET_IVLEN, iv_len, NULL);
    // Specify key and IV
    EVP_DecryptInit_ex(ctx, NULL, NULL, key, iv);
    // Zero or more calls to specify any AAD
    EVP_DecryptUpdate(ctx, NULL, &outlen, aad, aad_len);
    // Decrypt plaintext
    EVP_DecryptUpdate(ctx, outbuf, &outlen, payload, payload_len);
    print_hex("decrypted_payload", outbuf, outlen);

    EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_SET_TAG, icv_len, icv);
    rv = EVP_DecryptFinal(ctx, icv, &outlen);

    printf("Tag Verify %s\n", rv > 0 ? "Successful!" : "Failed!");
    EVP_CIPHER_CTX_free(ctx);
}

And this is my captured packet.

04:51:57.347960 IP (tos 0x0, ttl 64, id 48604, offset 0, flags [DF], proto ESP (50), length 136)
    pc2 > 10.10.10.100: ESP(spi=0xc08247f5,seq=0x2), length 116
        0x0000:  0011 2233 4401 0800 2758 2898 0800 4500
        0x0010:  0088 bddc 4000 4032 538b 0a0a 0a65 0a0a
        0x0020:  0a64 c082 47f5 0000 0002 7a2b 37d7 160c
        0x0030:  853c 870d 1119 5a34 9d95 e597 be6a 8bc7
        0x0040:  2037 a1f7 ba02 1ef2 a0be de5e 5406 a5b1
        0x0050:  0e03 c463 c235 5c45 9b51 6734 1f28 e364
        0x0060:  2b36 470b 64da bfa3 1a68 f209 94aa 44b0
        0x0070:  9131 ffe0 12f1 9208 3a7b aa95 da51 bafd
        0x0080:  31cd 3d0a 8733 56e0 ae0d d7b5 13fe 8c5e
        0x0090:  96d8 598d a74f

And this is my printed variables

original payload(88): 
    0x0000:  870d 1119 5a34 9d95 e597 be6a 8bc7 2037 
    0x0010:  a1f7 ba02 1ef2 a0be de5e 5406 a5b1 0e03 
    0x0020:  c463 c235 5c45 9b51 6734 1f28 e364 2b36 
    0x0030:  470b 64da bfa3 1a68 f209 94aa 44b0 9131 
    0x0040:  ffe0 12f1 9208 3a7b aa95 da51 bafd 31cd 
    0x0050:  3d0a 8733 56e0 ae0d 
key(16): 
    0x0000:  1a0f cccc 0315 2b58 1b1a 02ea 3664 485f 
iv(8): 
    0x0000:  7a2b 37d7 160c 853c 
icv(12): 
    0x0000:  d7b5 13fe 8c5e 96d8 598d a74f 
aad(8): 
    0x0000:  c082 47f5 0000 0002 
decrypted_payload(88): 
    0x0000:  860c 9110 87c6 db5a c40a a592 f862 b8ea 
    0x0010:  b476 3be6 0881 06d4 ad11 eb2a 2e1c 698d 
    0x0020:  a803 2417 8ffb 7130 444d 4fc8 b402 c602 
    0x0030:  1a0b 031b b89a 4ddb 707b f920 04ea 48c5 
    0x0040:  5424 5a9b bb34 a88a 08ee 2556 5532 3419 
    0x0050:  2621 19e6 f4e1 72b2 
Tag Verify Failed!

The decrypted payload should be some ICMP packet. But it is wrong! Is the decryption method wrong, or did the variables get in the packet incorrectly?

Solution

The message can be successfully decrypted if the current 8 byte IV (ESP-IV, see below):
```
              7a 2b 37 d7 16 0c 85 3c
```
is replaced by the following 12 byte IV (Nonce / AES-GCM-IV, see below):
```
  b3 0d e6 26 7a 2b 37 d7 16 0c 85 3c 
```

The decrypted message is:

  45 00 00 54 8e 07 40 00 40 01 82 da 0a 00 0b 64 
  0a 00 0a 64 08 00 04 f2 28 7d 00 01 ed 10 0b 5d 
  00 00 00 00 0e 4f 05 00 00 00 00 00 10 11 12 13
  14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23
  24 25 26 27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33
  34 35 36 37 01 02 02 04

Explanation: According to the AES-GCM-ESP description, Section 8.1, the key for AES-128-GCM-ESP has 20 bytes. The first 16 bytes are the AES-128-GCM key, the last 4 bytes are the salt. The nonce is composed of salt (4 bytes) and ESP-IV (8 bytes) and is therefore 12 bytes in size. The nonce is the AES-GCM-IV. For an explanation of the difference between ESP-IV and AES-GCM-IV, see here, section 2. In the current code, the ESP-IV is used as AES-GCM-IV, which is wrong. If the nonce is used as AES-GCM-IV, the decryption is successful.
The 4 bytes of the salt could be brute forced to:
```
  b3 0d e6 26
```
They should correspond to the last 4 bytes of the 20 byte AES-128-GCM-ESP key.
The _aes_gcm_decrypt-method contains a small flaw, which however has no consequences in GCM-mode. Nevertheless, the method should be modified as follows:
```
  ...
  EVP_DecryptUpdate(ctx, outbuf, &outlen, payload, payload_len);
  int plaintext_len = outlen;                                    // added

  EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_SET_TAG, icv_len, icv);

  rv = EVP_DecryptFinal_ex(ctx, outbuf + outlen, &outlen);       // modified
  plaintext_len += outlen;                                       // added
  ...
```
In plaintext_len the length of the decrypted message is stored. EVP_DecryptFinal decrypts a possible last incompletely filled block when padding is enabled. Since the GCM-mode doesn't use padding, the error has no effect. However, the length of the decrypted message stored in outlen is lost in the current code, see also here.