How to use nettle library, GCM mode

Question

I am using nettle cryptography library. I could not do GCM mode properly. Here is how I am doing it. What am I doing wrong?

#include<iostream>
#include<nettle/gcm.h>
#include<nettle/aes.h>
using namespace std;

int main()
{
    unsigned char key[] = "1234567890123456";
    unsigned char iv[] = "123456789012";
    unsigned char src[33] = "12345678901234567890123456789012";
    unsigned char encoded[32], digest[16], datum[8] = {0,}, decoded[32];

    struct gcm_key gk, gk2;
    struct gcm_ctx gc, gc2;
    struct aes128_ctx ac, ac2;

    aes128_set_encrypt_key(&ac, key);
    gcm_set_key(&gk, &ac, (nettle_cipher_func*)aes128_encrypt);
    gcm_set_iv(&gc, &gk, 12, iv);
    gcm_update(&gc, &gk, 8, datum);
    gcm_encrypt(&gc, &gk, &ac, (nettle_cipher_func*)aes128_encrypt, 32, encoded, src);
    gcm_digest(&gc, &gk, &ac, (nettle_cipher_func*)aes128_encrypt, 16, digest);

    aes128_set_decrypt_key(&ac2, key);
    gcm_set_key(&gk2, &ac2, (nettle_cipher_func*)aes128_decrypt);
    gcm_set_iv(&gc2, &gk2, 12, iv);
    gcm_update(&gc2, &gk2, 8, datum);
    gcm_decrypt(&gc2, &gk2, &ac2, (nettle_cipher_func*)aes128_decrypt, 32, decoded, encoded);
    gcm_digest(&gc2, &gk2, &ac2, (nettle_cipher_func*)aes128_decrypt, 16, digest);

    for(unsigned char c : src) cerr << hex << +c;
    cout << endl;
    for(uint8_t c : encoded) cerr << hex << +c;
    cout << endl;
    for(uint8_t c : decoded) cerr << hex << +c;
    cout << endl;
}

and the output is

31323334353637383930313233343536373839303132333435363738393031320 80435d9ceda763309ec12a876556f72c14641344ef19fbc5c9ca2f51ebeef f064f9e8db7ae3466979c7b79de95ba6c50714023758ad9abd6eac24d6f565

first line is source and last line is decoded one. They do not match.. Because I am trying to make a template wrapper class of GCM, I cannot use the gcm-aes functions..

Answer 1

GCM only uses the encrypt function of the underlying cipher, similarly to CTR mode. So you need to replace aes128_set_decrypt_key and aes128_decrypt with aes128_set_encrypt_key and aes128_encrypt in all places (1 and 3, respectively).

Your example works for me after that change.

To get proper authentication, you also need to compare the digest after decryption, preferably using memeql_sec.