How to construct radomized CTR mode?

Question

How can I change this deterministic counter mode to randomized counter mode cipher? I learned that we have to set a fixed counter value to random one... I don't know how to do it... Can someone explain it in detail? Coding noob requires a detailed explanation...

#include <stdio.h>
#include <openssl/bn.h>
#include <openssl/aes.h>
#define mSize 4


void block_xor(unsigned char z[16], unsigned char x[16], unsigned char y[16])
{
    for (int i=0; i<16; i++)
        z[i] = x[i] ^ y[i];
}

int main(int argc, char* argv[]){
    BIGNUM *key = BN_new();
    unsigned char userkey[32];
    int size = 128;
    AES_KEY enckey,deckey;

    BN_rand(key, size, BN_RAND_TOP_ANY, BN_RAND_BOTTOM_ANY);
    BN_bn2bin(key, userkey);

    AES_set_encrypt_key(userkey, size, &enckey);
    //AES_set_decrypt_key(userkey, size, &deckey);

    unsigned char m[16*mSize+1] = "Practice randomized CTR mode";
    unsigned char enc[16*mSize+1] = {0};
    unsigned char dec[16*mSize+1] = {0};

    unsigned char counter[16];
    unsigned char blockkey[16]; 

    for(int i =0; i<mSize; i++){
        sprintf(counter, "%d", i);
        AES_encrypt(counter, blockkey, &enckey); // *** F(k,i) == F(enkey, counter) == blockkey 
        block_xor(&enc[16*i], &m[16*i], blockkey);
        //enc[i] = m[i] ^ blockkey
    }
    printf("enc: %s\n", enc);

    for(int i =0; i<mSize; i++){
        sprintf(counter, "%d", i);
        AES_encrypt(counter, blockkey, &enckey);
        block_xor(&dec[16*i], &enc[16*i], blockkey);
    }
    printf("dec: %s\n", dec);

    return 0;

}

Answer 1

Create a nonce of a certain size by creating a byte array (a char array in C). For instance, you could use one of 8 bytes (which is probably the minimum, because you have to keep the birthday problem in mind). Then you fill this array with random bytes.

The nonce is then copied to the most significant bits of the counter. The counter is commonly a big endian value, so that means from the left-hand side (lower indices) to the right. The remaining bytes of the counter are set to zero.

Finally, if you have a randomized nonce, you need to include it with your ciphertext, otherwise the entity doing the encryption cannot know it and decryption will fail (you're cheating by doing both in the same method).

---

Now you can create a key stream by encrypting the counter, and increasing the counter afterwards. The plaintext / ciphertext is then XOR-ed with the key stream.

This functionality is either missing and / or incorrect in your code by the way. You don't see this because the same mistakes are made during encryption and decryption. Beware that running code doesn't mean secure code in cryptography.