RC2 Implementation differences C# and Python?

Question

For compatibility/legacy reasons I need to use RC2 encryption in CBC mode. I am writing a test - but I get completely different results in C#, Python and with Online Tools, with the (seemingly) same input values.

For all implementations I used the following data:

Data: 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 
Key: 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00    
IV: 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08   
Mode: CBC    
Padding: PKCS7

I tried testing this with Python, C# and by using an online tool called CyberChef. All gave me totally different results.

Python Result: d123f2ac56146f3cebd19b285eb1e1744b828a177778be07

C# Result: f278304ee422a8bbccd54c9157afa818ac4e5b21858ff267

CyberChef Result: c91e276fc97e71acb72426f29c3a6c6f5181d8e83dcf1a98

The python scripts:

from Crypto.Cipher import ARC2
from Crypto.Util.Padding import pad
input = bytes([0]*16)
key = bytes([0]*8)
iv = b"\x01\x02\x03\x04\x05\x06\x07\x08"

cipher = ARC2.new(key, ARC2.MODE_CBC, iv=iv)

msg = cipher.encrypt(pad(input,8,style='pkcs7'))
print("{} {}".format(len(msg), msg.hex()))

C# Script (parts):

public byte[] Encrypt(Rc2CryptoParameters cryptoParameters)
{
    using var outputStream = new MemoryStream();
    using var provider = GetRc2Provider(cryptoParameters);
    using var encryptor = provider.CreateEncryptor();
    using var cryptoStream = new CryptoStream(outputStream, encryptor, CryptoStreamMode.Write);

    cryptoStream.Write(cryptoParameters.Data);
    cryptoStream.Close();
    return outputStream.ToArray();
}

private RC2CryptoServiceProvider GetRc2Provider(Rc2CryptoParameters cryptoParameters)
{
    return new RC2CryptoServiceProvider
    {
        Key = cryptoParameters.Key,
        BlockSize = cryptoParameters.BlockSize,
        KeySize = cryptoParameters.KeySize,
        Padding = cryptoParameters.PaddingMode,
        Mode = cryptoParameters.CipherMode
    };
}

public Rc2CryptoParameters(byte[] data, byte[] key, byte[] iv)
{
    Data = data;
    Iv = iv;
    Key = key;
    CipherMode = CipherMode.CBC;
    PaddingMode = PaddingMode.PKCS7;
}

So - why am I getting different results everywhere? I tried using some CBC testvectors, the only ones I could find were these: http://cryptomanager.com/tv.html

How can I ensure which of the results is correct? Why are all the implementations producing different results?

Answer 1

RC2 is described in RFC2268. It's a block cipher with variable key length and has an additional parameter called effective key length in bits, see RFC2268, Section 2. In the two codes and on the website, a different effective key length in bits is used, causing the different results.

In the Python code, when using PyCryptodome, the effective key length in bits is specified with the parameter effective_keylen upon creation of the ARC2-cipher instance, which may have values between 40 and 1024, where 1024 is the default value. Since the parameter isn't explicitly specified in the posted Python code, the default value is used. Note that this parameter is described in the PyCrypto documentation, but not in the PyCryptodome documentation.

The ciphertext of the website results for effective_keylen = 128. On the website there seems to be no possibility to change the effective key length in bits.

The ciphertext of the C# code can't be reproduced, probably because the IV isn't set in GetRc2Provider (so that the randomly generated IV is used). If this is fixed, it turns out that the effective key length in bits (RC2CryptoServiceProvider#EffectiveKeySize) is implicitly set to the actual key length. If the parameter is explicitly switched to a another value, a System.Security.Cryptography.CryptographicUnexpectedOperationException: EffectiveKeySize must be the same as KeySize in this implementation. is thrown.