How to replace mcrypt with openssl?

Question

I am currently upgrading an existing PHP application from PHP 5.4 to PHP 8.1. I have managed to restore all functionality except for encryption and decryption of data. My application communicates with a third party server, so just running an old PHP version to decrypt using mcrypt and reencrypt using openssl is not possible. I have seen a lot of similar threads, was however unable to find a solution to my issue.

This is the old set of functions (using mcrypt):

static function encrypt($key, $plain, $salt = null)
    {
        if (is_null($salt))
        {
            $salt = QuickBooks_Encryption::salt();
        }
        
        $plain = serialize(array( $plain, $salt ));
        
        $crypt = mcrypt_module_open('rijndael-256', '', 'ofb', '');

        if (false !== stripos(PHP_OS, 'win') and 
            version_compare(PHP_VERSION, '5.3.0')  == -1) 
        {
            $iv = mcrypt_create_iv(mcrypt_enc_get_iv_size($crypt), MCRYPT_RAND);    
        }
        else
        {
            $iv = mcrypt_create_iv(mcrypt_enc_get_iv_size($crypt), MCRYPT_DEV_URANDOM);
        }

        $ks = mcrypt_enc_get_key_size($crypt);
        $key = substr(md5($key), 0, $ks);
        
        mcrypt_generic_init($crypt, $key, $iv);
        $encrypted = base64_encode($iv . mcrypt_generic($crypt, $plain));
        mcrypt_generic_deinit($crypt);
        mcrypt_module_close($crypt);
        
        return $encrypted;
    }

    static function decrypt($key, $encrypted)
    {
        $crypt = mcrypt_module_open('rijndael-256', '', 'ofb', '');
        $iv_size = mcrypt_enc_get_iv_size($crypt);
        $ks = mcrypt_enc_get_key_size($crypt);
        $key = substr(md5($key), 0, $ks);
        
        //print('before base64 [' . $encrypted . ']' . '<br />');
        
        $encrypted = base64_decode($encrypted);
        
        //print('given key was: ' . $key);
        //print('iv size: ' . $iv_size);
        
        //print('decrypting [' . $encrypted . ']' . '<br />');
        
        mcrypt_generic_init($crypt, $key, substr($encrypted, 0, $iv_size));
        $decrypted = trim(mdecrypt_generic($crypt, substr($encrypted, $iv_size)));
        mcrypt_generic_deinit($crypt);
        mcrypt_module_close($crypt);
        
        //print('decrypted: [[**(' . $salt . ')');
        //print_r($decrypted);
        //print('**]]');
            
        $tmp = unserialize($decrypted);
        $decrypted = current($tmp);
        
        return $decrypted;
    }

And this is my best approximation using openssl:

static function new_encrypt($key, $plain, $salt = null)
    {
        if (is_null($salt))
        {
            $salt = QuickBooks_Encryption::salt();
        }
        
        $plain = serialize(array( $plain, $salt ));
        
        $method = "AES-256-OFB";
        $iv_len = openssl_cipher_iv_length($method);
        $iv = openssl_random_pseudo_bytes($iv_len);

        $key = substr(md5($key), 0, 32);
        $encrypted = openssl_encrypt($plain, $method, $key, OPENSSL_RAW_DATA, $iv);

        return base64_encode($iv . $encrypted);
    }
    
    static function new_decrypt($key, $encrypted)
    {
        $iv_size = openssl_cipher_iv_length('aes-256-ofb');
        $key = substr(md5($key), 0, 32);

        $encrypted = base64_decode($encrypted);
        $iv = substr($encrypted, 0, $iv_size);
        $encrypted = substr($encrypted, $iv_size);

        $decrypted = openssl_decrypt($encrypted, 'aes-256-ofb', $key, OPENSSL_RAW_DATA, $iv);
        $tmp = unserialize($decrypted);
        $decrypted = current($tmp);

        return $decrypted;
    }

Both function sets can encrypt and decrypt data; however they are not compatible with each other.

What am I missing here?

Answer 1

First and foremost: Rijndael is not AES.

While AES is descended from/closely related to Rjindael, and may even have certain compatible implementations under very strict interpretations, more often than not they are not compatible at all. The implementations of Rijndael in mcrypt and AES OpenSSL are not compatible. [note: not a cryptographer, some amount of this paragraph might be BS aside from the words 'not compatible']

That said, you can use a library like phpseclib/mcrypt_compat to shim Rijndael and other functionality back into PHP8 and above.

However, since this library code will be running in userspace rather than a compiled extension the performance will likely be noticeably worse. My suggestion would be to use mcrypt_compat to migrate older encryption versions to something OpenSSL-compatible, or at least more current and broadly used/implemented, like AES.