Unpad/decode OAEP padded decrypted data

Question

I decrypt a byte array on a smartcard and receive data back from the smartcard. This decrypted data is still in the OAEP padding. Is there a simple way in C# to "unpad" decrypted data? The digest algorithm is sha256.

Unfortunately, I was unsuccessful with Bouncy Castle.

Update:

I have tried two variants:

1. I have implemented a CustomRSA cipher:

public class CustomRSA : IAsymmetricBlockCipher
{
    public string AlgorithmName
    {
        get
        {
            return "CustomRSA";
        }
    }

    public int GetInputBlockSize()
    {
        throw new NotImplementedException();
    }

    public int GetOutputBlockSize()
    {
        return 36;
    }

    public void Init(bool forEncryption, ICipherParameters parameters)
    {

    }

    public byte[] ProcessBlock(byte[] inBuf, int inOff, int inLen)
    {
        return inBuf;
    }
}

Unfortunately it does not work

CustomRSA ls = new CustomRSA();
var ba = new OaepEncoding(ls, new Sha256Digest(), new Sha256Digest(), null);
ba.Init(false, null);
return ba.ProcessBlock(paddedData, 0, paddedData.Length);

2. Another idea was to clone the OAEPEncoding class from bouncy castle. Here I tried to replace the line in the DecodeBlock method as follows:

byte[] data = engine.ProcessBlock(inBytes, inOff, inLen);

changed to

byte[] data = inBytes;

Unfortunately, the whole thing doesn't work either

Answer 1

OAEP produces a byte sequence whose length is equal to the key size k (i.e. the modulus length). The first byte is 0x00, see RFC8017, 7.1.1, step 2.i.

Your message may be shorter because the conversion of the numeric value into a byte sequence following decryption may not take the preceding 0x00 values into account.

---

As suggested in the comment by President James K. Polk, the OaepEncoding class of the low level API of BouncyCastle can be used for unpadding. However, this implementation requires as first parameter a class that represents the algorithm used to perform the decryption. This so-called engine implements the interface IAsymmetricBlockCipher (for OAEP the class RsaEngine).

Since in this use case no decryption is to be performed, but only unpadding, i.e. in other words an engine is required that does not decrypt, and there is no such engine among the built-in engines, a custom engine must be implemented.
A simple approach is a class that is derived from RsaEngine and overwrites ProcessBlock() so that no decryption takes place.
This solution requires an RSA key (otherwise: InvalidKeyException: Not an RSA key). The size of this key defines k, i.e. the key size must correspond to the size of the key used for decryption. Apart from this size constraint, any dummy key can be used, as no decryption is performed.

A possible implementation is (e.g. key size k=2048 bits, data size: k/8=256 bytes):

using Org.BouncyCastle.Crypto.Digests;
using Org.BouncyCastle.Crypto.Encodings;
using Org.BouncyCastle.Crypto.Engines;
using Org.BouncyCastle.Security;
...
public class CustomRSAEngine : RsaEngine
{
    public override byte[] ProcessBlock(byte[] inBuf, int inOff, int inLen)
    {
        return inBuf[1..]; // remove the leading 0x00
    }
}
...
// create dummy key
var g = new RsaKeyPairGenerator();
g.Init(new KeyGenerationParameters(new SecureRandom(), 2048)); // key size in bits, adjust to your key/data size!
var dummyKey = g.GenerateKeyPair().Private;
// unpad
var dataPadded = Convert.FromHexString("00853d1a77f1e400c602c26bd97aaaaa408c0dbb95d92ce85713d9418cfb51fab5fc956629e769d703330530ae7ce797919abc396846a5faa81e43769bf2dc3e9c19cb548aad34967d934c8d8ddffc1ac97002ca06c9caf32d5c2f5cab4e31c18276c3860c7f7f542fbd331586b227aaae63f4d0dd94bcd0089ffabf230381bc9d107fa8bb93679415b1a2d1c53d1100d1adf966271c6753ae91ccc29a84baa5b63c1e7c4bed07a23a18c366608a55806e9073485aef9c4a500f71ff8d9888dc92bd7b7f02108fda47e9af81bcc7074b46d87efa45c474be187f8ada412359a1630daf11ff9c3d85622715ec19537ed00289f124c334af49f7ef5e8f1f833be5");
var oaep = new OaepEncoding(new CustomRSAEngine(), new Sha256Digest(), new Sha256Digest(), null);
oaep.Init(false, dummyKey);
var dataUnpadded = oaep.ProcessBlock(dataPadded, 0, dataPadded.Length);

Note that in ProcessBlock() the leading 0x00 byte of the OAEP padding must be removed. If it has already been removed from your data, simply return inBuf.

---

If you want to use your CustomRSA class, the implementation of GetOutputBlockSize() must be corrected. To do this, apply the decryption branch of the GetOutputBlockSize() method of RsaEngine() or RsaEngineCore(). As key size, k i.e. the size of the RSA key used for decryption is to be applied:

public virtual int GetOutputBlockSize()
{
    int keySizeBits = 2048; // key size in bits, adjust to your key/data size!
    return (keySizeBits - 1) / 8; // from RsaEngine or RsaCoreEngine
}

For the ProcessBlock() method, use the implementation of CustomRSAEngine#ProcessBlock().

This approach does not require a dummy key (oaep.Init(false, null)). k is specified in GetOutputBlockSize().