Error RSA encrypting in C# and decrypting in Go

Question

I am getting an error decrypting a message in go that was encrypted in C# (using corresponding public/private keys)

My client is written in C# and my server is written in Go. I generated a private and public key via go's crypto/rsa package (using rsa.GenerateKey(random Reader, bits int)). I then store the public key file generated where the client can access it and the private key where the server can access it. I encrypt on the client with the following code (using bouncy castle):

   public static string Encrypt(string plainText)
   {
      byte[] plainTextBytes = Encoding.UTF8.GetBytes(plainText);

      PemReader pr = new PemReader(
        new StringReader(m_publicKey)
      );
      RsaKeyParameters keys = (RsaKeyParameters)pr.ReadObject();

      // PKCS1 OAEP paddings
      OaepEncoding eng = new OaepEncoding(new RsaEngine());
      eng.Init(true, keys);

      int length = plainTextBytes.Length;
      int blockSize = eng.GetInputBlockSize();
      List<byte> cipherTextBytes = new List<byte>();
      for (int chunkPosition = 0; chunkPosition < length; chunkPosition += blockSize)
      {
          int chunkSize = Math.Min(blockSize, length - chunkPosition);
          cipherTextBytes.AddRange(eng.ProcessBlock(
              plainTextBytes, chunkPosition, chunkSize
          ));
      }
      return Convert.ToBase64String(cipherTextBytes.ToArray());
}

The go server parses this string from the header and uses the private key to decrypt:

func DecryptWithPrivateKey(ciphertext []byte, priv *rsa.PrivateKey) []byte {
   hash := sha512.New()

   plaintext, err := rsa.DecryptOAEP(hash, rand.Reader, priv, ciphertext, nil)
   if err != nil {
       fmt.Fprintf(os.Stderr, err.Error())
   }
   return plaintext
}

The decryption function throws crypto/rsa: decryption error. If I try pasting the cipher text directly into go (rather then sending from the client), the same error occurs.

NOTE: in order to get the public key to load, I needed to change the header from:

-----BEGIN RSA PUBLIC KEY----- 
...

to

-----BEGIN PUBLIC KEY----- 
...

and the same for the footer. I am assuming this is a formatting issue but not sure how to go about solving.

EDIT: it seems that golang OAEP uses sha256 and bouncy castle uses SHA-1. Go's documentation specifies that the hash for encryption and decryption must be the same. This seems likely to be the issue? If it is, how can I change the hashing algorithm used by either go or C#?

Answer 1

Yes, you need to match the hash. In GoLang you've already set it to SHA-512 if I take a look at your code. Using SHA-256 at minimum should probably be preferred, but using SHA-1 is relatively safe as the MGF1 function doesn't rely on the collision resistance of the underlying hash. It's also the default for most runtimes, I don't know why GoLang decided against that.

Probably the best is to set SHA-512 for both runtimes, so here is the necessary constant for .NET.

---

Note that the underlying story is even more complex as OAEP uses a hash over a label as well as a hash within MGF1 (mask generation function 1, the only one specified). Both need to be specified in advance and generally the same hash function is used, but sometimes it is not.

The label is generally empty and most runtimes don't even allow setting it, so the hash value over the label is basically a hash-function specific constant that doesn't matter for security. The constant just manages to make things incompatible; "More flexible" isn't always a good thing.