I am trying to understand the behavior of some code which I did not create but which I may have to fix. The code behaves differently with accented characters according to the value (at JVM startup time - changing the value later programmatically does not seem to have an effect) of the JVM system property file.encoding.
The code creates and initializes an instance of RSACipher :
final Cipher cipher = Cipher.getInstance(DecryptInformation.RSA_ECB_PKCS1_PADDING);
cipher.init(Cipher.DECRYPT_MODE, decryptingKey);
And then in a loop it uses that instance to decrypt encryped bytes.
byte[] plain = cipher.doFinal(scrambled, 0, i);
I verified that regardless of the value of file.encoding, the value of scrambled stays unchanged, but the problem is that the bytes in plain change where there are accented characters. In my unit test (written in response to a problem seen at runtime in a wildfly application), there's a lowercase e-accute character (c3a9 in UTF-8). If the test is launched with the VM argument -Dfile.encoding=ISO-8859-1, I get c3a9 in the variable plain for the character. If, on the other hand, I start the test with -Dfile.encoding=UTF-8, I get c383c2a9 instead for the same input bytes, and I don't know how to fix that systematically for all characters.
Can someone explain the effect that file.encoding has on the decryption and how to prevent the character from being corrupted? The application runs, for example, on wildfly on linux, where the default file.encoding is UTF-8, and changing this value for the whole application may create other side-effects, so requiring a change to file.encoding is not an option.
Finally, it's kind of a false alert, because, although having the default system encoding in ISO allowed decryption that preserved accented characters, the real problem was during creation of the encrypted data, We received from a Swagger UI a String encoded in ISO. To fix the data I had to, before encryption, convert it to UTF via code like this :
new String(textToEncrypt.getBytes(java.nio.charset.StandardCharsets.ISO_8859_1), java.nio.charset.StandardCharsets.UTF_8);
Now the decoding works when the default file encoding is UTF-8. I thought it would be good to leave this info here in case other developpers encounter similar problems.