I'm trying to encrypt TempBuffer using RSA private key (64 bytes length) and send it to output buffer:
rsaCipher = Cipher.getInstance(Cipher.ALG_RSA_PKCS1_OAEP, false);
rsaKey = new KeyPair(KeyPair.ALG_RSA,KeyBuilder.LENGTH_RSA_512);
rsaCipher.init(rsaKey.getPrivate(), Cipher.MODE_ENCRYPT);
len = rsaCipher.doFinal(SeedBuffer, (short)0, len_buffer, buffer, (short)0);
why does the last line returns an ILLEGAL_VALUE CryptoException (0x0001)?
which one of inputs is wrong?
SeedBuffer is a 256 byte length buffer which maintains the secret.
buffer has enough length since apdu.setIncommingAndReceive() is executed before.
len_buffer has the value 0x40 meaning 64 bytes as length.
The most likely issue is that you cannot encrypt 256 bytes using OAEP directly. Normally you can fix this by encrypting a random symmetric key and then use that to encrypt the 256 byte value. However, in this case that would only work because RSA-OAEP using SHA-1 has an overhead of 42 bytes, leaving 22 bytes which is just enough for an AES key.
Encryption operations are performed with the public key and decrypt with the private key. So that may throw an error, although in principle the implementation could use modular exponentiation with the private key to create an (insecure ciphertext).
Note as well that most of the Java Card implementations are required to be secure, and RSA-512 is definitely not considered secure. Breaking RSA-512 takes about 2^30 ops minimum, which means that the security in bits is about 30. So a JC implementation may error even if you switch to the public key (which you should).
The final solution would be to use RSA-2048 (or something close if you need to fit it into a single APDU response), using the public key to encrypt. If the "seed" is going to act as a (symmetric) secret or to feed a RNG of sorts, then please note that 256 bits of entropy is more than enough. So although you could go for an RSA / AES hybrid crypto-system, you might as well directly encrypt the seed.