I've compiled the libcrypto.so shared library from openssl-1.0.1g for various platforms. Using the shared library, my app is able to encrypt/decrypt data between any of the following:
These can decrypt their own data, but NOT each other, nor any of the above:
I first thought this was an endian issue, butAIX/RISC and Linux/PPC are BE, and they seem to play well wit the Linux i386 and x86_64 (LE).
The Solaris systems are:
SunOS pluto 5.9 Generic_Virtual sun4u sparc SUNW,SPARC-Enterprise-T5120
SunOS nemo 5.9 Generic_118559-11 i86pc i386 i86pc
The libcrypto.so was compiled with:
#[openssl-1.0.1g] ./Configure solaris-sparcv9-cc shared
#[openssl-1.0.1g] ./Configure solaris-x86-cc shared
NOTE that the Solaris systems both use "/opt/SUNWspro/bin/cc" as the compiler, and all others use "gcc".
I have verified that I am always starting with the same IV/counter and symkey when encrypting and decrypting. The encryption routines are well embededed in my programs, but this is basically all that I'm doing:
uchar ivec[AES_BLOCK_SIZE], ecount[AES_BLOCK_SIZE], symkey[16];
uint num = 0, bits = 128;
uchar *buf, *ebuf; /* these are malloc'd and buf set to data to encrypt/decrypt */
...
memcpy(symkey, MYTESTKEY, 16);
memset(ecount, 0, AES_BLOCK_SIZE);
memset(ivec, 0, AES_BLOCK_SIZE);
memcpy(ivec, SOMEDATA, 8); /* from RAND_seed() and RAND_bytes() */
AES_set_encrypt_key(symkey, bits, &outkey);
...
AES_ctr128_encrypt(inp, outp, bytes, key, ivec, ecount, &num);
Again, this always works when encrypting/decrypting on the same system, only decodes to junk when one of the systems is Solaris. I'm using very old Solaris OS and boxes, however, since I have to maintain backward compatibility. Thanks in advance!
Just to update, I converted to EVP but no difference. I recompiled libcrypto.so with GCC also with no difference (using solaris64-sparcv9-gcc) - until I removed the link to cc from /usr/bin. Then I recompiled using GCC again and this time it worked. I didn't have time to investigate further, but it implies that some of the objects were still being compiled with cc, even after specifying *-gcc. Unfortunately there's no GCC available for our old Solaris-x86 V9 system, so we won't be supporting V9 on intel anymore.