Using Ruby's OpenSSL lib, when I try to both encrypt and sign a message as a PKCS7, I find that some extra \r characters get inserted, and I can't decrypt the message.
Below I have a toy example. I more thorough example is available on this paste.
# The sender encrypts the message
crypted = OpenSSL::PKCS7.encrypt([recipient_cert], "message").to_der
# The sender signs the message
signed = OpenSSL::PKCS7.sign(sender_cert, sender_key, crypted).to_der
# The recipient extracts data from the signed PKCS7
p7_signed = OpenSSL::PKCS7.new(signed)
store = OpenSSL::X509::Store.new
p7_signed.verify(nil, store, nil, OpenSSL::PKCS7::NOVERIFY)
unsigned = p7_signed.data
# The tries to decrypt the data
OpenSSL::PKCS7.new(unsigned).decrypt(recipient_key, recipient_cert)
# => ArgumentError: Could not parse the PKCS7: nested asn1 error
Why does the data extracted from p7_signed not match the data in crypted? When I examine the two, I find that unsigned contains a few \r characters here and there which crypted does not have. What can I do about this?
(I'm doing this on Linux, in case the question of CRLF vs LF comes up.)
You should supply the PKCS7_BINARY flag to PKCS7.sign(), probably available as OpenSSL::PKCS7::BINARY. I don't know Ruby at all but reading the docs at ruby-doc.org and the openssl man pages leads me to believe this is the issue.
From the openssl man page for PKCS7 (with a minor correction by me):
Normally the supplied content is translated into MIME canonical format (as required by the S/MIME specifications), but if PKCS7_BINARY is set no translation occurs. This option should be used if the supplied data is in binary format otherwise the translation will corrupt it.
Since your input to the OpenSSL::PKCS7.sign(...) is the DER-encoded output from OpenSSL::PKCS7.encrypt(...), that's essentially a binary format and not text.