Byte difference between python cryptography and JS Forge when making pkcs #7 signing

Question

I need to connect to AirWatch API with pkcs 7 signed data for security.

On this page I found a sample of how to make the connection in python, and this works correctly when I provide my certificate and key files. Here is a sample of using the cryptopgraphy library using a generic certificate and key for testing:

import base64
from cryptography.hazmat.primitives.serialization import pkcs12, pkcs7
from cryptography.hazmat.primitives import hashes, serialization
import requests

signing_data = "/api/mdm/devices/search" # the part of the url to be signed

certfile = open("/home/claude/certificate.p12", 'rb')
cert = certfile.read()
certfile.close()


#p12 format holds both a key and a certificate
key, certificate, additional_certs = pkcs12.load_key_and_certificates(cert, "motdepasse".encode())

#print (certificate.subject)
#print (key)

pem = key.private_bytes(
    encoding=serialization.Encoding.PEM,
    format=serialization.PrivateFormat.TraditionalOpenSSL,
    encryption_algorithm=serialization.NoEncryption()
)

options = [pkcs7.PKCS7Options.NoCapabilities,  pkcs7.PKCS7Options.DetachedSignature]

signed_data = pkcs7.PKCS7SignatureBuilder().set_data(
    signing_data.encode("UTF-8"))

signed_data = signed_data.add_signer(certificate, key, hashes.SHA256())

signed_data = signed_data.sign(serialization.Encoding.DER, options)

print(signed_data)

For

and here is an sample of the byte output converted to UTF-8. I can use this signed data to connect to the AirWatch API correctly:

0Ç D    *ÜHܘ
   †Ç 50Ç 1   1 0
    `ÜH e      0    *ÜHܘ
   †Ç  0Ç  0Ç ˘    (ƒ'±M
–1ï◊ÌÛ«õR ]µ0
    *ÜHܘ
     0E1 0    U    AU1 0   U   
Some-State1!0   U 
  Internet Widgits Pty Ltd0  
230531184611Z 
230829184611Z0E1 0    U    AU1 0   U   
Some-State1!0   U 
  Internet Widgits Pty Ltd0Ç "0
    *ÜHܘ
      Ç   0Ç 
 Ç   ÷ü$’% I        ∞ˆv∆ ºe_Ö4*Úˇ ≥Ì‘ {≈i‰Aı C>˘Ã\åZÔu€óŒ7£V9DÇ fr ˜uô ≠í
‹%nˇòg^∞ñÁ —˝Î ÒW &ììl‡û∏`± ´  ä‹ÖpO‰-n “æ∆È >î=ÈØπÌ  xcï¢Ñ«ˇo‚ ¬¬ ¸1 8 &cqë yÛÇ®tB ùé%ªy ˛5É≈  ŸπÎT êπ¯„?C'*(v ºª
·r®µ≤µ8 V   C…›wÑb¢n¿Éµ÷ß∏-7ÚqÑE Û2‹F /E®ñ5 üj>»¶&∆˙ôî˝¡Agå≥˘V˘ ñ"‹flA⁄Ñ·¯ ^   "%     0
    *ÜHܘ
      Ç   ¡Íà ˘\7T
9 s¿˛‚L4î6  ¶Säπ:È[Q}˝œ≠R§Cß=  ‹• ¥¸œ9É]o˝ iÅ\B)7n ≈©“Úò √F‘‚¥Œ• I≤^Fá≥¡v9R  a%  Í V ~y5 ®≠—_vø Jë>a}°8èj$Ü5  Éî’êµ<® ˙ÅåwÏò≠´6ªâ[◊NèQ‚ú  GG´˛uoN¸€óª ªˇçðs;Ε Ô2Ò+Æ ∫"_ä
1P≈†ß…çø∆yû  QöRtm= ƒÙ ÔßsÏ
∆≈∞ ™fl„¡r <ÑkxÓhª rZú‡6 Xı“´‘”æE '3U≈èˇÕá™ U%1Ç Û0Ç Ô   0]0E1 0   U    AU1 0   U   
Some-State1!0   U 
  Internet Widgits Pty Ltd    (ƒ'±M
–1ï◊ÌÛ«õR ]µ0
    `ÜH e      †i0      *ÜHܘ
     1      *ÜHܘ
   0    *ÜHܘ
     1  
230618112934Z0/     *ÜHܘ
     1"  SnÄ∞ïÓoDW≤NiÅ ËKéA 8ÈA„wø œ‘?p’ˆ0
    *ÜHܘ
      Ç   éü≠6 õ  ÷ÚˆÂûÛñI<2‡¿ıp∞$Bfl‹‹Óµ  ?Ü—s Ökû0s#y û˚•£îc· ù,4∆≠ë1Á§  ˙ F$ Û:n{$£Á?ÎKDΩ˜™êÛJCp v æ7H
ÉaâS^äjm“ÓK‘≠:Ëàp ÆéQ©Õ4≥∫ `˛`9 TÓa xYz]z‹–-·qÖ$E7Mvøuà äâq°Ú@LVPT3=—÷íTÍG√ Ì˘1j Eû']◊b” flèo)d?˚9Æ:íd%y€Îªë=ø5ZlÄ2πËãöN•*K  a#›·hD› ̸qò§Ω¨™r-H 3”2OwÄg»-π…6KI[

now I would like to use this in javscript. here is my code using forge-js:

function withPem() {
  var signing_data = "/api/mdm/devices/search"; // the part of the url to be signed
  var keyAndCert = getKeyAndCert()
  var key = keyAndCert.key;
  var cert = keyAndCert.cert;

  options = {}

  options['detached'] = true
  options['certificate'] = forge.pkcs7.OID_CAPABILITY_NONE
  var p7 = forge.pkcs7.createSignedData(options=options);
  p7.content = forge.util.createBuffer(signingData, 'utf8');

  p7.addCertificate(cert);
  p7.addSigner({
    key: key,
    certificate: cert,
    digestAlgorithm: forge.pki.oids.sha256,
    authenticatedAttributes: [{
            type: forge.pki.oids.contentType,
            value: forge.pki.oids.data,
        }, 
        {
            type: forge.pki.oids.messageDigest
        }, 
        {
            type: forge.pki.oids.signingTime, 
            value: new Date()
        }]
  });



  var signed = p7.sign({ detached: true })
  
  var bufferOut = forge.asn1.toDer(p7.toAsn1()).getBytes();

}


function getKeyAndCert() {
  var certPem = "cert as string"
  var privateKeyPem = "private key as string"
  return { cert: certPem, key: privateKeyPem };
}

but when I provide the the same key and certificate, the output is very very similar but not the same:

0_  *H÷
 P0L10
    `He�0&  *H÷
 /api/mdm/devices/search 00ù (Ä'±M
Ð1×íóÇR]µ0
    *H÷
�0E10   UAU10U
Some-State1!0U
Internet Widgits Pty Ltd0
230531184611Z
230829184611Z0E10   UAU10U
Some-State1!0U
Internet Widgits Pty Ltd0"0
    *H÷
��0
�Ö$Õ%I      °övƼe_4*òÿ³íÔ{ÅiäAõC>ùÌ\ZïuÛÎ7£V9Dfr÷u­
Ü%nÿg^°çÑýëñW&là¸`±«ÜpOä-nÒ¾ÆéÊ>=鯹íxc¢ÇÿoâÂÂü18&cqyó¨tB%»yþ5ÅÙ¹ëTʹøã?C'*(v¼»
ár¨µ²µ8V    CÉÝwb¢nÀµÖ§¸-7òqEó2ÜF/E¨5j>Ȧ&ÆúýÁAg³ùVù"ÜßAÚáø^"%�0
    *H÷
��ÁêÌè÷\7T
9sÀþâL46¦S¹:é[Q}ýÏ­R¤C§=ÊÜ¥´üÏ9]oýi\B)7nÅ©ÒòÃFÔâ´Î¥I²^F³Áv9ðR a%êV~y5¨­Ñ_v¿J>a}¡8j$ð5Õµ<¨úwì­«6»[×NQâGG«þuoNüÛ»»ÿÌ¡s;ë¥ï2ñ+®º"_
1PÅ §É¿ÆyQRtm=Äôï§sì
ÆÅ°ªßãÁr<kxîh»ÊrZà6 XõÒ«ÔÓ¾E'3UÅÿͪU%1ó0ï0]0E10 UAU10U
Some-State1!0U
Internet Widgits Pty Ltd (Ä'±M
Ð1×íóÇR]µ0
    `He� i0 *H÷
    1   *H÷
0/  *H÷
    1" Sn°îoDW²NièKA8éAãw¿ÏÔ?pÕö0   *H÷
    1
230626090435Z0
    *H÷
��s4Q ±cPC%¯[óBÚà Q3¶ZF°'Úø?öÌôÁËñ¨+7"¬áÜtÒ1N�ñÞ&@áKÓ@a[5[¶ú)ãë#µÈN¾²Ç GE!eQ²H×=ÜwÂ)ÛÕ8=êÿÈPz1;m=ÕPi[®
{?¥NñÎ:7 ·ó@q![k|þûú~ÞÏZº>S-¼º¼Ù/ܾíg{}>mé;RmÀ!S:öÒã8ê¬Ð§KêGFv¿Üq~ÅÅ®r].àK]¼Ý¼dzlußø5³ÀON

I can't find anything that I am going wrong with the forge package but the output does not work with the API. Is there a difference between the way forge does pkcs7 signing compared to cryptography in python? what can I do to fix this js code? unfortunately I need to use JS as it's the requirement I have been given for my project.

Edit:

Here is the P12 in pem format (don't worry, this is not my production key):

Bag Attributes
    localKeyID: B5 EA C1 95 EF 41 D4 41 D2 3C FB 63 01 74 36 10 80 55 CC BF 
subject=C = AU, ST = Some-State, O = Internet Widgits Pty Ltd
issuer=C = AU, ST = Some-State, O = Internet Widgits Pty Ltd
-----BEGIN CERTIFICATE-----
MIIDETCCAfkCFCAdKMQnsU0K0DGV1+3zx5tSDF21MA0GCSqGSIb3DQEBCwUAMEUx
CzAJBgNVBAYTAkFVMRMwEQYDVQQIDApTb21lLVN0YXRlMSEwHwYDVQQKDBhJbnRl
cm5ldCBXaWRnaXRzIFB0eSBMdGQwHhcNMjMwNTMxMTg0NjExWhcNMjMwODI5MTg0
NjExWjBFMQswCQYDVQQGEwJBVTETMBEGA1UECAwKU29tZS1TdGF0ZTEhMB8GA1UE
CgwYSW50ZXJuZXQgV2lkZ2l0cyBQdHkgTHRkMIIBIjANBgkqhkiG9w0BAQEFAAOC
AQ8AMIIBCgKCAQEA1p8k1SV/SQkJsPZ2xg68ZV+FNCry/w6z7dQUe8Vp5EH1AkM+
+cxcjFrvdduXzjejVjlEghNmcgj3dZkZrZIN3CVu/5hnXrCW5wbR/esS8VcSJpOT
bOCeuGCxFqsYBYrchXBP5C1uC9K+xunKPpQ96a+57Q4deGOVooTH/2/iC8LCG/wx
CDgeJmNxkQh584KodEIanY4lu3l//jWDxR8C2bnrVMqQufjjP0MnKih2Fby7DeFy
qLWytTgVVglDyd13hGKibsCDtdanuC038nGERQXzMtxGfy9FqJY1FJ9qPsimJsb6
mZT9wUFnjLP5VvkdliLc30HahOH4EV4dDA8iJQIDAQABMA0GCSqGSIb3DQEBCwUA
A4IBAQDB6swG6PdcN1QNORNzwP7iTDSUNhseplOKuTrpW1F9/c+tUqRDpz3KGdyl
HbT8zzmDXW/9G2mBXEIpN24dxanS8pgZw0bU4rTOpRlJsl5Gh7PBdjnwUhEgYSUE
BeoaVhh+eTUUqK3RX3a/B0qRPmF9oTiPaiSG8DV/EYOU1ZC1PKgL+oGMd+yYras2
u4lb106PUeKcBghHR6v+dW9O/NuXux+7/43MoXM766UB7zLxK64HuiJfigoxUMWg
p8mNv8Z5nhwOUZpSdG09C8T0He+nc+wNxsWwEqrf48FyGDyEa3juaLvKclqc4DYg
WPXSq9TTvkUQJzNVxY//zYeqC1Ul
-----END CERTIFICATE-----
Bag Attributes
    localKeyID: B5 EA C1 95 EF 41 D4 41 D2 3C FB 63 01 74 36 10 80 55 CC BF 
Key Attributes: <No Attributes>
-----BEGIN PRIVATE KEY-----
MIIEvwIBADANBgkqhkiG9w0BAQEFAASCBKkwggSlAgEAAoIBAQDWnyTVJX9JCQmw
9nbGDrxlX4U0KvL/DrPt1BR7xWnkQfUCQz75zFyMWu9125fON6NWOUSCE2ZyCPd1
mRmtkg3cJW7/mGdesJbnBtH96xLxVxImk5Ns4J64YLEWqxgFityFcE/kLW4L0r7G
6co+lD3pr7ntDh14Y5WihMf/b+ILwsIb/DEIOB4mY3GRCHnzgqh0QhqdjiW7eX/+
NYPFHwLZuetUypC5+OM/QycqKHYVvLsN4XKotbK1OBVWCUPJ3XeEYqJuwIO11qe4
LTfycYRFBfMy3EZ/L0WoljUUn2o+yKYmxvqZlP3BQWeMs/lW+R2WItzfQdqE4fgR
Xh0MDyIlAgMBAAECggEAKQHbXcZ+XYwWh/NvmkQyhwQLRX53U3iRtH1zNHrx0qUv
lTEYFU6Q2Fh/rHs6tDI5ST5D8r6WMm+4KIYKO/nOICQe40NRbOw8yQOql+OUiPxk
AW7tGj6I1R3UeEpUmqp/nBdrjGOJxUSNIyCEfhSBB+eFlN+/jcMpUhYgyJOuEyTX
oM7lSlqBL+oOZa+UY/hRBGdR8aFAmGXskPoOGDtf3qmkhMSC1/QjF6T9M7bb5xcL
fT2YhrfUUOsYtqpuAyZGDk1fmbf/nLegs9370lblCTX/bsQqyZV8SaOQi5XUOL/z
EYeGDyuprIEBX9jy5KU33J9tCcOretWXWMRUUebRAQKBgQDhQVviO7sQ7dJT/jc9
yN5Tf4fg2dRKxSEHilebDhyyvhRR/5/YP2fi6lIpJCMXgLh+vv5TI/32nhAyZOTD
ZRp2eCx/jUOwNRPyNxmfBNg1Pt1Qthy5D0Opl2Wt2inoNc3j+q3IkLZiaDJSSnRO
EoqtaIJy0TWUsnyUfgJiTyMdoQKBgQDz6jwW1qIGGQ7CEkQMDPFYQNAs6AKXLVRS
NDEDNFd7CighW9n5har8FXtCIOyO6a0h2QaT65Wx2mh/QwSYUls0ovoyRfUsYQGQ
1oS9rgtROtefUsOw+VbnCGbS43uFvnpfrwKC0LHwmJXFTiP5APa/oH7knOVxzpzJ
3sMx4YjOBQKBgQCL5+1q8ZB5rkzhsFadQGKeV+qMRJ9vpUqjhVBuVPCMMDUszOl6
Bb+/l6xaM0C8e02cI4KRHxzBDWGf+zx/BA/Qn0l8G8B79CukWIbIVtj3EUmitMnY
Q1vSPN+BgKxgtvJfdDZ2CTPOoUsIA4iDaU7K78t+BuURq15nWHCgoOh9oQKBgQCP
jKk0n7jXcePXn7xggzV+xRY/d4QeyNS5VHIL+sAJb57SkyYjzeElXtcdwha2vRvh
scJHR/zfoTSiwSRxKPb4cXpiH/380lKDlVyl7UpH0iOYZrM48mWMrsslDjBiNAn9
ShhmOMCgYoyyhBxzrXeKq8BCd3wpkHmB7RJfxuYmqQKBgQCGdzWUTzOnl2JHa9OS
yY5UbElsY/G4JwEjzV+C4eIEKJAmA8wnuhNOdKASM+HTRe0aE0w/ePNqBPU1RLcS
lwJe+NLnEq2pQbc+iwlUw5gdpMgpBanI6ZNW3QFAJyDQBMtBAe/jCdQrONFW0/HG
u4025i+kacAZrZrWTGG7LWYsWA==
-----END PRIVATE KEY-----

I also posted this to the Forge github as I'm starting to consider it unintended behaviour: https://github.com/digitalbazaar/forge/issues/1040

Answer 1

The outputs of both codes are ASN.1/DER encoded, so they should be output with a binary-to-text encoding like hex or Base64 and not UTF8 decoded. UTF-8 decoding corrupts the data.

For instance, for hex encoding in the Python code use signed_data.hex() and in the NodeJS code use Buffer.from(bufferOut, 'latin1').toString('hex').

The hex or Base64 encoded data can then be loaded into an ASN.1 parser, e.g. https://lapo.it/asn1js/, which makes inspecting the data and comparing it much easier.

This shows that the two outputs differ for three reasons:

• Both files have a timestamp corresponding to the signing time. This is generally different.
• The order of signing time and message digest is reversed.
• Python uses as OID for the signature algorithm: 1.2.840.113549.1.1.11 sha256WithRSAEncryption, and forge: 1.2.840.113549.1.1.1, rsaEncryption. Both are correct, but the former is a more specific OID.

ASN.1 from Python code:

ASN.1 from NodeJS code:

The different signing time should not be a problem. To make sure that both codes give identical output regarding the last two points, the NodeJS code has to be changed e.g. as follows:

p7.addSigner({
    key: key,
    certificate: cert,
    digestAlgorithm: forge.pki.oids.sha256,
    authenticatedAttributes: [{
            type: forge.pki.oids.contentType,
            value: forge.pki.oids.data,
        }, 
        {
            type: forge.pki.oids.signingTime,                  // Change 1: switch order
            value: new Date()
        },
        {
            type: forge.pki.oids.messageDigest,
        }],
});
p7.signers[0]['signatureAlgorithm'] = '1.2.840.113549.1.1.11'  // Change 2: apply 1.2.840.113549.1.1.11   

With this, both codes provide identical output except for the signing time (and consequently the signature).

If the problem is really caused by the differences regarding the order and/or the OID, it should not occur after the changes. In this case, you can change the NodeJS code one by one to determine if the order or the OID (or both) is the exact cause of the problem (this is not possible for me as it would require access to the AirWatch API or knowledge of the AirWatch API logic/code).
However, if the problem still occurs after the changes, it must have some other cause (e.g. not related keys, etc.).