Decoding https traffic with scapy

I am trying to capture https traffic with scapy. So far I can see the TLS handshakes, but I am having trouble decoding the application data using a NSS keyfile. I'm using scapy 2.5.0rc2.dev39.

The structure of my script is: (1) set up the conf to point to the NSS file that will be created; (2) kick off a thread to get the sniff going; and then (3) do a https request on the main thread that generates the traffic that gets sniffed on the other thread.

I suspect my problem is a chicken-and-egg problem: I'm giving scapy the name of a file that doesn't exist yet and asking sniff to use it to decode traffic. But to generate the file, I need to generate traffic. But then it is too late to set up the sniff. And so on.

I'm basing the setup on the conf file on the code example in https://github.com/secdev/scapy/pull/3374. Is there a way to get the NSS file parsed and applied after the packets are sniffed? (Note: I have looked at https://github.com/secdev/scapy/blob/master/doc/notebooks/tls/notebook3_tls_compromised.ipynb, but that example applies the NSS file to a captured pcap file. I don't see how to apply that to my scenario. Unless I export the sniffed packets to a pcap file and then apply the now-existent NSS file to it?)

Here's my code:

from scapy.all import *
from time import sleep
from IPython import embed
myiface = 'lo'
mycount = 30
response_time_delta = 0.0
NSS_SECRETS_LOG = "secrets.log"
SERVER_HOST = "127.0.0.1"
SERVER_PORT = 8443

def analyze_https_sniffed_traffic():
    # sniff traffic for mycount packets
    myfilter = ""
    myprn = lambda x:x.summary()
    sniff_out = sniff(filter=myfilter,prn=myprn,count=mycount,iface=myiface)

    # iterate through the sniffed packets to report on contents
    for idx,s in enumerate(sniff_out):

        print("===\npacket %d\n%s" % (idx,s.summary()))
        # if we can convert to a TLS packet, print out TLS summary
        if s.haslayer('IP') and hasattr(s,'load'):
            tls_r = TLS(s.load)
            print(tls_r.summary())
            # if this is TLS application data, do a complete show
            if tls_r.type == 23:
                print(tls_r.show())
            #embed()
    return

def send_https_request_and_analyze():
    import http.client, ssl
    # start another thread that sniffs traffic and analyzes their contents
    t = threading.Thread(target=analyze_https_sniffed_traffic)
    t.start()

    # use python requests to make a HTTPS query to a local server
    # put SSLKEYLOGFILE into the environment so I can decode captured TLS traffic
    import os; os.environ["SSLKEYLOGFILE"] = NSS_SECRETS_LOG
    time.sleep(3)
    # unverified context: using self signed cert, make requests happy
    conn = http.client.HTTPSConnection(SERVER_HOST, SERVER_PORT, context=ssl._create_unverified_context())
    conn.request('GET', '/')
    r = conn.getresponse()
    print("response: %s" % r.read())

    # wait for the sniff thread to finish up
    t.join()

load_layer("tls")
# conf commands from https://github.com/secdev/scapy/pull/3374
conf.tls_session_enable = True
conf.tls_nss_filename = NSS_SECRETS_LOG
print("scapy version: %s" % scapy.__version__)
print("conf contents:\n%s" % conf)
send_https_request_and_analyze()

Here's what I get back for the first application data packet:

===
packet 22
Ether / IP / TCP 127.0.0.1:46066 > 127.0.0.1:8443 PA / Raw
TLS None:None > None:None / TLS / TLS Application Data
###[ TLS ]### 
  type      = application_data
  version   = TLS 1.2
  len       = 91    [deciphered_len= 91]
  iv        = b''
  \msg       \
   |###[ TLS Application Data ]### 
   |  data      = '\\xce\\xd2:\\x87\\xd0\\ h\x7f\\x81C\\xbd\x1af\\xd6y\\x91\x07wnn\\xca!ji3\\xb2\\xbbT\t\\xfc\\x80F@\\x88\x13<\\x83\\xa4\x08p\\x96\\xfb\\xf7\\x875u\\xfa\\xb9\x11\\x97M\\xf9\\xf5\\xb0\\x8fR\\x8c\ue650sI/ɇop\x1d\\xe2\n\\x91"\\x80\\x91la"d\\xe5\\xa0yk\\xc2\\xfa\\xe2A\\x8d\\x8dKB'
  mac       = b''
  pad       = b''
  padlen    = None

None

Any ideas on how I can make this work?

PS: Shout out to the devs. Scapy is amazing.

EDIT: I tried to get a write to pcap, set the NSS file in the conf, read the pcap approach, but ran into the same problem. I think I'm running into https://github.com/secdev/scapy/issues/3722.

EDIT #2: I went through the same steps decoding the raw data in https://github.com/secdev/scapy/blob/master/doc/notebooks/tls/notebook3_tls_compromised.ipynb in the section "Decrypt a PCAP file" and it worked fine. When I compared the wireshark output for the notebook's pcap file and the one I wrote with wrpcap, I see duplicate packets. It's my PCAP file that is breaking the decrypt, I think. I'm out of time today, but I will try something along the lines of Scapy: Processing partial TLS segments and report back.

Solution

I see I didn't update this with my eventual solution. In one thread I kicked off the HTTPS request and generated the NSS key files; in another thread I sniffed the traffic, then afterwards I iterated through the packetlist, updating the session with the NSS keys and mirroring the session as needed, as described in the "Decrypting Manually" section of the Scapy TLS documentation here: https://github.com/secdev/scapy/blob/master/doc/notebooks/tls/notebook3_tls_compromised.ipynb.

Example code here: https://github.com/PollyP/scapy_tls_example