Search code examples
pythonnetwork-programmingweb-scrapingfirefox-developer-tools

How to read from the network tab in Python?


Using Selenium, Requests, & Beautiful Soup, I'd like to be able to locate & print this .m3u8 link (or any of the links shown here) but I don't know to show requests in Python.

enter image description here

def locator(url):
    driver = sp.driver # just geckodriver with profile
    driver.get(url)
    sleep(4)
    # from here needs to somehow access the network tab & locate GET requests with Host == "cfvod.kaltura.com"

Solution

  • Had to use ChromeDriver, PyChrome, and The DevTools Protocol, but this worked:

    def outputstart(**kwargs):
        print("START ", kwargs)
    
    driver = sp.driver # my chromedriver profile with an argument added for port 8000
    
    dev_tools = pychrome.Browser(url="http://localhost:8000")
    tab = dev_tools.list_tab()[0]
    tab.start()
    
    url = 'https://google.com'
    
    start = time.time()
    driver.get(url)
    tab.call_method("Network.emulateNetworkConditions",
                offline=False,
                latency=100,
                downloadThroughput=93750,
                uploadThroughput=31250,
                connectionType="wifi")
    
    def outputstart(**kwargs):
        print("START ", kwargs)
    
    tab.call_method("Network.enable", _timeout=20)
    tab.set_listener("Network.requestWillBeSent", outputstart)