Search code examples
javascriptdebuggingweb-scrapingfiddler

How to see what processes are occurring when a button is clicked on a webpage?

I am currently trying to scrape some data from the trading dashboard on Plus500. The information I am interested in is only visible (on the right-hand side) when the 'Information' button is clicked next to a particular security, as shown in the screenshot below:

info

I have so far written some basic JavaScript which clicks on the information button next to each security on the page and attempts to scrape the data in the info pane. However, when the 'Information' button is clicked it takes a few seconds for the new data to load:

loading

As a consequence, my Javascript is scraping from the info panel before it has loaded and the information it is collecting is incorrect. This means that I am now trying to work out how to tell my Javascript to wait until the new data has loaded.

I suspected (perhaps incorrectly) that an Ajax request was being used to fetch this new information. However, looking at Fiddler, no such processes are being captured when I click the info buttons.

I also wondered whether the information was all already there on the page, but hidden until the info button was clicked. This would explain why I cannot see an Ajax request being made in Fiddler. However, a brief inspection of the HTML seemed to suggest that this was not the case.

To avoid the guess-work, how can I definitively find out what process(es) are being run when I click these 'information' buttons?

NB: If anyone thinks there is a better way to be scraping this data then please let me know!

Cheers.

Solution

  • Use breakpoints in the JS debugger of your browser:

    1. Open the dev tools of your browser (usually F12)
    2. Go to the tab "Debugger" (Firefox) or "Sources" (Chrome/WebKit)
    3. Find the panel for breakpoints (rightmost panel or bottom left panel depending on your dev tools dimensions)
    4. Navigate to Event Listener Breakpoints -> Mouse and there you can set a breakpoint for all click events

    Now whenever you make a click that has a JS event listener, the JS debugger will halt in the JS code of the listener and you can debug it. Since most websites minify their JS code, you might have to fight through an unreadable mess, but at least it's a start.

    You can also see all event listeners for a node in the DOM inspector:

    • Firefox: The DOM inspector shows a small bubble "event" next to each node with event listeners. Click on it to list all listeners bound to this node. Note that events might bubble up and be handled by a higher node, so there might be an event listener not shown at the node directly still handling events for the node.
    • Chrome: Select the node, then in the right panel go to the tab "Event Listeners" and select the event type you are interested in.