Search code examples
javascriptseleniumdomscreen-scrapingpuppeteer

document.querySelector() returns null when DOM elements are already visible

I am trying to build an automated Puppeteer script to download my monthly bank transactions from my bank website.

However, I am encountering a strange error (see attached Imgur for pictures of this behavior)

https://i.sstatic.net/suBnQ.jpg

Problem: querySelector returns null on DOM element that is clearly visible:

Screenshot: https://i.sstatic.net/XV2Ka.jpg

(1) Input box for username is clearly visible on site (https://internet.ocbc.com/internet-banking/), (2) However, when I run document.querySelector('#access-code'), console returns null.

I'm wondering why this behavior is so, and what are the circumstances that a browser would return null on a querySelector(#id) query when the DOM node is clearly visible.

# EDIT: Weird workaround that works:

I was continuing to play around with the browser, and used DevTools to inspect the DOM element and use it to Copy the JS Path.

Weirdly, after using Chrome Devtools to copy the JS Path, document.querySelector('#access-code') returned the correct element.

Screenshot of it returning the correct element: https://i.sstatic.net/suBnQ.jpg

In both cases, the exact same search string is used for document.querySelector.

Solution

  • I believe that you cannot get proper value using document.querySelector('#access-code') because a website use frameset.

    In the website there is frame with src to load content

    <frame src="/internet-banking/Login/Login">

    DOMContentLoading is executed when main document is loaded and not wait for frame content to be loaded.

    First of all you need to have listener for load event.

    window.addEventListener("load",function() {
   ...
 });

    And later on you cannot simply use document.querySelector('#access-code') because input yuo want to get is inside frame. You will need to find a way to access frame content and than inside of it use simple querySelector.

    So something like:

    window.addEventListener("load",function() {
   console.log(window.frames[0].document.querySelector('#access-code'));
 });

    BTW please see in: view-source:https://internet.ocbc.com/internet-banking/ looks like website is mostly rendered client-side.