Search code examples
vbaweb-scrapingxmlhttprequest

Post request can't fetch response from targeted page


Writing a macro in post request when i run it, it brings unexpected response which i don't want. Perhaps it is unable to fetch response from the targeted page. Can't identify the mistake I'm doing? The original url I'm pasting under my code.

Box to be checked before performing search:

Industry Role = Professional Services Providers

Other Criterion = APEX

Sub Xmlpost()
Dim http As New MSXML2.XMLHTTP60
Dim html As New HTMLDocument
Dim Items As Object, Item As Object, Elem As Object
Dim postdata As String

postdata = "DoMemberSearch=1&mas_last=&mas_comp=&mas_city=&mas_stat=&mas_cntr=&mas_type=Professional+Services+Providers&OtherCriteria=1"
With http
    .Open "POST", "https://www.infocomm.org/cps/rde/xchg/infocomm/hs.xsl/memberdirectory.htm", False
    .setRequestHeader "Content-Type", "application/x-www-form-urlencoded"
    .setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"
    .send postdata
    html.body.innerHTML = .responseText
End With

Set Items = html.getElementById("paginationDataPool").getElementsByTagName("a")
For Each Item In Items
    x = x + 1
    Cells(x, 1) = Item.innerText
Next Item
End Sub

Original Page:"https://www.infocomm.org/cps/rde/xchg/infocomm/hs.xsl/memberdirectory.htm"

Search should be made like:

enter image description here

The output I'm getting is this:

enter image description here


Solution

  • You are looking for elements that use the class paginationDisplayItem, but this class is only added dynamically by JavaScript running in your browser, looking like this:

    <div class="paginationDisplayItem">
    

    In your html object, however, there is just the plain HTML response from your POST request. Just save it to a file and have a look for yourself, instead of the class attribute the same div contains an id attribute:

    <div id="paginationItem_1">
    

    Each successive entry has that trailing number increased by one.

    If you adapt your loop to retrieve elements based on that id, everything will work as you expect.

    Proof of concept:

    For x = 1 To 57
        Set Item = html.getElementById("paginationItem_" & x)
        Cells(x, 1) = Item.getElementsByTagName("a")(0).innerText
    Next x
    

    You will obviously not want to explicitly loop to 57 in all cases, so feel free to refactor this to your likings.

    Btw.: You should declare Items As IHTMLElementCollection and Item As IHTMLElement - this way IntelliSense will work on your objects and you'll have type safety.