Search code examples
vbaexcelweb-scrapingtimeoutinternet-explorer-11

Unable to set timeout option within my scraper to save it from infinite looping


I've written a script in vba using IE to initiate a search in a webpage in it's searchbox to populate the result according to the search by hitting the search button. The webpage loads it's searchbox few seconds later it is made to open. However, my below script can handle this barrier and perform the search in the right way.

Now, I've got a slightly different question which is: suppose I've defined a wrong ID name or There is no such ID in that webpage then the way I've written my script will loop forever. This is certainly a serious issue.

How can I modify my script putting some timeout option within the loop so that the scraper will wait sometime for the element to appear and if the element is not present within that time then it will break out of the loop and quit the browser. Thanks in advance.

Here is my script:

Sub GetItems()
    Dim HTML As HTMLDocument, post As Object

    With New InternetExplorer
        .Visible = True
        .navigate "https://torrentz2.eu/"
        While .Busy Or .ReadyState < 4: DoEvents: Wend
        Set HTML = .Document

        With HTML
            Do     ''Wish to set timeout options within this loop to save it from infinite looping
                Set post = .getElementById("thesearchbox")
                DoEvents
            Loop While post Is Nothing

            post.innerText = "Udemy"

            .getElementById("thesearchbutton").Click
        End With
        .Quit
    End With
End Sub

Solution

  • Try replacing:

        With HTML
            Do     ''Wish to set timeout options within this loop to save it from infinite looping
                Set post = .getElementById("thesearchbox")
                DoEvents
            Loop While post Is Nothing
    
            post.innerText = "Udemy"
    
            .getElementById("thesearchbutton").Click
        End With
    

    with

    Dim dTime as Single ' as pointed out by @Mathieu Guindon in comments
    '____________________________
    
    dTime = Timer
    
    With HTML
        Do     ''Wish to set timeout options within this loop to save it from infinite looping
            Set post = .getElementById("thesearchbox")
            If Timer - dTime > 60 Then Exit Do ' 60: num of seconds, equal 1 min
            DoEvents
        Loop While post Is Nothing
    
        post.innerText = "Udemy"
    
        .getElementById("thesearchbutton").Click
    End With