chromedp click is not working in my golang code. can you find what's wrong?

Question

I'm working on scrapper with chromedp.

To get what i want (page html), i have to click a specific button.

So I used chromedp.click, and chromedp.outerhtml, but i only got html of page before click, not the html of page after click have done.

Can you see my code and advice me how to fix it?

func runCrawler(URL string, lineNum string, stationNm string) {
    
        // settings for crawling
    opts := append(chromedp.DefaultExecAllocatorOptions[:],
        chromedp.Flag("headless", false))
    
        // create chrome instance
    contextVar, cancelFunc := chromedp.NewExecAllocator(context.Background(), opts...)
    defer cancelFunc()

    contextVar, cancelFunc = chromedp.NewContext(contextVar)
    defer cancelFunc()


    var htmlContent string

    err := chromedp.Run(contextVar,
        chromedp.Navigate(URL),
        chromedp.WaitVisible(".end_footer_area"),
        chromedp.Click(".end_section.station_info_section > div.at_end.sofzqce > div > div.c10jv2ep.wrap_btn_schedule.schedule_time > button"),
        chromedp.OuterHTML("html", &htmlContent, chromedp.ByQuery),
    )
    fmt.Println("html", htmlContent)
    checkErr(err)

i also give you homepage and button i need to click

Page URL: https://pts.map.naver.com/end-subway/ends/web/11321/home

Button Area I need to click:

Thank you very much

Answer 1

The page you want to get is open in a new tab (target).

In this case, we can use chromedp.WaitNewTarget to create a chan from where we can receive the target id of the new tab. Then create a new context with the chromedp.WithTargetID option so that we can connect to the new tab. From here everything is what you are already familiar with.

package main

import (
    "context"
    "fmt"
    "strings"

    "github.com/chromedp/cdproto/target"
    "github.com/chromedp/chromedp"
)

func main() {
    opts := append(chromedp.DefaultExecAllocatorOptions[:],
        chromedp.Flag("headless", false),
    )

    ctx, cancel := chromedp.NewExecAllocator(context.Background(), opts...)
    defer cancel()

    ctx, cancel = chromedp.NewContext(ctx)
    defer cancel()

    var htmlContent string

    ch := chromedp.WaitNewTarget(ctx, func(i *target.Info) bool {
        return strings.Contains(i.URL, "/timetable/web/")
    })

    err := chromedp.Run(ctx,
        chromedp.Navigate("https://pts.map.naver.com/end-subway/ends/web/11321/home"),
        chromedp.WaitVisible(".end_footer_area"),
        chromedp.Click(".end_section.station_info_section > div.at_end.sofzqce > div > div.c10jv2ep.wrap_btn_schedule.schedule_time > button"),
    )
    if err != nil {
        panic(err)
    }

    newCtx, cancel := chromedp.NewContext(ctx, chromedp.WithTargetID(<-ch))
    defer cancel()

    if err := chromedp.Run(newCtx,
        chromedp.WaitReady(".table_schedule", chromedp.ByQuery),
        chromedp.OuterHTML("html", &htmlContent, chromedp.ByQuery),
    ); err != nil {
        panic(err)
    }
    fmt.Println("html", htmlContent)
}