Search code examples
javascriptweb-scrapingluascrapyscrapy-splash

splash:mouse_click couldn't open pop up window

I'm trying to scrape part-time job information from townwork(A Japanese job information site). To get the address, I want to open a popup window. Then I started to use Splash. Although I could open other links and popups using splash:mouse_click, I couldn't open the desired pop-up window.

Any answer would be greatly appreciated.

The link that I wanted to click. enter image description here

Desired result. enter image description here

Target page URL https://townwork.net/detail/clc_2578209001/joid_Y00BKDF4/

Code

function main(splash, args)
  assert(splash:go(args.url))
  local map = splash:select('#jsi-inflow-target > div > dl:nth-child(3) > dd > p > span > a')

  local bounds = map:bounds()
  local x_dif = bounds.right - bounds.left
  
  assert(map:mouse_click({x=bounds.width/3, y=bounds.height/3}))
  splash:wait(1)
  return {
    html = splash:html(),
    png = splash:png(),
    bounds = bounds
    
  }
end
Solution

  • I think that the speed at which this popup opens is the matter.

    I found another solution other than opening a popup page to get the address: getting latitude and longitude from the first page.

    def parse(self, response):
  dt_list = response.css('dt')
    for dt in dt_list:
      if dt.css('::text').get == "勤務地":
        a_tag = BeautifulSoup(dt.xpath('./following-sibling::dd').get()).find('a')
        if a_tag is None:
          continue
        item['lat'] = a_tag['data-lat']
        item['lon'] = a_tag['data-lon']