Search code examples
pythonhtmlweb-scrapingbeautifulsoup

web scraping of webpage on chartink.com

Please help me to scrape this link. link - https://chartink.com/screener/time-pass-48 I am trying to web scrape but it is not showing the table which I want. please help me the same.

I have tried this code, but it is not giving me the desired result.

import requests
from bs4 import BeautifulSoup

URL = 'https://chartink.com/screener/time-pass-48'
page = requests.get(URL)
print(page)

soup = BeautifulSoup(page.content, 'html.parser')
print(soup)
Solution

  • Data indeed comes from a POST request. You don't need to allow JavaScript to run. You simply need to pick up one cookie (ci_session - which can be done using Session object to hold cookies from initial landing page request to pass on with subsequent POST), and one token (X-CSRF-TOKEN - which can be pulled from a meta tag in the initial request response):

    import requests
from bs4 import BeautifulSoup as bs
import pandas as pd

data = {
  'scan_clause': '( {cash} ( monthly rsi( 14 ) > 60 and weekly rsi( 14 ) > 60 and latest rsi( 14 ) > 60 and 1 day ago  rsi( 14 ) <= 60 and latest volume > 100000 ) ) '
}

with requests.Session() as s:
    r = s.get('https://chartink.com/screener/time-pass-48')
    soup = bs(r.content, 'lxml')
    s.headers['X-CSRF-TOKEN'] = soup.select_one('[name=csrf-token]')['content']
    r = s.post('https://chartink.com/screener/process', data=data).json()
    #print(r.json())
    df = pd.DataFrame(r['data'])
    print(df)