Search code examples
pythonhtmlpython-requests

BeautifulSoup can't find the class in nasdaq table

I am trying to scrape data from nasdaq earning calendar.

URL = "https://www.nasdaq.com/market-activity/earnings"

What I want is to get all the table actually, but first, I started to try how do I do this So I tried to scrape symbols. Html that contains the symbol is like this:

<div part="table-cell" role="cell" class="table-cell
                        fixed-column-size-50px
                        text-align-left">
                    <!--?lit$8289700385$--><a href="/market-activity/stocks/aapl/earnings">AAPL</a>
                  </div>

when I copy the name of the class by clicking 2 times on it, and paste it to python code, resulting code is:

class_name = """table-cell
                        fixed-column-size-50px
                        text-align-left"""
table_cell = soup.find_all("div", {"class": class_name})

I also tried:

class_name = "table-cell fixed-column-size-50px text-align-left"

Lastly, I typed and executed $0.className in the console and it returned:

'table-cell\n                        fixed-column-size-50px\n                        text-align-left'

I also put it as class_name but it didn't work too. I am not familiar with html, so I searched for class rules but I couldn't find a solution that solves my case.

Solution

  • The data is loaded from the REST API, so beautifulsoup doesn't see it:

    import json
import requests

import pandas as pd


url = 'https://api.nasdaq.com/api/calendar/earnings?date=2025-01-30'

headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:134.0) Gecko/20100101 Firefox/134.0'} 
data = requests.get(url, headers=headers).json()

# print(json.dumps(data, indent=4))
df = pd.DataFrame(data['data']['rows'])
print(df.head(10))

    Prints:

      lastYearRptDt lastYearEPS              time symbol                          name           marketCap fiscalQuarterEnding epsForecast noOfEsts
0     2/01/2024       $2.18  time-after-hours   AAPL                    Apple Inc.  $3,601,495,987,980            Dec/2024       $2.36       11
1     1/25/2024       $2.41  time-after-hours      V                     Visa Inc.    $623,094,472,478            Dec/2024       $2.66       14
2     1/31/2024       $3.18   time-pre-market     MA       Mastercard Incorporated    $502,383,991,486            Dec/2024       $3.68       14
3     1/31/2024       $5.67   time-pre-market    TMO  Thermo Fisher Scientific Inc    $223,491,079,836            Dec/2024       $5.93       10
4     2/01/2024       $2.22   time-pre-market   SHEL                     Shell PLC    $197,454,356,091            Dec/2024       $1.78        3
5     2/05/2024       $5.23   time-pre-market    CAT             Caterpillar, Inc.    $188,432,983,041            Dec/2024       $4.97       11
6     1/25/2024       $0.84   time-pre-market  CMCSA           Comcast Corporation    $143,686,577,112            Dec/2024       $0.88        7
7     1/25/2024       $1.11   time-pre-market     BX               Blackstone Inc.    $133,996,480,907            Dec/2024       $1.46        7
8     2/01/2024       $0.89   time-pre-market    SNY                        Sanofi    $133,992,908,715            Dec/2024       $0.71        5
9     1/30/2024       $2.47   time-pre-market    UPS   United Parcel Service, Inc.    $115,563,305,544            Dec/2024       $2.52        9