limited options chains information python

I'm trying to retrive some option chains information (first expiry date of the option chain) for more than 800 tickers (e.g. AAPL, IBM, JPM) however when I run the code below it seems to be retrieving only 119 values, I tried to capture errors but I still don't get all the values. What could be the reason of this?

import yfinance as yf
import pandas as pd
from datetime import date
from yahoo_fin import stock_info as si
import statistics
import requests,time
from bs4 import BeautifulSoup
import pandas_datareader.data as web


url = "https://finviz.com/screener.ashx?v=111&f=cap_large"
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36','accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'}
tickers = []

while True:
    r = requests.get(url, headers=headers)
    html = BeautifulSoup(r.text, "html.parser")

    for a in html.select('table[bgcolor="#d3d3d3"] a.screener-link-primary'):
        tickers.append(a.text)

    if html.select_one('.tab-link:-soup-contains("next")'):
        url = "https://finviz.com/"+html.select_one('.tab-link:-soup-contains("next")')['href']
    else:
        break
    time.sleep(1)

opts = []

try:
    for i in tickers:
        opts.append(yf.Ticker(i).options[0])
    
except:
    opts.append("Error")

Solution

I ran the scraper portion of your code and got a list of 813 ticker symbols, then just put that list in the script directly so I didn't have to scrape every time when debugging.

I also broke up the line yf.Ticker(i).options[0] into individual components so I could tell which part of it was generating exceptions. By having a bunch of separate things on one line like that, you're making debugging a lot harder.

You also were just doing except:, which explicitly throws away the error message, meaning I had no clue what it was. I also had it explicitly tell me which tickers were erroring, and to just continue to add tickers to the list if an error was encountered.

Your code would only loop until an error was reached, in which case the string "Error" would be added to the list opts, and then the program would just end, because you were not catching the error inside the loop and then continuing to the next item. Meaning it would just stop at the first error, hence why your list didn't have many items in it.

The first error was encountered after the item at index 116, which explains why your list only had that number of items in it.

Here is how my test code looks (tickers list truncated):

import yfinance as yf

#I actually have all tickers in the list,
#I just removed a big chunk from the middle for example purposes
tickers = ['A', 'AA', 'AAL', 'AAP', 'ABB', 'ABC', 'ZM', 'ZNH', 'ZS', 'ZTO', 'ZTS']

opts = []

for i in range(len(tickers)):
    ticker = tickers[i]
    try:
        ticker_obj = yf.Ticker(ticker)
    except Exception as e:
        print('cannot create yf.Ticker object', i, ticker, e)
        continue
    try:
        ticker_obj_options = ticker_obj.options
    except Exception as e:
        print('cannot get options', i, ticker, e)
        continue
    try:
        first_option = ticker_obj_options[0]
    except Exception as e:
        print('cannot get first option', i, ticker, e)
        continue
    opts.append(first_option)

print(opts)

The output of this code was:

cannot get first option 116 BTO tuple index out of range
cannot get first option 141 CEA tuple index out of range
cannot get first option 286 FERG tuple index out of range
cannot get first option 373 IHG tuple index out of range
cannot get first option 392 IX tuple index out of range
cannot get first option 397 JHX tuple index out of range
cannot get first option 525 NVR tuple index out of range
cannot get first option 600 RELX tuple index out of range
cannot get first option 637 SHG tuple index out of range
cannot get first option 676 SUZ tuple index out of range
cannot get first option 701 TLK tuple index out of range
cannot get first option 767 WBK tuple index out of range

Meaning, that for those ticker symbols, the yf.Ticker object had an empty options tuple. As for why, I don't understand finance stuff like this so that's up to you to find out why Yahoo Finance doesn't have options for those symbols. Maybe they should have items in their options, maybe they shouldn't, but I don't know.