I don't know why this isn't working
from bs4 import *
import time
import pandas as pd
import pickle
import html5lib
from requests_html import HTMLSession
s = HTMLSession()
url = "https://cryptoli.st/lists/fixed-supply"
def get_data(url):
r = s.get(url)
soup = BeautifulSoup(r.text, 'html.parser')
return soup
def get_next_page(soup):
page = soup.find('ul', {'class': 'pager'})
if not page.find('a', {'class': 'btn btn-default current disabled'}):
url = 'https://cryptoli.st/lists/fixed-supply' + \
str(page.find('li', {'class': 'paginate_button'}).find(
'a')[{'class': 'btn btn-default next'}])
return url
else:
return
get_data(url)
print(get_next_page(soup))
I have seen other scripts that return variables from one function to use in another but this keeps saying "soup" is not defined. Then if I make soup a global variable then I get the error that page is a Nonetype and I can't call the .find attribute off it. Any help would be appreciated.
Your last line for print(get_next_page(data))
is running the function get_next_page
with the parameter data
passed in. However, data
is never defined, and so it passes in None. So then inside of get_next_page
, it assigns soup = None
. Then you are running everything else on None
.
In the second-to-bottom line you need to do data = get_data(url)
, and then when you call get_next_page(data))
, data will be equal to the soup
that you returned from the first function.
Also, you probably need that s = HTMLSession()
to either be inside of the get_url
function, or pass it in like you do url