I am scraping a webpage and trying to print out all the bullet points on the page, but my for loop keeps repeating it in a strange way which I can't quite figure out why. This is the exact response I got for Tuesday.
Tuesday
🎓In Class Today:
Read Chapter 4
AnnotationsÂ
Book Study
Tuesday
Read Chapter 4
Tuesday
AnnotationsÂ
Tuesday
Book Study
Tuesday
💥Due Today:
Tuesday
🔖Homework for Next Class:
Study Stems
Annotations and Book Study 1-4 due BOC Wed
Tuesday
Study Stems
Tuesday
Annotations and Book Study 1-4 due BOC Wed
Here is the section of the html(I cant give the page itself because it is hidden behind a login)
<p data-uw-styling-context="true"><img src="https://fisd.instructure.com/courses/56950/files/4791824/download" alt="tear drop line 3.png" data-api-endpoint="https://fisd.instructure.com/api/v1/courses/56950/files/4791824" data-api-returntype="File" style="max-width: 676px;" data-uw-styling-context="true"></p>
<h2 data-uw-styling-context="true">📙 Monday</h2>
<ul data-uw-styling-context="true">
<li style="list-style-type: none;" data-uw-styling-context="true">
<ul data-uw-styling-context="true">
<li data-uw-styling-context="true">No School</li>
</ul>
</li>
</ul>
<hr data-uw-styling-context="true">
<h2 data-uw-styling-context="true">📕 Tuesday</h2>
<ul data-uw-styling-context="true">
<li style="list-style-type: none;" data-uw-styling-context="true">
<ul data-uw-styling-context="true">
<li data-uw-styling-context="true">🎓In Class Today:
<ul data-uw-styling-context="true">
<li data-uw-styling-context="true">Read Chapter 4</li>
<li data-uw-styling-context="true">Annotations </li>
<li data-uw-styling-context="true">Book Study</li>
</ul>
</li>
<li data-uw-styling-context="true">💥Due Today:</li>
<li data-uw-styling-context="true">🔖Homework for Next Class:
<ul data-uw-styling-context="true">
<li data-uw-styling-context="true">Study Stems</li>
<li data-uw-styling-context="true">Annotations and Book Study 1-4 due BOC Wed</li>
</ul>
</li>
</ul>
</li>
</ul>
<hr data-uw-styling-context="true">
<h2 data-uw-styling-context="true">📘 Wednesday</h2>
<ul data-uw-styling-context="true">
<li style="list-style-type: none;" data-uw-styling-context="true">
<ul data-uw-styling-context="true">
<li data-uw-styling-context="true">🎓In Class Today:
<ul data-uw-styling-context="true">
<li data-uw-styling-context="true">Subject Complement Notes </li>
<li data-uw-styling-context="true">"There Will Come Soft Rains" </li>
</ul>
</li>
<li data-uw-styling-context="true">💥Due Today:
<ul data-uw-styling-context="true">
<li data-uw-styling-context="true">Annotations and Book Study Ch. 1-4</li>
</ul>
</li>
<li data-uw-styling-context="true">🔖Homework for Next Class:
<ul data-uw-styling-context="true">
<li data-uw-styling-context="true">Study Stems </li>
</ul>
</li>
</ul>
</li>
</ul>
<hr data-uw-styling-context="true">
<h2 data-uw-styling-context="true">📗 Thursday</h2>
<ul data-uw-styling-context="true">
<li style="list-style-type: none;" data-uw-styling-context="true">
<ul data-uw-styling-context="true">
<li data-uw-styling-context="true">🎓In Class Today:
<ul data-uw-styling-context="true">
<li data-uw-styling-context="true">Subject Complement Practice</li>
<li data-uw-styling-context="true">TWCSR</li>
</ul>
</li>
<li data-uw-styling-context="true">💥Due Today:</li>
<li data-uw-styling-context="true">🔖Homework for Next Class:
<ul data-uw-styling-context="true">
<li data-uw-styling-context="true">Study Stems </li>
</ul>
</li>
</ul>
</li>
</ul>
<hr data-uw-styling-context="true">
<h2 data-uw-styling-context="true">📓 Friday</h2>
<ul data-uw-styling-context="true">
<li style="list-style-type: none;" data-uw-styling-context="true">
<ul data-uw-styling-context="true">
<li data-uw-styling-context="true">🎓In Class Today:
<ul data-uw-styling-context="true">
<li data-uw-styling-context="true">Stems Quiz 5 Major Grade</li>
<li data-uw-styling-context="true">TWCSR (Due Monday BOC)</li>
</ul>
</li>
<li data-uw-styling-context="true">💥Due Today:</li>
<li data-uw-styling-context="true">🔖Homework for Next Class:</li>
</ul>
</li>
</ul>
Here is my for loop,
page = open("page.html", 'r')
soup = BeautifulSoup(page, "lxml")
for x in days:
for day in soup.select('h2:-soup-contains('+x+')'):
for item in day.find_next('li').find_all('li'):
print(x+'\n', item.text+'\n\n')
If anyone could help me figure this out that would be great.
I interpreted your question in the way that you do not want to print duplicate entries. The reason for them is that the outer bullet points also include the child tags. In order to avoid them, you only print the first item for each tag:
page = open("page.html", 'r')
soup = BeautifulSoup(page, "lxml")
for x in days:
for day in soup.select('h2:-soup-contains('+x+')'):
for item in day.find_next('li').find_all('li'):
bullet_point = next(item.stripped_strings)
print(bullet_point)
Output:
🎓In Class Today:
Read Chapter 4
Annotations
Book Study
💥Due Today:
🔖Homework for Next Class:
Study Stems
Annotations and Book Study 1-4 due BOC Wed