Search code examples
pythonjsonweb-scrapingbeautifulsoup

Retrieve synonyms and similarity

I would like to scrape a couple of pages from www.thesaurus.com. I am interested in both synonyms and antonyms of a word. If for example the word I am looking for is angry, I am interested in the words in the following pictures (in the page there are many more, but I am interested only in the first 2 blocks):

enter image description here

and: enter image description here I can find those words (and others related) with this code:

import requests
from bs4 import BeautifulSoup

word = "angry"
url = 'https://www.thesaurus.com/browse/{}'.format(word)
r = requests.get(url)
returned_words_list = []

soup = BeautifulSoup(r.text, 'html.parser')
word_ul = soup.find("ul", {"class":'css-1lc0dpe et6tpn80'})
for idx, elem in enumerate(word_ul.findAll("a")):
    returned_words_list.append(elem.text.strip())

print (returned_words_list)

But I am also interested in the similarity (the color of the words).

Looking at the source, there is a JSON-like file:

<script>window.INITIAL_STATE = {"routerReducer":{"location":null},"searchData":{"isFetchingTunaApi":false,"isFetchingSpellSuggestion":false,"isFetchingRelatedWordsApi":false,"searchTerm":"angry","tunaApiData":{"entry":"angry","type":"normal","slugLuna":"angry","slug":"angry","pronunciation":{"audio":{"audio\u002Fogg":"https:\u002F\u002Fstatic.sfdict.com\u002Faudio\u002Flunawav\u002FA04\u002FA0484200.ogg","audio\u002Fmpeg":"https:\u002F\u002Fstatic.sfdict.com\u002Faudio\u002FA04\u002FA0484200.mp3"},"spell":"\u003Cspan class=\"bold\"\u003Eang\u003C\u002Fspan\u003E-gree","ipa":"ˈæŋ gri"},"posTabs":[{"isInformal":null,"isVulgar":"0","definition":"being mad, often extremely mad","thesRid":"842","pos":"adj.","synonyms":[{"similarity":"100","isInformal":"0","isVulgar":null,"term":"annoyed","targetTerm":"annoyed","targetSlug":"annoyed"},{"similarity":"100","isInformal":"0","isVulgar":null,"term":"bitter","targetTerm":"bitter","targetSlug":"bitter"},{"similarity":"100","isInformal":"0","isVulgar":null,"term":"enraged","targetTerm":"enraged","targetSlug":"enraged"},.....

But I don't know how to read it. In the end I would like to have an output like this:

"synonyms":[{"similarity":"100","isInformal":"0","isVulgar":null,"term":"annoyed","targetTerm":"annoyed","targetSlug":"annoyed"},{"similarity":"100","isInformal":"0","isVulgar":null,"term":"bitter","targetTerm":"bitter","targetSlug":"bitter"},{"similarity":"100","isInformal":"0","isVulgar":null,"term":"enraged","targetTerm":"enraged","targetSlug":"enraged"},
"antonyms":[{"similarity":"-100","isInformal":"0","isVulgar":null,"term":"calm","targetTerm":"calm","targetSlug":"calm"},{"similarity":"-100","isInformal":"0","isVulgar":null,"term":"cheerful","targetTerm":"cheerful","targetSlug":"cheerful"},

where I can read term and similarity (or just an output list of tuples) like this:

[("annoyed", 100), ("bitter", 100)...]
[("calm", -100), ("cheerful", -100)...]
Solution

  • Credit Goes to Andrej Kesely

    import re
import json
import requests

url = 'https://www.thesaurus.com/browse/angry?s=t'

txt = re.findall(r'INITIAL_STATE\s*=\s*({.*})', requests.get(url).text)[0]

data = json.loads(txt)

# print(json.dumps(data, indent=4))  # <-- uncomment to see all data

print(data.keys())