Search code examples
jsonbeautifulsouppython-requestsalgolia

Extract data from an unreachable JsonObject()


I'm trying to reach a JsObject to scrape it.

import requests
from bs4 import BeautifulSoup

url ='https://fjrgcwbcbo-dsn.algolia.net/1/indexes/*/queries?x-algolia-agent=Algolia%20for%20vanilla%20JavaScript%20(lite)%203.27.0%3Binstantsearch.js%202.8.0%3BJS%20Helper%202.26.0&x-algolia-application-id=FJRGCWBCBO&x-algolia-api-key=a214a1e7afd822b517723830f05e9449'
jsonObj = requests.get(url).json()

print(JsonObj)

There are four JsonObject from this URL, but each one seems to be blocked by an API. Website URL

Thanks a lot.


Solution

  • I find another solution to scrape all of the email adresses in one shot.

    import requests
    import json
    from algoliasearch import algoliasearch
    
    url ='https://fjrgcwbcbo- 
    dsn.algolia.net/1/indexes/*/queries'
    
    client = algoliasearch.Client("FJRGCWBCBO", "a214a1e7afd822b517723830f05e9449")
    index = client.init_index('CAPI-agents-stage')
    
    for hit in index.browse_all({"query": ""}):
        print(hit['email_address'])
    

    If it could help anyone, thanks a lot Chitown88