Search code examples
pythondeep-learningartificial-intelligence

Http error when calling the duckduckgo api


I am currently following a course on machine learning : the course from the fast.ai web site. The course is a video, but it is linked to a a jupyter notebook on kaggle (https://www.kaggle.com/code/jhoward/is-it-a-bird-creating-a-model-from-your-own-data). However when i try to run cell number 11: the one that contains this code:

#NB: `search_images` depends on duckduckgo.com, which doesn't always return correct responses.
#    If you get a JSON error, just try running it again (it may take a couple of tries).
urls = search_images('bird photos', max_images=1)
urls[0]

I get this error :

HTTPError                                 Traceback (most recent call last)
/tmp/ipykernel_17/2432147335.py in <module>
      1 #NB: `search_images` depends on duckduckgo.com, which doesn't always return correct responses.
      2 #    If you get a JSON error, just try running it again (it may take a couple of tries).
----> 3 urls = search_images('bird photos', max_images=1)
      4 urls[0]

/tmp/ipykernel_17/3153793264.py in search_images(term, max_images)
      4 def search_images(term, max_images=30):
      5     print(f"Searching for '{term}'")
----> 6     return ddg_images(term, max_results=max_images).itemgot('image')

/opt/conda/lib/python3.7/site-packages/duckduckgo_search/compat.py in ddg_images(keywords, region, safesearch, time, size, color, type_image, layout, license_image, max_results, page, output, download)
     80         type_image=type_image,
     81         layout=layout,
---> 82         license_image=license_image,
     83     ):
     84         results.append(r)

/opt/conda/lib/python3.7/site-packages/duckduckgo_search/duckduckgo_search.py in images(self, keywords, region, safesearch, timelimit, size, color, type_image, layout, license_image)
    403         assert keywords, "keywords is mandatory"
    404 
--> 405         vqd = self._get_vqd(keywords)
    406         assert vqd, "error in getting vqd"
    407 

/opt/conda/lib/python3.7/site-packages/duckduckgo_search/duckduckgo_search.py in _get_vqd(self, keywords)
     93     def _get_vqd(self, keywords: str) -> Optional[str]:
     94         """Get vqd value for a search query."""
---> 95         resp = self._get_url("POST", "https://duckduckgo.com", data={"q": keywords})
     96         if resp:
     97             for c1, c2 in (

/opt/conda/lib/python3.7/site-packages/duckduckgo_search/duckduckgo_search.py in _get_url(self, method, url, **kwargs)
     87                 logger.warning(f"_get_url() {url} {type(ex).__name__} {ex}")
     88                 if i >= 2 or "418" in str(ex):
---> 89                     raise ex
     90             sleep(3)
     91         return None

/opt/conda/lib/python3.7/site-packages/duckduckgo_search/duckduckgo_search.py in _get_url(self, method, url, **kwargs)
     80                 )
     81                 if self._is_500_in_url(str(resp.url)) or resp.status_code == 202:
---> 82                     raise httpx._exceptions.HTTPError("")
     83                 resp.raise_for_status()
     84                 if resp.status_code == 200:

HTTPError: 

I tried running all the cells above, successfully : my Kaggle account is verified and I have enabled internet. I suspect that the duckduckgo.com website does not work (because the error thrown is inside an if statement , and that statement is verified if the Http response code contains a 500) : is there a way for me to check if this is the case only for me or for everyone else, if so : does anyone of have found a way to bypass this problem and still finish lesson one of the course?


Solution

    1. On the right side of the webpage, under "Notebook options", find a drop down menu called "ENVIRONMENT".
    2. Change this to "Always use latest environment".
    3. Rerun ALL the cells (including the cell with the import statement) and see if the error persists.