I'm going a bit crazy here and feel I must be missing something obvious.
I am trying to scrape data from a website using python and urllib3. My code looks like
>>> import urllib3
>>> from bs4 import BeautifulSoup
>>>
>>> http = urllib3.PoolManager()
>>> url = 'https://www.google.com/'
>>> r = http.request('GET', url)
>>> data = BeautifulSoup(r.data)
If I open up my terminal (I am on a mac), activate my conda virtual environment, then pull up the python interpreter this code works exactly as expected, pulling down the html request and parsing the data.
When I put the code into my Python IDE and set it to exactly the same virtual environment, the HTTP get request fails and I receive the following error
{MaxRetryError}HTTPSConnectionPool(host='www.nts.live', port=443): Max retries exceeded with url: /api/v2/shows/powell/episodes?offset=0&limit=12 (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))
I have done similar requests in the past that have worked within Pycharm, so this is definitely user error at some stage. I am unsure what I should check other than that my virtual environment is the same? I have tried multiple different links and the issue persists. Any idea on why this error might be coming in would hugely appreciated!
This error comes not from the IDE but using the code within a module with other methods of retrieving http requests. The above lived in my data_pull.py file, however I had another file spotify_request.py that was also imported. Instead of urllib3, this file used import requests then requests.get(url).
When I ran my code standalone in the terminal, this second import was never carried out and the code ran smoothly. To fix I replaced all urllib3 get requests with the requests library (https://pypi.org/project/requests/). I also tested by making all requests utilise urllib3 and the code ran fine too. I'm not sure WHY using both doesn't work but this has fixed my error.
TLDR; I was unable to use BOTH urllib3 and requests for http calls in my package - stick to one!