Search code examples
pythonproxybingbing-apihttp.client

How to apply proxy with authentication using http.client for API


I'm attempting to use Bing's Entity search API, however I need to go through a proxy in order to establish a connection.

My company has giving me a proxy in the following format to use: 'http://username:[email protected]:8080'

I've tried using HTTPConnection.set_tunnel(host, port=None, headers=None) with no avail. Does anyone have any idea how to run the following Python query below using the proxy I was provided?

import http.client, urllib.parse
import json


proxy = 'http://username:[email protected]:8080' 
subscriptionKey = 'MY_KEY_HERE'
host = 'api.cognitive.microsoft.com'
path = '/bing/v7.0/entities'
mkt = 'en-US'
query = 'italian restaurants near me'

params = '?mkt=' + mkt + '&q=' + urllib.parse.quote (query)

def get_suggestions ():
    headers = {'Ocp-Apim-Subscription-Key': subscriptionKey}
    conn = http.client.HTTPSConnection(host)
    conn.request("GET", path + params, None, headers)
    response = conn.getresponse()
    return response.read()

result = get_suggestions ()
print (json.dumps(json.loads(result), indent=4))

As a side note, I was able to run the following sucessfully with the proxy.

nltk.set_proxy('http://username:[email protected]:8080')
nltk.download('wordnet')

Solution

  • I ended up using the requests package, they make it very simple to channel your request through a proxy, sample code below for reference:

    import requests
    
    proxies = {
        'http':  f"http://username:[email protected]:8080",
        'https': f"https://username:[email protected]:8080"
    }
    url = 'https://api.cognitive.microsoft.com/bing/v7.0/entities/'
    # query string parameters
    
    params = 'mkt=' + 'en-US' + '&q=' + urllib.parse.quote (query)
    
    # custom headers
    headers = {'Ocp-Apim-Subscription-Key': subscriptionKey}
    
    #start session
    session = requests.Session()
    #persist proxy across request
    session.proxies = proxies
    
    # make GET request
    r = session.get(url, params=params, headers=headers)