Using SRV DNS records with the python requests library

Question

Is it possible to have the Python requests library resolve a consul domain name with a SRV record and utilize the correct IP address and port when making the request?

For example, given that I have serviceA running with the IP address 172.18.0.5 on port 8080 and this service is registered with consul. And given that DNS for the host is set to use consul to resolve queries. Can I make a request like:

requests.get('http://serviceA.service.consul')

and have it be equivalent to the request:

requests.get('http://172.18.0.5:8080')

Answer 1

I ended up writing a patch for requests that would do this using this answer. I had to make some changes due to updates to the requests library. This patch works with requests version 2.11.1.

I used the dnspython library to resolve the SRV records and it expects the IP address and port that consul is listening for DNS requests on to be available as the environment variable CONSUL_DNS_IP_PORT. To use the patch import the requests_use_srv_records function from whatever module the patch is in and then call it. It will only attempt to use consul SRV records for hosts that end with .service.consul, other hosts will be resolved regularly.

Here's the patch:

# Python Imports
import os
from socket import error as SocketError, timeout as SocketTimeout

# 3rd Party Imports
from dns import resolver
from requests.packages.urllib3.connection import HTTPConnection
from requests.packages.urllib3.exceptions import (NewConnectionError,
                                                  ConnectTimeoutError)
from requests.packages.urllib3.util import connection

def resolve_srv_record(host):

    consul_dns_ip_port = os.environ.get('CONSUL_DNS_IP_PORT', 
                                        '172.17.0.1:53')
    consul_dns_ip, consul_dns_port = consul_dns_ip_port.split(':')

    res = resolver.Resolver()
    res.port = consul_dns_port
    res.nameservers = [consul_dns_ip]

    ans = resolver.query(host, 'SRV')

    return ans.response.additional[0].items[0].address, ans[0].port

def patched_new_conn(self):

    if self.host.endswith('.service.consul'):
        hostname, port = resolve_srv_record(self.host)
    else:
        hostname = self.host
        port = self.port

    extra_kw = {}

    if self.source_address:
        extra_kw['source_address'] = self.source_address

    if self.socket_options:
        extra_kw['socket_options'] = self.socket_options

    try:
        conn = connection.create_connection((hostname, port),
                                            self.timeout,
                                            **extra_kw)

    except SocketTimeout as e:
        raise ConnectTimeoutError(
            self, "Connection to %s timed out. (connect timeout=%s)" %
            (self.host, self.timeout))

    except SocketError as e:
        raise NewConnectionError(
            self, "Failed to establish a new connection: %s" % e)

    return conn

def requests_use_srv_records():
    HTTPConnection._new_conn = patched_new_conn