I need to issue a series of parallelized web requests from the same server to a specific domain, but control what IP address these requests actually go to. Originally, I came up with a scheme where I would request the IP I wanted specifically, and then manually set the Host: www.example.com header on the request, and use a series of handlers to make sure that redirects issued followed the same pattern.
This seemed to work for some time, but lately I've been having trouble with redirects to HTTPS. The handshake will fail, and the request in turn. I have tried disabling SSL verification in a variety of ways, including:
local $ENV{ PERL_LWP_SSL_VERIFY_HOSTNAME } = 0;
local $ENV{ HTTPS_DEBUG } = 1;
$ua->ssl_opts(
SSL_ca_file => Mozilla::CA::SSL_ca_file(),
verify_hostname => 0,
SSL_verify_mode => 0x00,
);
IO::Socket::SSL::set_ctx_defaults(
SSL_verifycn_scheme => 'www',
SSL_verify_mode => 0,
);
I have also tried using LWP::UserAgent::DNS::Hosts to solve the problem, but it persists.
<edit>I should note that the reason why turning off peer validation for SSL is not solving the problem is likely because for some reason requesting this way is actually causing the handshake to fail, not failing on a validation point.</edit>
One thing that works is making an entry in /etc/hosts to point the domain at the appropriate IP, however this is not practical, because I may need to run tens, or hundreds, of tests, in parallel, on the same domain.
Is there a way to emulate the functionality of adding an entry to /etc/hosts that does not involve requesting the IP specifically and overriding the Host: ... HTTP header?
DEBUG: .../IO/Socket/SSL.pm:1914: new ctx 140288835318480
DEBUG: .../IO/Socket/SSL.pm:402: socket not yet connected
DEBUG: .../IO/Socket/SSL.pm:404: socket connected
DEBUG: .../IO/Socket/SSL.pm:422: ssl handshake not started
DEBUG: .../IO/Socket/SSL.pm:455: not using SNI because hostname is unknown
DEBUG: .../IO/Socket/SSL.pm:478: set socket to non-blocking to enforce timeout=180
DEBUG: .../IO/Socket/SSL.pm:491: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:501: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:511: waiting for fd to become ready: SSL wants a read first
DEBUG: .../IO/Socket/SSL.pm:531: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:491: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:1388: SSL connect attempt failed with unknown error
DEBUG: .../IO/Socket/SSL.pm:497: fatal SSL error: SSL connect attempt failed with unknown error error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
DEBUG: .../IO/Socket/SSL.pm:1948: free ctx 140288835318480 open=140288835318480
DEBUG: .../IO/Socket/SSL.pm:1953: free ctx 140288835318480 callback
DEBUG: .../IO/Socket/SSL.pm:1956: OK free ctx 140288835318480
And in the response I get:
Can't connect to redacted.org:443
SSL connect attempt failed with unknown error error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure at /System/Library/Perl/Extras/5.18/LWP/Protocol/http.pm line 51.
It fails equally well on our server (using an older legacy version of Perl, which I will not disclose here as it seems irrelevant).
The server initially responds to a non-HTTPS request with a 301 redirect to the HTTPS site. Then the failure occurs. I will post reproducing code with the specific details of my request removed, but any site which redirects non-HTTPS traffic to HTTPS should suffice.
use IO::Socket::SSL qw/ debug4 /;
use LWP::UserAgent;
use LWP::UserAgent::DNS::Hosts;
use HTTP::Request;
use Mozilla::CA;
use Data::Dumper;
LWP::UserAgent::DNS::Hosts->register_hosts(
'recacted.org' => '127.0.0.1', # no I am not redirecting to loopback in reality, this is anonymized
'www.redacted.org' => '127.0.0.1',
);
LWP::UserAgent::DNS::Hosts->enable_override;
my $ua = LWP::UserAgent->new;
$ua->ssl_opts( SSL_ca_file => Mozilla::CA::SSL_ca_file() );
my $request = HTTP::Request->new(GET => 'http://redacted.org/');
my $response = $ua->request($request);
print $response->content; #Dumper ( $response->is_success ? $response->headers : $response );
Again, that is not the production code, just enough code to reproduce the issue. It doesn't seem to have anything to do with SSL verification, but moreover an inability to negotiate the request, presumably because LWP::UserAgent::DNS::Hosts is doing exactly what I was doing: changing the request target to the desired IP, and then writing the Host: ... header manually. Why this causes the SSL handshake to fail, I do not know.
On my local machine debugging
openssl version -a: 1.0.2j 26 Sep 2016
IO::Socket::SSL->VERSION == 1.966
Net::SSLeay->VERSION == 1.72
On a server of ours
openssl version -a: 1.0.1t 3 May 2016
IO::Socket::SSL->VERSION == 1.76
Net::SSLeay->VERSION == 1.48
Given that it works with an explicit /etc/hosts file but not with just replacing PeerAddr or using LWP::UserAgent::DNS::Hosts this looks like a problem with the SNI extension. This TLS extension is used to provide the TLS server with the requested hostname (similar to the HTTP Host header) so that it can choose the appropriate certificate. If this SNI extension is missing some servers return a default certificate while others throw an error, like in this case.
The fix is to provide the hostname using SSL_hostname in ssl_opts. Such fix could probably also help with LWP::UserAgent::DNS::Hosts, i.e in LWP/Protocol/https/hosts.pm:
12 if (my $peer_addr = LWP::UserAgent::DNS::Hosts->_registered_peer_addr($host)) {
13 push @opts, (
14 PeerAddr => $peer_addr,
15 Host => $host,
16 SSL_verifycn_name => $host,
NEW SSL_hostname => $host, # for SNI
17 );
18 }