Perl HTTP request skips proxy server and connects to target directly

Question

I'm trying to write a perl script to pull data from a web address through a Squid proxy server. I have Squid server running on CentOS7, listening on port 3128 and authenticating with /usr/lib64/squid/basic_ncsa_auth. However, I keep getting perl seems to bypass the proxy entirely and connects to the target directly.

I was able to do this through curl but not with perl.

I have rewritten my perl script several ways. I've updated perl versions (currently on 5.28).

use LWP::UserAgent;

$ENV{'HTTP_PROXY'} = 'http://localhost:3128/';
$ENV{'HTTP_PROXY_USERNAME'} = 'user1';
$ENV{'HTTP_PROXY_PASSWORD'} = 'password1';

$ua = LWP::UserAgent->new();

$ua->proxy('http', 'http://localhost:3128/');

my $req = HTTP::Request->new('GET', 'https://stackoverflow.com');

$req->proxy_authorization_basic("user1", "password1");
my $res = $ua->request($req);

print $res->code ."\n";

So, the above code produces a '200' however, I see no entry in my squid access logs. When I attempt the maneuver in curl

curl --proxy-user user1:password1 -x http://localhost:3128 https://stackoverflow.com)

I get

1560893906.291 513 ::1 TCP_TUNNEL/200 270148 CONNECT stackoverflow.com:443 user1 HIER_DIRECT/151.101.65.69 -

in the access log.

Obviously, there's some step that I'm missing, but I've copied examples from the perl documentation and was unable to get it to work.

Answer 1

$ua->proxy('http', 'http://localhost:3128/');
my $req = HTTP::Request->new('GET', 'https://stackoverflow.com');

Given that you access a https:// URL you need to set the proxy for the protocol https, not http as you did.

$ENV{'HTTP_PROXY'} = 'http://localhost:3128/';

This setting is only used if you explicitly enable proxy setting by environment, i.e. $ua->env_proxy. Moreover the setting should be for HTTPS_PROXY not HTTP_PROXY here too.