I'm trying to get something from a webpage using something like this
#!/usr/bin/perl
use LWP::Simple;
use LWP::Protocol::https;
$url = 'https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=protein&id=YP_009724389.1&rettype=ft&retmode=text';
$res = getprint($url);
print $res;
This code works perfectly fine on my local machine (MacOS), but I can't get it working on the server. Python wget works from the server on this URL, and Perl works on other urls. I'm new to Perl, here are the server and local versions of the modules.
Server: perl (v5.22.1) libwww-perl is already the newest version (6.15-1)
Local: perl (v5.30.2) LWP::Simple (Not sure but it just got installed, so up-to-date)
Edit: The error message is this: 500 Can't connect to eutils.ncbi.nlm.nih.gov:443 at test.pl
Edit 2: HTTPS protocol is on the ubuntu server liblwp-protocol-https-perl is already the newest version (6.06-2)
Edit 3: verbose erro message looks like this
DEBUG: .../IO/Socket/SSL.pm:2733: free ctx 38676848 open=38676848
DEBUG: .../IO/Socket/SSL.pm:2738: free ctx 38676848 callback
DEBUG: .../IO/Socket/SSL.pm:2745: OK free ctx 38676848
DEBUG: .../IO/Socket/SSL.pm:2700: new ctx 39560272
DEBUG: .../IO/Socket/SSL.pm:612: socket not yet connected
DEBUG: .../IO/Socket/SSL.pm:614: socket connected
DEBUG: .../IO/Socket/SSL.pm:636: ssl handshake not started
DEBUG: .../IO/Socket/SSL.pm:669: using SNI with hostname eutils.ncbi.nlm.nih.gov
DEBUG: .../IO/Socket/SSL.pm:704: request OCSP stapling
DEBUG: .../IO/Socket/SSL.pm:723: set socket to non-blocking to enforce timeout=180
DEBUG: .../IO/Socket/SSL.pm:736: call Net::SSLeay::connect
DEBUG: .../IO/Socket/SSL.pm:739: done Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:749: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:759: waiting for fd to become ready: SSL wants a read first
DEBUG: .../IO/Socket/SSL.pm:779: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:736: call Net::SSLeay::connect
DEBUG: .../IO/Socket/SSL.pm:739: done Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:742: SSL connect attempt failed
DEBUG: .../IO/Socket/SSL.pm:742: local error: SSL connect attempt failed error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
DEBUG: .../IO/Socket/SSL.pm:745: fatal SSL error: SSL connect attempt failed error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
DEBUG: ...erl5/Net/HTTPS.pm:69: ignoring less severe local error 'IO::Socket::IP configuration failed', keep 'SSL connect attempt failed error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure'
DEBUG: .../IO/Socket/SSL.pm:2733: free ctx 39560272 open=39560272
DEBUG: .../IO/Socket/SSL.pm:2738: free ctx 39560272 callback
DEBUG: .../IO/Socket/SSL.pm:2745: OK free ctx 39560272
I've now also tried using the LWP::UserAgent equivalent code and not requiring a certificate.
Edit 4:
1..3
ok 1 - loaded
# openssl version compiled=0x1000205f linked=0x1000207f -- OpenSSL 1.0.2g 1 Mar 2016
# Net::SSLeay version=1.72
# parent IO::Socket::IP version=0.37
ok 2 - IO::Socket::SSL::DEBUG 1
ok 3 - Net::SSLeay::trace 1
Output from curl, which succeeds.
* Trying 130.14.29.110...
* Connected to eutils.ncbi.nlm.nih.gov (130.14.29.110) port 443 (#0)
* found 129 certificates in /etc/ssl/certs/ca-certificates.crt
* found 516 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384
* server certificate verification OK
* server certificate status verification SKIPPED
* common name: *.ncbi.nlm.nih.gov (matched)
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: RSA
* certificate version: #3
* subject: C=US,ST=Maryland,L=Bethesda,O=National Library of Medicine,CN=*.ncbi.nlm.nih.gov
* start date: Tue, 20 Jul 2021 00:00:00 GMT
* expire date: Wed, 10 Aug 2022 23:59:59 GMT
* issuer: C=US,O=DigiCert Inc,CN=DigiCert TLS RSA SHA256 2020 CA1
* compression: NULL
* ALPN, server accepted to use http/1.1
> GET /entrez/eutils/efetch.fcgi?db=protein&id=YP_009724389.1&rettype=ft&retmode=text HTTP/1.1
> Host: eutils.ncbi.nlm.nih.gov
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Fri, 17 Sep 2021 15:27:15 GMT
< Server: Finatra
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
< Content-Security-Policy: upgrade-insecure-requests
< X-Test-Test: test42
< Cache-Control: private
< NCBI-PHID: 322C09A5C8CE1A1500004CCDDDF835FE.1.1.m_5
< X-RateLimit-Remaining: 2
< NCBI-SID: 1CC49D756062E8D4_7834SID
< X-RateLimit-Limit: 3
< Access-Control-Allow-Origin: *
< Access-Control-Expose-Headers: X-RateLimit-Limit,X-RateLimit-Remaining
< Content-Type: text/plain
< Content-Disposition: attachment; filename="sequence.txt"
< Set-Cookie: ncbi_sid=1CC49D756062E8D4_7834SID; domain=.nih.gov; path=/; expires=Sat, 17 Sep 2022 15:27:15 GMT
< Vary: Accept-Encoding
< X-UA-Compatible: IE=Edge
< X-XSS-Protection: 1; mode=block
< Transfer-Encoding: chunked
... OpenSSL 1.0.2g 1 Mar 2016
Pretty old. Based on the information I would assume Ubuntu 16.04 or similar.
The server supports only a few ciphers and none of these is in the default cipher set used by IO::Socket::SSL in this old version. To work around explicitly allow a broader set of ciphers using ssl_opts and SSL_cipher_list:
#!/usr/bin/perl
use strict;
use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
$ua->ssl_opts(SSL_cipher_list => 'DEFAULT');
my $url = 'https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=protein&id=YP_009724389.1&rettype=ft&retmode=text';
my $res = $ua->get($url);
print $res->decoded_content;