I'm writing a podcast downloader using Java and Apache HttpClient. It works fine for most RSS feeds, but this one is failing with a "406 Not Acceptable" error.
The link is http://sqrpt.com/feed/podcast/
The response I get back is:
HttpResponseProxy{HTTP/1.1 406 Not Acceptable [Server: nginx/1.12.0, Date: Sat, 03 Jun 2017 03:40:17 GMT, Content-Type: text/html; charset=iso-8859-1, Content-Length: 226, Connection: keep-alive] ResponseEntityProxy{[Content-Type: text/html; charset=iso-8859-1,Content-Length: 226,Chunked: false]}}
My Java code is:
HttpClient httpClient = HttpClients.custom().setUserAgent( "Mozilla/5.0" ).build();
HttpGet httpGet = new HttpGet( url );
httpGet.setHeader( "Accept", "*/*" );
httpGet.setHeader( "Accept-Encoding", "gzip, deflate, sdch" );
httpGet.setHeader( "Accept-Language", "en-US,en;q=0.8" );
HttpResponse httpResponse = httpClient.execute( httpGet );
return httpResponse.getEntity();
I'm unsure what to do to debug this. Most posts say its the Accept header at fault. I've set mine to accept everything, yet am still failing.
When I go to the site in Chrome I get the following headers. Could it be the 304 return code that is causing the problem?
General
Request URL:http://sqrpt.com/feed/podcast/
Request Method:GET
Status Code:304 Not Modified
Remote Address:192.185.32.200:80
Referrer Policy:no-referrer-when-downgrade
Response Headers
view source
Connection:keep-alive
Date:Sat, 03 Jun 2017 03:46:27 GMT
ETag:"c06764644cd3ec282be2807a54a3484c"
Server:nginx/1.12.0
Request Headers
view source
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding:gzip, deflate, sdch
Accept-Language:en-US,en;q=0.8
Cache-Control:max-age=0
Connection:keep-alive
Host:sqrpt.com
If-Modified-Since:Fri, 02 Jun 2017 01:36:27 GMT
If-None-Match:"c06764644cd3ec282be2807a54a3484c"
Upgrade-Insecure-Requests:1
User-Agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36
What else can I try?
It seems that it checks the User-Agent. I got it to work with the following user agent:
CloseableHttpClient httpClient = HttpClients.custom().setUserAgent( "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36" ).build();