I'm doing:
HTTPoison.get! "https://www.nature.com/articles/d41586-020-00741-x"
Which works fine in the browser but gives me a 303 when I try and fetch the page with HTTPoison.
What am I doing wrong?
The 303 status code is a redirection code. In order to follow the redirection, you need to pass the option to HTTPoison to follow it:
HTTPoison.get!("https://www.nature.com/articles/d41586-020-00741-x", [], follow_redirect: true)
However, you most likely will need to pass a :force_redirect option to Hackney as well, since AFAIK it doesn't accept 303 as a valid redirect status code.
So, a working version would be:
HTTPoison.get!("https://www.nature.com/articles/d41586-020-00741-x",
[],
follow_redirect: true,
hackney: [{:force_redirect, true}])
Which will produce:
%HTTPoison.Response{
body: "\n\n\n\n\n\n\n\n<!DOCTYPE html>\n<html lang=\"en\" class=\"grade-c\">\n<head>\n <meta charset=\"utf-8\">\n<link rel=\"dns-prefetch\" href=\"//ajax.googleapis.com\"/>\n<link rel=\"dns-prefetch\" href=\"//fonts.googleapis.com\"/>\n<link rel=\"dns-prefetch\" href=\"//fonts.gstatic.com\"/>\n<meta http-equiv=\"X-UA-Compatible\" content=\"IE=edge\">\n<meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0, shrink-to-fit=no\">\n\n <title>What China’s coronavirus response can teach the rest of the world</title>\n <meta name=\"description\" content=\"Researchers are studying the effects of China's lockdowns to glean insights about controlling the viral pandemic.\"/>\n <meta property=\"og:url\" content=\"https://www.nature.com/articles/d41586-020-00741-x\"/>\n <meta property=\"og:type\" content=\"article\"/>\n <meta property=\"og:title\" content=\"What China’s coronavirus response can teach the rest of the world\"/>\n <meta property=\"og:description\" content=\"Researchers are studying the effects of China's lockdowns to glean insights about controlling the viral pandemic.\"/>\n <meta property=\"og:image\"\n content=\"https://media.nature.com/lw1024/magazine-assets/d41586-020-00741-x/d41586-020-00741-x_17788864.jpg\"/>\n <meta name=\"twitter:card\" content=\"summary_large_image\"/>\n <meta name=\"twitter:site\" content=\"@nature\"/>\n <meta name=\"twitter:title\" content=\"What China’s coronavirus response can teach the rest of the world\"/>\n <meta name=\"twitter:description\" content=\"Researchers are studying the effects of China's lockdowns to glean insights about controlling the viral pandemic.\"/>\n <meta name=\"twitter:image\"\n content=\"https://media.nature.com/lw1024/magazine-assets/d41586-020-00741-x/d41586-020-00741-x_17788864.jpg\"/>\n \n\n <meta name=\"journal_id\" content=\"41586\"/>\n\n <meta name=\"dc.title\" content=\"What China’s coronavirus response can teach the rest of the world\"/>\n\n <meta name=\"dc.source\" content=\"Nature 2020\"/>\n\n <meta name=\"dc.format\" content=\"text/html\"/>\n\n <meta name=\"dc.publisher\" content=\"Nature Publishing Group\"/>\n\n <meta name=\"dc.date\" content=\"2020-03-17\"/>\n\n <meta name=\"dc.type\" content=\"News Explainer\"/>\n\n <meta name=\"dc.language\" content=\"En\"/>\n\n <meta name=\"dc.copyright\" content=\"2020 Nature\"/>\n\n <meta name=\"dc.rightsAgent\" content=\"[email protected]\"/>\n\n <meta name=\"dc.description\" content=\"Researchers are studying the effects of China's lockdowns to glean insights about controlling the viral pandemic. Researchers are studying the effects of China's lockdowns to glean insights about controlling the viral pandemic.\"/>\n\n <meta name=\"prism.publicationName\" content=\"Nature\"/>\n\n <meta name=\"prism.publicationDate\" content=\"2020-03-17\"/>\n\n <meta name=\"prism.section\" content=\"News\"/>\n\n <meta name=\"prism.startingPage\" content=\"\"/>\n\n <meta name=\"prism.endingPage\" content=\"\"/>\n\n <meta name=\"prism.copyright\" content=\"2020 Nature\"/>\n\n <meta name=\"prism.rightsAgent\" content=\"[email protected]\"/>\n\n <meta name=\"prism.url\" content=\"https://www.nature.com/articles/d41586-020-00741-x\"/>\n\n <meta name=\"prism.doi\" content=\"doi:10.1038/d41586-020-00741-x\"/>\n\n <meta name=\"dc.identifier\" content=\"doi:10.1038/d41586-020-00741-x\"/>\n\n <meta name=\"DOI\" content=\"10.1038/d41586-020-00741-x\"/>\n\n <meta name=\"description\" content=\"Researchers are studying the effects of China's lockdowns to glean insights about controlling the viral pandemic. Researchers are studying the effects of China's lockdowns to glean insights about controlling the viral pandemic.\"/>\n\n <meta name=\"dc.creator\" content=\"David Cyranoski\"/>\n\n <meta name=\"dc.subject\" content=\"Government\"/>\n\n <meta name=\"dc.subject\" content=\"Infection\"/>\n\n <meta name=\"dc.subject\" content=\"Public health\"/>\n\n\n\n<script>(function(e){var t=e.documentElement,n=e.implementation;t.className='js';if(n&&n.hasFeature('http://www.w3.org/TR/SVG11/feature#Image','1.1')){t.className+=' svg'}})(document)</script>\n<link rel=\"stylesheet\" href=\"/static/css/m" <> ...,
headers: [
...
],
request: %HTTPoison.Request{
body: "",
headers: [],
method: :get,
options: [follow_redirect: true, hackney: [force_redirect: true]],
params: %{},
url: "https://www.nature.com/articles/d41586-020-00741-x"
},
request_url: "https://www.nature.com/articles/d41586-020-00741-x",
status_code: 200
}