Search code examples
powershellboostzip

Powershell webrequest not downloading file correctly

With recent migration of boost, some of our scripts broke. One of them was responsible for downloading the zip file. Here is the link for the file itself

https://boostorg.jfrog.io/ui/native/main/release/1.69.0/source

File I am interested in is, "boost_1_69_0.zip". The one with size 156.8Mb.

I use the following command to download the file

Invoke-WebRequest -Uri "https://boostorg.jfrog.io/ui/native/main/release/1.69.0/source/1.69.0.zip" -OutFile "C:\pwrShell\1.69.0.zip"

However, the downloaded file is only 15kb, and not a valid zip. Wondering what I am missing?

Solution

  • The site is a bit tricksy - the boost_1_69_0.zip link on https://boostorg.jfrog.io/ui/native/main/release/1.69.0/source might look like it links to a zip file but when you click it you actually run some javascript that takes the browser to a web page - /ui/api/v1/download?repoKey=main&path=release%252F1.69.0%252Fsource%252Fboost_1_69_0.zip.

    This page gives a "302 Found" response that takes the browser to the actual download link at https://jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com/aol-boostorg/filestore/07/074cb678b05fd327be1096b39a962aeb249372ac?x-jf-traceId=2593752ec87a6551&response-content-disposition=attachment%3Bfilename%3D%22boost_1_69_0.zip%22&response-content-type=application%2Fzip&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20211019T222020Z&X-Amz-SignedHeaders=host&X-Amz-Expires=60&X-Amz-Credential=AKIASG3IHPL63WBBRCUD%2F20211019%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=8e8b21c8a319e8cdc96044b8794dd3a359e265753bb7cf4913999fa753ffb47d

    You can see all of this happen if you run a dev proxy like Fiddler in the background while you interact with the browser:

    enter image description here

    • Request #3 is my browser loading the index page (including 200+ javascript bundle files).
    • Request #252 is when I clicked the boost_1_69_0.zip link, which triggered a HEAD request and a GET request to /ui/api/v1/download?repoKey=main&path=release%252F1.69.0%252Fsource%252Fboost_1_69_0.zip
    • Request #253 is the GET request which receives a 302 Found response redirecting the browser to the actual download link
    • Request #256 is the download from the s3.amazon.aws.com site initiated by the browser

    So, if you want to replicate what the browser is doing, your best bet is to do something like this:

    Invoke-WebRequest -Uri "https://boostorg.jfrog.io/ui/api/v1/download?repoKey=main&path=release%252F1.69.0%252Fsource%252Fboost_1_69_0.zip" -OutFile "1.69.0.zip";

    Invoke-WebRequst will automatically follow the 302 response returned by https://boostorg.jfrog.io/ui/api/v1/download?repoKey=main&path=release%252F1.69.0%252Fsource%252Fboost_1_69_0.zip and will download the file from the url specified in the Location header in the response.

    PS

    Your 15kb file that you download from https://boostorg.jfrog.io/ui/native/main/release/1.69.0/source/1.69.0.zip is another web page - try putting the url in your browser and see what happens.