Search code examples
bashshellwget

How to download the best track JTWC data files from 1945 to 2020 using loop?


I want to download the data files from the URLs using a loop from 1945 to 2020, only one number changes in the URL, The URLs are given below https://www.metoc.navy.mil/jtwc/products/best-tracks/1945/1945s-bio/bio1945.zip https://www.metoc.navy.mil/jtwc/products/best-tracks/1984/1984s-bio/bio1984.zip https://www.metoc.navy.mil/jtwc/products/best-tracks/2020/2020s-bio/bio2020.zip

I tried the following code, but it throws an error

    for i in {1945..2020}
    do
        wget "https://www.metoc.navy.mil/jtwc/products/best-tracks/$i/$is-bio/bio$i.zip"
    done

Solution

  • I did changed your code slightly

    for i in {1945..1947}
      do
        echo "https://www.metoc.navy.mil/jtwc/products/best-tracks/$i/$is-bio/bio$i.zip"
      done
    

    when run it does output

    https://www.metoc.navy.mil/jtwc/products/best-tracks/1945/-bio/bio1945.zip
    https://www.metoc.navy.mil/jtwc/products/best-tracks/1946/-bio/bio1946.zip
    https://www.metoc.navy.mil/jtwc/products/best-tracks/1947/-bio/bio1947.zip
    

    Notice that first one is not https://www.metoc.navy.mil/jtwc/products/best-tracks/1945/1945s-bio/bio1945.zip as you might expect - 2nd $i did not work as intended, as it is followed by s it was understand as variable is which is not defined. Enclose variable names in { } to avoid confusion, this code

    for i in {1945..1947}
      do
        echo "https://www.metoc.navy.mil/jtwc/products/best-tracks/${i}/${i}s-bio/bio${i}.zip"
      done
    

    when run does output

    https://www.metoc.navy.mil/jtwc/products/best-tracks/1945/1945s-bio/bio1945.zip
    https://www.metoc.navy.mil/jtwc/products/best-tracks/1946/1946s-bio/bio1946.zip
    https://www.metoc.navy.mil/jtwc/products/best-tracks/1947/1947s-bio/bio1947.zip
    

    which is compliant with example you gave. Now you might either replace echo using wget or save output of code with echo to file named say urls.txt and then harness -i option of wget as follows

    wget -i urls.txt
    

    Note: for brevity sake I use 1945..1947 in place of 1945..2020