I am looking to pass a variable through bash to an api call to download a file. How do I properly pass variables to an http call? Below is my bash file:
#!/bin/bash
#rm existing file
rm Desktop/ems/ems_raw_update.csv
#create a yesterday variable looking back 120 days
yesterday=$(date -d "$date -120 days" +"%Y-%m-%d")
#download data using that 120 days lookback
echo $yesterday | curl https://data.cityofnewyork.us/resource/76xm-jjuj.csv?$where=incident_datetime>='$yesterday' -o Desktop/ems/ems_raw_update.csv
The download fails I am assuming because of the way i've passed the variable $yesterday. For example entering the following into your browser will download the proper file:
https://data.cityofnewyork.us/resource/76xm-jjuj.csv?$where=incident_datetime>=‘2019-12-25’
I also note that in my directory i can see the $yesterday variable.
total 3.8G
-rw-rw-r-- 1 me me 0 Mar 22 19:48 =2019-11-23
-rw-rw-r-- 1 me me 0 Mar 22 19:46 =2019-12-30
-rwxr-xr-x 1 root root 488 Mar 23 13:12 downloader.sh
-rw-rw-r-- 1 postgres me 3.8G Mar 22 12:17 ems_raw.csv
-rw-rw-r-- 1 me me 64 Mar 23 12:51 ems_raw_update.csv
-rw-rw-r-- 1 me me 0 Mar 23 13:12 =$yesterday
How do I download this file properly and make sure i remove local variables?
Try this:
curl "https://data.cityofnewyork.us/resource/76xm-jjuj.csv?\$where=incident_datetime>='$yesterday'" -o Desktop/ems/ems_raw_update.csv
Explanation: Inside double-quotes, the shell will recognize $ as initiating variable substitution, so $yesterday will be expanded to the variable's value. It would not be expanded in single-quotes, and it looks like it's in single-quotes here, but since the single-quotes are themselves inside a double-quoted section, they don't have any meaning; they're just normal characters (and will get passed to curl as part of the URL argument). The ? and the > (in >=) are also special characters in shell syntax, but again the double-quotes make them just normal characters.
The $ in $where=incident_datetime is a different story. As I said, inside double-quotes, $ is treated as starting a substitution, so by itself the shell would try to expand a shell variable named where (which probably doesn't exist, so it'd come out blank). Escaping it with a backslash (\$where=incident_datetime) prevents this, turning it into just a normal character to be passed to curl (which will pass it to the server as part of the URL, and that'll presumably treat it as part of a database query).
It would also be possible to mix quoting modes: put parts of the URL in single-quotes and the part with the variable in double-quotes, but IMO this is more confusing in cases like this.
Oh, and don't try to pipe the variable to curl (the echo $yesterday |) part. That's for things that read data from their input stream, which curl doesn't (unless you tell it to, which you shouldn't here -- it's not relevant).
BTW, in your example browser URL, there are curly unicode quotes around the date value. I presume this is unintentional (due to some text editor "helpfully" converting them), and plain ASCII quotes are intended. If not, replace them as needed.