Search code examples
bashcurlquoting

Failed to unmarshall yaml


I have an execute endpoint that runs a child process which in turn runs a rancher-compose command to start up some containers within a host on rancher. When I hit this /execute endpoint from the node app it work fine and starts the containers as expected. When hitting the /execute endpoint from a bash script through cURL command it fails and gives me the error below. I don't understand why it throws an error that it cannot unmarshall my yaml file just when the command is run through a bash script. This is leading me to believe that this is not an issue with my yaml file. Below I put the error and the command that's failing and the cURL command i'm using.

I'm absolutely lost and any help would be greatly appreciated!

ERROR starting job, unable to run sub-process for container deployment. >ERR=Error: Command failed: /rancher-tools/rancher-compose --project-name mic-iwbl3g97 --verbose --file docker-compose-job-submitter.yml up -d

job deployment output: time="2016-12-05T04:35:49Z" level=debug msg="Environment Context from file : map[]" time="2016-12-05T04:35:49Z" level=debug msg="Opening compose file: docker-compose-job-submitter.yml" time="2016-12-05T04:35:49Z" level=debug msg="Looking for stack mic-iwbl3g97" time="2016-12-05T04:35:49Z" level=info msg="Creating stack mic-iwbl3g97" time="2016-12-05T04:35:49Z" level=error msg="Failed to unmarshall: yaml: unmarshal errors:\n line 15: cannot unmarshal !!str NaN into int64\n line 16: cannot unmarshal !!str NaN into int64\ncommand:\n- mic-iwbl3g97\n- NaN\n- $input_csv_file\n- $start_year\n- $end_year\ncontainer-name: mic-iwbl3g97\ncpu_shares: 25\nimage: $repo_name:$tag\nlabels:\n io.rancher.container.pull_image: always\n io.rancher.container.start_once: true\n io.rancher.scheduler.affinity:host_label: MIC-Use=TPC_Modeling\n io.rancher.sidekicks: locking, input, output, param, input-helper\nmem_limit: NaN\nmemswap_limit: NaN\nnetwork_mode: none\nvolumes_from:\n- input\n- output\n- param\n- locking\n"

curl -s -X POST -H "Content-Type: application/json" -d '{"S3_param_file":"$input_csv_file", "memory_constraint":"$memory_constraint", "time_constraint":"$time_constraint", "start_year":"$start_year", "end_year":"$end_year", "tag":"$tag", "repo":"$repo_name"}' http://my_ip_address:3000/execute

Solution

  • Variables will not be expanded in single quotes, use double quotes to expand:

    curl -s -X POST \
      -H "Content-Type: application/json" \
      -d '{"S3_param_file":"'"$input_csv_file"'", "memory_constraint":"'"$memory_constraint"'", "time_constraint":"'"$time_constraint"'", "start_year":"'"$start_year"'", "end_year":"'"$end_year"'", "tag":"'"$tag"'", "repo":"'"$repo_name"'"}' \
      http://my_ip_address:3000/execute
    

    See how i swap between single quotes and double quotes:

    '{"json_here": "'"$parameter_here"'"}'
    sssssssssssssssssdddddddddddddddddssss
    

    Where s represent single quotes, and d represent double quotes.

    At evidenced above it quickly becomes a mess, one can however use jq to create JSON:

    $ jq -n --arg a "a val" --arg b 'b val'  '{$a, $b}'
    {
      "a": "a val",
      "b": "b val"
    }