Passing a private key through Fly Cli

The company I work for has recently started to use Concourse CI to do all our CI needs. At the current moment one of my jobs consists of a task with a script that scp's and ssh's into our aws ec2 instances and configures those servers. The issue I am having, however, is getting the private key to ssh into those instances. One way discussed here (https://concourse-ci.org/fly-set-pipeline.html) is to pass the key in through a variable. In my script, I take that variable passed in and echo it to a new .pem file and I set the permissions to 600. When I echo just the variable and later cat the new .pem file, they look exactly the same as the original .pem file. The container I am trying to ssh from is the standard ubuntu docker image.

When I try to use this file to scp and ssh I am confronted with the prompt about entering the passphrase. If I try to ssh with the original file I don't get this prompt at all. Is there something I am missing? I would greatly appreciate some insight into this issue.

pipeline.yml

jobs:
- name: edge-priceconfig-deploy
  plan:
  - aggregate:
    - get: ci-git
    - get: pricing-config
  trigger: true
  - task: full-price-deploy
    file: ci-git/ci/edge/edge-price-config-task.yml
    params:
      USER_AND_SERVER: {{edge_user_and_server}}
      DEPLOY_KEY_PAIR: {{deploy_key_pair}}

task.yml

---
platform: linux

image_resource:  
  type: docker-image
  source: {repository: ubuntu}

inputs:
- name: ci-git
- name: pricing-config

run:
  path: ./ci-git/ci/edge/edge-priceconfig-deploy.sh

task.sh

#!/bin/bash
touch DeployKeyPair.pem

echo $DEPLOY_KEY_PAIR
echo $DEPLOY_KEY_PAIR > DeployKeyPair.pem
cat DeployKeyPair.pem

apt-get update && apt-get -y install sudo
sudo apt-get -y install openssh-client
sudo chmod 400 ci-git/key/DeployKeyPair.pem
sudo chmod 600 DeployKeyPair.pem

mkdir company-price-config-edge
mv pricing-config/fsconfig/conf/com.company.api.v1.pricing/*.xlsx company-price-config-edge/

commandstr="sudo rm -f /etc/company/edge/fsconfig/*xlsx; \
        ls -l /etc/company/edge/fsconfig; \
        sudo mv /home/ec2-user/company-price-config-edge/*xlsx /etc/company/edge/fsconfig/; \
        sudo rm -rf /home/ec2-user/company-price-config-edge;"


scp_link="$USER_AND_SERVER:/home/ec2-user/"
scp_link="$(echo $scp_link | tr -d ' ')"
echo $scp_link

sudo echo -ne '\n' | scp -r -oStrictHostKeyChecking=no -i DeployKeyPair.pem company-price-config-edge $scp_link
sudo ssh -oStrictHostKeyChecking=no -i DeployKeyPair.pem $USER_AND_SERVER $commandstr

credentials.yml

username: |
  username

password: |
  password

access_token: |
  token

ci_scripts_github: |
  ci-script-link

edge_user_and_server: |
  server.com

staging_user_and_server: |
  staging

training_user_and_server: |
  training

production_user_and_server: |
  production

deploy_key_pair:
  -----BEGIN RSA PRIVATE KEY-----
  ...
  -----END RSA PRIVATE KEY-----

Solution

Had this same problem today and took me a bit to figure out what was going on.

You're specifying the private key as a multiline yaml property, but when it's being echoed out to a file it's stripping out the linebreaks and replacing them with spaces, so your key is a large single line file.

I had to use sed to replace the space with newlines to get it to work

echo $DEPLOY_KEY_PAIR | sed -e 's/$KEY-----$\s/\1\n/g; s/\s$-----END$/\n\1/g' | sed -e '2s/\s\+/\n/g' > DeployKeyPair.pem

First sed command uses two sets of replacement groups, both with the same purpose

First group - Grab header text and break it out to a new line

s/$KEY-----$\s/\1\n/g

$KEY-----$ puts the last part of the header block in a capture group
\s matches whitespace
/1\n/g Puts capture group (#1) back in, then adds a newline
/g Unnecessary global capture that I forgot to remove

Second sed group does the same but grabs the space and first part of the footer text to put it on a new line

Third group does a global replacement on whitespace and replaces it with newline. This had to be a separate command so processing was finished for first command and it was then a 3-line file/stream

'2s/\s\+/\n/g'

2s/ Perform replacement only on line 2
\s\+ matches any whitespace
\n replace with newline
/g global match to grab every instance