Search code examples
amazon-s3ibm-cloudgithub-actions

Cannot cp or sync with s3 but I can ls


I'm working on a actions workflow and wouldlike to download a file from s3. Our s3 is via IBM cloud and seems to have some nuances around usage e.g. one must add --endpoint-url {endpoint} in between the command to call aws client and the corresponding aws command, e.g. aws --endpoint-url {endpoint} s3 ls

Here are the relevant steps in my workflow.yml file:

jobs:
  retreive-model-s3:
    runs-on: ubuntu-latest
    steps:
      - name: checkout current repo
        uses: actions/checkout@master
      - name: make dir to sync with s3
        run: mkdir models
      - name: s3
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          AWS_S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }}
          AWS_S3_ENDPOINT: ${{ secrets.AWS_S3_ENDPOINT }}
          AWS_REGION: 'us-south'
        run: |
          aws --endpoint-url ${{ secrets.AWS_S3_ENDPOINT }} s3 ls ${{ secrets.AWS_S3_BUCKET }}
          aws --endpoint-url ${{ secrets.AWS_S3_ENDPOINT }} s3 cp ${{ secrets.AWS_S3_BUCKET }}/regression_model_full.rds models/regression_model_full.rds
      - name: dir after

Up to here works:

aws --endpoint-url ${{ secrets.AWS_S3_ENDPOINT }} s3 ls ${{ secrets.AWS_S3_BUCKET }}

In the actions console output I am able to see the files in ${{ secrets.AWS_S3_BUCKET }}, including a file called regression_model_full.rds.

In the next line I attempt to download it to the runner with:

aws --endpoint-url ${{ secrets.AWS_S3_ENDPOINT }} s3 cp ${{ secrets.AWS_S3_BUCKET }}/regression_model_full.rds models/regression_model_full.rds

But this results in error:

usage: aws s3 cp <LocalPath> <S3Uri> or <S3Uri> <LocalPath> or <S3Uri> <S3Uri>
Error: Invalid argument type
Error: Process completed with exit code 252.

Given I am able to successfully run the ls command for s3 I'm reassured that my connection to s3 is sound. But why would I not be able to download using cp? Is my syntax incorrect?

enter image description here


Solution

  • Plesae check addid s3: ''' aws --endpoint-url ${{ secrets.AWS_S3_ENDPOINT }} s3 cp s3://${{ secrets.AWS_S3_BUCKET }}/regression_model_full.rds models/regression_model_full.rds '''