dbt-athena-comunity package with gitActions

Unable to properly execute workflow on gitActions using dbt-athena-community package. When executing the following credential issue appears:

error log from workflow git action execution

dbt.exceptions.DbtProfileError: Runtime Error
Credentials in profile "default", target "dbt_dev" invalid: Runtime Error
Could not find adapter type athena!

dbt-athena-community package demands a live connection, but it seems the credentials profile does not work.

Here is my project structure as well as config files.

Project folder structure

./.github/workflows/lint_models.yml
./models <----.sql files to lint
./.sqlfluff
./profiles.yml

profiles.yml

default:
  outputs:
    dbt_dev:
      database: awsdatacatalog
      region_name: eu-west-2
      s3_staging_dir: s3://athena-query-results-XXX/
      schema: dbt_test
      type: athena
      aws_profile_name: prof_name  <-- as per comment removed this line
      work_group: primary
  target: dbt_dev

.sqlfluff

[sqlfluff]
templater = dbt
dialect = athena
sql_file_exts = .sql

[sqlfluff:templater:dbt]
project_dir = ./
profiles_dir = ./
profile = default  <-- as per comment removed this line
target = dbt_dev   <-- as per comment removed this line

lint_models.yml

name: dbt sqlfluff
on:
  pull_request:
jobs:
  test-check:

    name: runner / sqlfluff (github-check)
    runs-on: ubuntu-latest
    steps:

      - name: install python
        uses: actions/setup-python@v4
        with:
          python-version: 3.11.2
      
      - name: checkout repo
        uses: actions/checkout@v3

      - name: install dbt adapters
        run: |
          pip install dbt-athena-community==1.4.1
          dbt deps
          dbt --version
      
      - name: sqlfluff
        uses: yu-iskw/[email protected]
        env:
            AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
            AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
            AWS_DEFAULT_REGION: ${{ secrets.AWS_DEFAULT_REGION }} 
        id: lint-sql
        with:
          github_token: ${{ secrets.github_token }}
          reporter: github-pr-review
          sqlfluff_version: "1.4.5"
          sqlfluff_command: "fix" # Or "lint"
          config: "${{ github.workspace }}/.sqlfluff"
          paths: '${{ github.workspace }}/models'
          
      - name: 'Show outputs (Optional)'
        shell: bash
        run: |
          echo '${{ steps.lint-sql.outputs.sqlfluff-results }}' | jq -r '.'
          echo '${{ steps.lint-sql.outputs.sqlfluff-results-rdjson }}' | jq -r '.'

Note, both secrets are the same used to run in local machine through AWS Boto credentials, no issue on local machine.

AWS credentials

When running locally credentials do work, installed AWS cli and configure boto credentials for the profile.

I'm assuming something similar should be done in the workflow that I'm missing.

Solution

As you are passing AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in your env therefore you don't need to setup the aws_profile_name: prof_name in your profiles. The adapter uses the standard boto3 authentication, and it looks for prof_name in your ~/.aws/config or ~/.aws/credentials, if you don't set any profile the AWS_* variables are used.

You might want to use 2 different profiles, one for working locally, with aws_profile_name set, and one for the CI/CD without the aws_profile_name set.