Search code examples
airflowaws-secrets-managerairflow-2.xdbt-cloud

dbt Cloud airflow conn_id from secret manager breaking parsing

I'm using aws secret manager to store my connections in airflow. So when creating the dbt connection string in the secret , i followed this pattern https://airflow.apache.org/docs/apache-airflow-providers-dbt-cloud/1.0.2/connections.html

dbt-cloud://account_id:api_token@

eg: dbt-cloud://111:dbts_-abc/Qo==@

The api_token generated by dbtcloud has a "/ "in it and is causing an error.

File "/usr/lib64/python3.7/urllib/parse.py", line 172, in port
port = int(port, 10)
ValueError: invalid literal for int() with base 10: 'dbts_-abc'

Did anybody run into this issue before? How do i fix this? If I just use the airflow connections ui to get the api_token then it works fine.

Solution

  • You need to escape forward slashes with %2F when you generate the URI. Check out this related StackOverflow answer.

    Also, the Airflow docs show an easy way to generate the Connection URI too.