Search code examples
airflowsftpgoogle-cloud-composer

Google Cloud Composer - Unknown hook type "sftp"

Trying to add SFTP connection to Apache Airflow run on the Composer.
Here is some packages' listing;

package-name version
apache-airflow 2.2.5+composer
apache-airflow-providers-ftp 3.2.0
apache-airflow-providers-google 2022.10.17+composer
apache-airflow-providers-sftp 4.1.0
apache-airflow-providers-ssh 3.2.0
paramiko 2.12.0
sshtunnel 0.4.0

This is the message I get when hitting the Test button:

enter image description here


Any help would be appreciated.

Solution

  • This is an expected behavior on Composer 1 because in Composer 1 if DAG serialization is on, the Airflow webserver is using a base image that doesn’t include customer installed pypi packages.

    Therefore, this functionality doesn’t work properly.

    You may upgrade to Composer 2 so that everything will work properly.

    However, if you want to stay on Composer 1, you may try changing the conn_type from ‘sftp’ to ‘ftp’.