Search code examples
sftpgoogle-cloud-composermodulenotfounderror

GCP Composer - ModuleNotFoundError: No module named 'airflow.providers.sftp'

I trying to getting data from FTP server's txt file by GCP Composer Tasks.

So I imported SFTPOperator package in code.

but error occurred:

ModuleNotFoundError: No module named 'airflow.providers.sftp'

then, I tried few ways:

but didn't work.😭

My GCP Composer Environment is as below:

  • Image Version : composer-1.17.7-airflow-2.1.4
  • python version : 3
  • Network VPC-native : Enable

How can I use SFTPOperator ?

Solution

  • For this you will have to install sftp package, pip install 'apache-airflow[sftp]' . You can check the built-in and extras packages that airflow components have when installed (varies from version).

    Once you have it installed you should be able to use SFTPOperator by importing the operator inside your DAG.

    from airflow.providers.sftp.operators.sftp import SFTPOperation,SFTPOperator

with DAG(...) as dag:

   upload_op = SFTPOperator(
    task_id="test_sftp",
    ssh_conn_id="ssh_default",
    local_filepath="/tmp/file.txt",
    remote_filepath="/tmp/tmp1/tmp2/file.txt",
    operation=SFTPOperation.GET, 
    dag=dag
   )

   ...

    You can also find a mock tests on the airflow git hub project that can provide you some guidance, check this link.

    UPDATE 17/08/2022: As commented by Diana, Composer has a documented way to install its components as mention on this link. Be advised to pick up the composer version your project uses as there is version1 and version2 guides.