Search code examples
google-cloud-platformgoogle-cloud-storagesftpgoogle-cloud-data-fusion

Error while Data Ingestion from SFTP to GCS or BigQuery using Cloud Data Fusion

I am trying to move CSV files in SFTP folder to GCS using Data Fusion. But I am unable to do it and throwing below error:

Here are the properties of both FTP and GCS plugins. Surprisingly, I could see the data in PREVIEW mode in all the stages but when I try to deploy the pipeline it fails. I tried using CSVParser as well as a TRANSFORM in between source(FTP) and sink (GCS). Still it shows the same error. I am using FTP plugin in Hub with version 3.0.0. Please help me to solve it.

enter image description here

And the error is as below, when I try to deploy the pipeline, eventhough Preview Data I was able to see the data.

enter image description here

Solution

  • I solved this issue by changing the Pipeline execution engine from SPARK to MAPREDUCE in Data Fusion. Now it is working.