Search code examples
amazon-web-serviceshadoopamazon-s3apache-nifihortonworks-data-platform

Use NiFi to write into AWS S3 Bucket with MFA enabled

Is it possible to use the NiFi processor PutS3Object to write data into a S3 Bucket, while MFA is active?

And if not, what would be an alternative considering writing data to S3 from hive/HDFS?

Solution

  • As of version 1.8.0, I do not believe this is supported by the Apache NiFi *S3Object processors (or the AWSCredentialsProviderControllerService). I recommend opening a feature request ticket on the Jira site. You may be able to accomplish this immediately by using one of the following options:

    • Use ExecuteStreamCommand or ExecuteProcess to call a terminal command or shell script using the AWS S3 CLI
    • Use ExecuteScript to run a custom Python (/Ruby/Groovy/etc.) script which uses the AWS S3 SDK directly (example code)
    • Write a custom processor which uses the AWS S3 SDK directly