pySpark Hadoop AWS s3 requester-pays.enabled config doesn't work

Question

I am trying to read AWS S3 bucket with pyspark. The bucket requires requester to pay to read.

However, it doesn't seem to work although the similar credentials on aws-cli works and the reason that I believe spark.hadoop.fs.s3a.requester-pays.enabled config is the reason is because if I remove the parameter --request-payer requester on aws-cli I get the exactly same error.

Below is my code for pyspark configuration

spark = SparkSession.builder  \
    .appName("MainnetBlocksStreamingJob") \
    .config("spark.jars.packages", "org.apache.hadoop:hadoop-aws:3.2.0,com.amazonaws:aws-java-sdk-bundle:1.11.375") \
    .config("spark.hadoop.fs.s3a.access.key", S3_ACCESS_KEY) \
    .config("spark.hadoop.fs.s3a.secret.key", S3_SECRET_KEY) \
    .config("spark.hadoop.fs.s3a.endpoint", "s3.amazonaws.com") \
    .config("spark.hadoop.fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem") \
    .config("spark.hadoop.fs.s3a.path.style.access", "true") \
    .config("spark.hadoop.fs.s3a.requester-pays.enabled", "true") \
    .config("spark.hadoop.fs.s3a.requester.pays.enabled", "true") \
    .config('spark.hadoop.fs.s3a.aws.credentials.provider', 'org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider')\
    .getOrCreate()

And I running pyspark with the command

spark-submit \--packages io.delta:delta-spark_2.12:3.3.0,org.apache.hadoop:hadoop-aws:3.2.0,com.amazonaws:aws-java-sdk-bundle:1.11.375 \
--conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=file:log4j.properties" \
--conf spark.hadoop.fs.s3a.requester-pays.enabled=true \
dataproc_jobs/streaming.py

Thank you.

Answer 1

It's fs.s3a.requester.pays.enabled as per the s3a documentation.

However, the feature only came in with HADOOP-14661 S3A to support Requester Pays Buckets, which says it is a Hadoop 3.3.5 feature.

Upgrade all your hadoop-* dependencies to 3.3.5 or later for this to work.