amazon-web-servicesapache-sparkemr-serverless
EMR serverless- Pass jars in console
I'm new with EMR-serverless and I want to know how to pass, in a spark application, jar and packages as for example:
spark-submit --deploy-mode client --jars /usr/lib/hudi/hudi-spark3.3-bundle_2.12-0.11.1-amzn-0.jar,/usr/lib/hudi/hudi-utilities-bundle_2.12-0.11.1-amzn-0.jar ...
I want to set when I submit a job but I cannot find a way about how to do it.
Can someone help me with this, please?
Solution
When submitting a job to EMR Serverless in the console and you want to provide additional options to spark-submit, you can use the "Spark properties" section. Instead of --jars, you can use the spark.jars key and set the value appropriately.
Your Spark application will be a Python script or JAR file on S3 provided as the "Script location" aka entrypoint.
Also note that Hudi is available on the EMR Serverless image and there's some documentation on using Hudi with EMR Serverless.
- AWS RDS restoring snapshot with upgraded engine version giving error
- Error with not existing instance profile while trying to get a django project running on AWS Beanstalk
- Timeout when trying to retrieve EC2 instance-id metadata from within it
- Lease table is not updated when new shards are added causing stale workers in KCL
- libsndfile.so.1: cannot open shared object file: No such file or directory
- Why does ECS binpack strategy on memory always scale up a new EC2 instance despite available resources?
- How to pull every single image (tag or untagged) from AWS ECR?
- In CloudFormation how can I link RDS and Secrets Manager so I don't have to hardcode the password?
- How to install postgresql-client to Amazon EC2 Linux machine?
- PyCharm: Why "Python Console" is not accessing ~\.aws\credentials file? How to set it within "Python Console"
- Is it possible to run aws s3 sync with boto3?
- Python 3 asyncio with aioboto3 seems sequential
- Serverless / AWS Lambda - Create the triggers for the published lambda versions
- AWS Glue missing permissions
- Invoking lambda from API gateway test, but hitting the endpoint does not invoke the lambda. 500 returned
- terraform destroy needs original terraform code to destroy?
- How to correctly/safely access parameters from AWS SSM Parameter store for my Python script on EC2 instance?
- Why is my image not displaying on my Web page?
- How to copy blobs from azure to aws using python?
- S3 Bucket action doesn't apply to any resources
- Couldn't proceed with upgrade process as new nodes are not joining node group standard-workers
- How to customize "From" name for SNS emails
- Changing the name of a Load Balancer on AWS Console
- AWS S3 Bucket Access Issue with IAM User Permissions
- how to decrease the exceution time of aws lambda function nodejs
- Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect)
- Downloading an entire S3 bucket?
- Unable to unzip .zip file on linux machine
- Missing Authentication Token while accessing API Gateway?
- AWS S3 Access Denied when open object URL in browser