Search code examples
apache-sparkpysparkhadoop-yarn

Spark/Yarn: Ramp up number of executors slowly over a period of time

I have a Spark job to run on a cluster managed with Yarn (with dynamic allocation turned on). My job needs to process data by hitting a few company-internal REST APIs (this is not optional). The API team has told me that they can auto-scale to a large number of concurrent requests, but that they need some time to scale up. So they've requested that I limit the rate of increase in requests/sec generated by my Spark job.

Is there any simple way to tell Spark to "ramp up" the number of executors evenly over a predefined period of time?

Solution

  • From the Spark Docs, you can slow down the rate that new executors are requested with spark.dynamicAllocation.schedulerBacklogTimeout and spark.dynamicAllocation.sustainedSchedulerBacklogTimeout. This feels like a bit of a hack, though.