Search code examples
apache-sparkbigdataspark-streamingdynamic-allocation

Spark Streaming - dynamic allocation do not remove executors in middle of windows interval

I have a spark streaming job with batch Interval of 10 minutes and slides/window of 1 hour, I have activated the dynamic allocation with Spark.

But executors get removed after 1 hours and not after each 10 min of Batch Interval, So I have to wait 1 hour to see executors removed (in case of no data to process).

Is there a way to tell spark to revise excecutors removal after each batch interval instead of Window interval ?

Thanks.

Solution

  • Please check executor idle timeout property settings in job level or default config (spark-defaults.conf)

    • spark.dynamicAllocation.executorIdleTimeout
    • spark.dynamicAllocation.cachedExecutorIdleTimeout

    Refer this link for more details.