when to update driver resources

Question

All,

Since the driver does not hold any data or process any data, under what circumstances would the default resources assigned to spark.driver.memory and spark.driver.cores need to be updated with higher values.

Can someone point me in the right direction with some links that I can read?

Thanks,
grajee

Answer 1

TLDR answer :

spark.driver.memory : When ever you are bringing data to driver. More specifically like a Collect or Broadcast operation

spark.driver.cores : After all - driver process is either a Scala/Java/Python process. If you want to do some multithreading (like background process ) then you can use higher value for cores.