I want to make sure my spark job doesn't take more memory than what I pass, let's say 400GB is the max the job can use, from my understanding turning off dynamic allocation (spark.dynamicAllocation.enabled = false) and passing --num-executors --executor-memory --driver-memory do the job in Cloudera stack? correct if wrong.
is there any other setting that I have to set to make sure spark job doesn't go out of limit.
found a solution at my work Cloudera cluster has a special yarn parameter which doesn't let a job to go over certain limit which have to turned off or reset it.
https://community.cloudera.com/t5/Support-Questions/Yarn-memory-allocation-utilization/td-p/216290