I'd like to retrieve the status of a spark job running in cluster mode on a mesos master via the following:
spark-submit --master mesos://<ip>:7077 --status "driver-...-..."
It exits 0 with no logging, no matter what the driver's status is.
I know that it's doing something right, since if I run the command with a an invalid mesos ip/port, I get
Exception in thread "main" org.apache.spark.deploy.rest.SubmitRestConnectionException: Unable to connect to server
at org.apache.spark.deploy.rest.RestSubmissionClient$$anonfun$requestSubmissionStatus$3.apply(RestSubmissionClient.scala:165)
and if I run with an invalid submission id, I get
2018-10-02 18:47:01 ERROR RestSubmissionClient:70 - Error: Server responded with message of unexpected type SubmissionStatusResponse.
Any idea why spark-submit --status isn't returning anything?
I found a workaround by accessing the dispatcher's api directly:
curl -s "http://$DISPATCHER/v1/submissions/status/$SUBMISSION_ID"
Still no clear answer why spark-submit --status does not behave as documented though.