I have a spark cluster in yarn mode, and some batch and streaming jobs to run. I find that spring xd could process batch and stream jobs, with its embedded spark.
My question is, can spring xd use an external spark cluster in yarn mode? How to do that? Thanks a lot.
You can't launch a Spark job in yarn mode with Spring XD, only launch local or on a Spark cluster using master URL. With Spring Cloud Data Flow you can launch a spark-yarn task, see http://docs.spring.io/spring-cloud-task-app-starters/docs/1.0.1.RELEASE/reference/html/_spark_yarn_task.html