Opening a XMLRPC Port from Apache Spark Java Code

My requirement is to serve a web service request using Apache Spark.

I developed a XML RPC server which will be triggered by a servlet code and it will in-turn run a spark-submit application. XML RPC will then wait for spark-submit to complete and send the result back to servlet.

My questions are:

Can I write a XML RPC server inside my spark job, and serve all request directly from my servlet with using standalone XML RPC server?
What is the correct method to trigger Apache spark for each web service request?

Solution

Integrating spark with servlets can be challenging due to dependency issues, but leaving that aside, you might be better suited to having a long running driver program with a shared spark context since creating a spark context can take time (and you can't share cached data between different spark contexts). The Ooyala Spark job server can be useful for this, IBM also has a project intended for doing something similar called the Spark Kernel. Hope one of these projects