Search code examples
scalaapache-sparkooyala

use of spark job server

I am new to spark. I want know the use to spark job server like ooyala and Livy. is spark code cannot communicate with HDFS directly. Do we need a medium like webservice to send and receive data.

Thanks and Regards,

Solution

  • ooyala and Livy

    Persistent Spark Context or not help you run spark like a http-service which you can request to create or run a spark job directly from a http call instead of calling from command like to run or cronjob for instance.

    is spark code cannot communicate with HDFS directly

    No. It can communicate with HDFS directly.

    Do we need a medium like webservice to send and receive data.

    Actually ooyala and Livy they return the result by json like you do call in a API. So it depends on you build a medium or not.