Search code examples
apache-sparksnappydata

SnappyData REST API to Submit Job

I am trying to submit Snappy Job using REST API.

  • We have been able to submit SnappyJob using snappy-job submit Command Line tool.
  • I could not find any documentation how to do the same thing through REST API.

  • I found somewhere mentioned in the forum that SnappyData is using the spark jobserver REST API.

    Could you point to the Documentation / User Guide how to do that?

Solution

  • Snappydata internally uses spark-jobserver for submitting the jobs. Hence, all the spark-jobserver REST APIs are accessible on Snappydata's lead node.

    You can refer to all spark-jobserver API here: https://github.com/SnappyDataInc/spark-jobserver#api

    Here are some useful curl commands to clarify it further:

    • deploy application jar on job-server:

    curl --data-binary @/path/to/applicaton.jar localhost:8090/jars/testApp

    testApp is the name of the job server app which will be used to submit the job

    • create context:

    curl -X POST "localhost:8090/contexts/testSnappyContext?context-factory=org.apache.spark.sql.SnappySessionFactory"

    testSnappyContext is the name of the context which will be used to submit the job.

    Also, note that we are passing a custom context-factory argument here which is necessary for submitting snappy job.

    • submit the job:

    curl -d "configKey1=configValue1,configKey2=configValue2" "localhost:8090/jobs?appName=testApp&classPath=com.package.Main&context=testSnappyContext"

    com.package.Main is the fully-qualified name of the class which is extending org.apache.spark.sql.SnappySQLJob.

    • stop the job

    curl -X DELETE localhost:8090/jobs/bfed84a1-0b06-47ca-81a7-9b8defb51e38

    bfed84a1-0b06-47ca-81a7-9b8defb51e38 is the job-id which you will get in the response of job submit request

    • stop the context

    curl -X DELETE localhost:8090/contexts/testSnappyContext

    • undeploying the application jar

    The version of job-server being used by snappydata doesn't have a RESTful API exposed for undeploying the jar. However, deploying any jar with the same app name (testApp in our example) will override the previously deployed jar for the same app.