how to call databricks notebook from python using rest api

Question

I want to create a python notebook on my desktop that pass an input to another notebook in databricks, and then return the output of the databricks notebook. For example, my local python file will pass a string into a databricks notebook, which will reverse the string and then output the result back to my local python file. What would be the best way to achieve this?

This is what I tried but when I try to create a new run, I get this error. Is my json formatted incorrectly or am I missing something else? Thanks

import os
from databricks_cli.sdk.api_client import ApiClient
from databricks_cli.clusters.api import ClusterApi

os.environ['DATABRICKS_HOST'] = "https://adb-################.##.azuredatabricks.net/"
os.environ['DATABRICKS_TOKEN'] = "token-value" 

api_client = ApiClient(host=os.getenv('DATABRICKS_HOST'), token=os.getenv('DATABRICKS_TOKEN'))

runJson = """
        {
        "name": "test job",
        "max_concurrent_runs": 1,
        "tasks": [
            {
            "task_key": "test",
            "description": "test",
            "notebook_task":
                {
                "notebook_path": "/Users/[email protected]/api_test"
                },
            "existing_cluster_id": "cluster_name",
            "timeout_seconds": 3600,
            "max_retries": 3,
            "retry_on_timeout": true
            }
            ]
        }
        """

runs_api = RunsApi(api_client)
runs_api.submit_run(runJson)

Error: Response from server:

{ 
  'error_code': 'MALFORMED_REQUEST',
  'message': 'Invalid JSON given in the body of the request - expected a map'}

Answer 1

You should provide the payload as Python dict, not as string. Just remove """ around the runJson payload.