Return HTTP response and continue processing - python bottle API

Question

I have an analysis service using Python's bottle that is invoked by posting a url to the raw data.

What is the simplest/cleanest/best way to immediately return a 200 response to the callee and continue processing a potentially long-running logic in the application?

Spawn a new thread/process? Use a queue? Async?

from bottle import post, HTTPResponse, request

@post('/postprocess')
def records():
   data_url = request.json.get('data_url', False)
   postback_url = request.json.get('postback_url', False)

   if not data_url or not postback_url:
       return HTTPResponse(status=400, body="Missing data paramater")

   #Immediately return response to callee
   return HTTPResponse(status=200, body="Complete")

   #Continue processing long-running code
   result = get_and_process_data(data_url)
   #POST result to another endpoint 

Answer 1

There is no simplest solution to this. For a production system I would first look into an existing system created for these kinds of situations to determine if that would be a good fit, and if not, only then develop something more suitable to my situation. To that end, I would recommend you take a look at Celery