I'm writing an API where a user can create a job to be processed with a POST request to /jobs. This returns a 201 with an ID the user can use to check the status of the job later.
While the job is processing, a GET to /jobs/{id} gives back a 204. If the job is complete, it returns a 200 with the results.
I'm unsure what status code to return in the exceptional situation where processing the job fails for some reason. I don't want to return a 204 since that's for jobs that are still going, and not a 404 since the job exists (ie the ID is valid) but it just didn't complete successfully. The closest I've got is a 500, but the API isn't having trouble handling the request, it knows the job has failed. I just don't know how best to communicate that with the status code.
Yes, you must not use 4xx codes as they are client errors. Also 5xx isn’t a variant as there is no error during processing of your state request actually.
HTTP codes doesn’t fit such case so it’s better to use your own method to return error message with 200 status code. If you still want to use HTTP status for returning result you can check 205 Reset Content code.