Celery - Exception re-raise information

Question

With celery, if I define the following task:

@app.task(bind=True)
def Cmd(self):
    self.log.info(f"Running cmd")
    with open(f'/tmp/nopermission', 'w') as file:
         ...

I start my celery worker, and then from my main.py:

res = Cmd.delay()
print(res.get())

in the worker's log, I get:

Traceback (most recent call last):
  ...
  File "tasks.py", line 5, in Cmd
    with open(f'/tmp/nopermission', 'w') as file:
PermissionError: [Errno 13] Permission denied: '/tmp/nopermission'

But, the print(res.get()) gives me:

 File "site-packages/celery/result.py", line 211, in get
    self._maybe_reraise_parent_error()
  File "site-packages/celery/result.py", line 234, in _maybe_reraise_parent_error
    node.maybe_throw()
  File "site-packages/celery/result.py", line 333, in maybe_throw
    self.throw(value, self._to_remote_traceback(tb))
  File "site-packages/celery/result.py", line 326, in throw
    self.on_ready.throw(*args, **kwargs)
  File "vine/promises.py", line 244, in throw
    reraise(type(exc), exc, tb)
  File "vine/five.py", line 195, in reraise
    raise value
PermissionError: [Errno 13] Permission denied

Basically, I do not have information on the exact error (in this case, the name of the file). Why is that?

And, as a more general question, what is the most elegant way of attaching extra data to my result? For example, I have the name of the worker executing the task in app.conf.workername. Would I use on_failure / on_success handler from a task base class? (app = Celery(task_cls='path:baseTask'))

Answer 1

I found a way to do it, but in doing so, I think I discovered a bug.

in the baseTask class when initiating celery (Celery(task_cls='path:baseTask'))

class baseTask(Task):    
    def on_failure(self, exc, task_id, args, kwargs, einfo):
        self.log.error(f'{self.name} on {self.worker} failed after {self.execTime} seconds: {exc}')
        self.update_state(state=states.FAILURE, meta={
                'exc_type': type(exc).__name__,
                'exc_message': traceback.format_exc().split('\n'),
                'extra': "something"})

then, running any task:

try:
    task.delay()
    print(task.get())
except Exception as exc:
    time.sleep(2)
    info = task.backend.get(res.backend.get_key_for_task(task.id)).decode(encoding='UTF-8')
    info = json.loads(info)
    print(info['result']['extra'])
    raise exc

Why sleep? Well, if I don't, and keep running my Cmd task, half the time, I do not get 'extra' in 'result'. It's fairly random but that sleep seems to do the trick. I'd love to know exactly why though :) I have celery 4.4.2, transport: amqp, results: redis.