Twisted how to get rid of Unhandled error in Deferred?

Question

I have a deferredList that will fire on the first callback but I'm actually playing with the nature of callback chaining and so if no callback is ever fire, errbacks will fire instead.

The problem is once I add the handlers for the errbacks, addErrback and a Failure that I created, triggers and the stdout still yells me the following:

Unhandled error in Deferred:

Traceback (most recent call last):
Failure: __main__.notFound: Match not found!

With the final errback doing the following:

def emptyQuery(error, name):
    errorType = error.trap(notFound)
    if errorType == notFound:
        print("[-] Item not found in given sources: {name}".format(name=name))
    print("->", errorType)
    return

Thus,

[(False, <twisted.python.failure.Failure __main__.notFound: Match not found!>)]
[-] Item not found for current sources: GIA-S12
-> <class '__main__.notFound'>
Unhandled error in Deferred:

Traceback (most recent call last):
Failure: __main__.notFound: Match not found!

How can I get rid of that "Unhandled thing"? I traped the error, I don't know why it yells me this way.

It is actually triggered with this for loop I implemented:

for item in items:
        name, price = item
        itemQuery_deferreds = []
        for urlObject in URLS:
            urlFire = poolsem.acquire().addCallback(initQuery, urlObject.url, name, googleClient)
            itemQuery_deferreds.append(urlFire)
        itemQuery = DeferredList(itemQuery_deferreds, fireOnOneCallback=True)
        itemQuery.addCallback(parseData)
        itemQuery.addErrback(emptyQuery, name)

Of course, before going to the errback handler emptyQuery I have to expect the callback the be called first:

def parseData(data):
    print(data)
    for code, status in data:
        if not code:
            return status
        else:
            store = status

ALSO, every Deferred that I have in all my code contains an errback handler:

def onError(error):
    return error

Note: I find this very hard to debug, if you need more info please just ask.

Answer 1

Pass consumeErrors=True to your DeferredList construction.

You're filling the DeferredList with Deferreds created like:

urlFire = poolsem.acquire().addCallback(initQuery, urlObject.url, name, googleClient)

If urlFire ever fails then the DeferredList observes this failure and makes it available in its own result. If you don't pass consumeErrors=True then DeferredList makes sure urlFire keeps its failure result. If you do pass consumeErrors=True then DeferredList turns any failures on its elements into None.

ALSO, every Deferred that I have in all my code contains an errback handler:

def onError(error):
   return error

There is no point to this. This onError errback is a no-op. It takes an error and then propagates it. d and d.addErrback(onError) will behave exactly the same way. If you want to get rid of an error, you have to not re-raise it or return it.

What DeferredList(consumeErrors=True) effectively does is make DeferredList add an errback like:

def onError(reason):
    return None

to each Deferred. The reason you don't want to do this to each urlFire Deferred you make is that it is the equivalent of "except: pass". It will suppress all errors from those Deferreds. This will prevent even DeferredList from doing anything with them. DeferredList(consumeErrors=True) makes sure that the errors get propagated somewhere else so it's fine to suppress them on the individual Deferreds.