Search code examples
pythonexceptionfabricslack

Python fabric, send notification on failure

I've been trying to figure out what the best way is to do something when my fabric script fails (for example send a slack notification message via python module slackbot).

I've made an example where I try to do the above here:

fab_failtest.py my_slackclient.py

You can run above example by downloading both files to a directory, pip install fabric and slackbot, then run:

fab --fabfile=fab_failtest.py fail_test1 or

fab --fabfile=fab_failtest.py fail_test2

(you also have to have a machine you can ssh to, in this example I have mrbluesky@elo with open ssh port on 22)

  • fail_test1 uses try-except so I can get exception error info and so forth
  • fail_test2 uses try-finally plus a simple boolean variable so no exception info is available

At first I thought I had it with the fail_test1 example but I've seen it fail several times to send the slack message on failure, I'm wondering if there might be a race condition or something involved? I could start using fail_test2 instead but I really like to have access to the stack-trace like in fail_test1.

Is there a better way to do this, like, something provided in python fabric that does excatly what I'm trying to accomplish in above example?

Solution

  • I disagree with both your approaches. Im a strong believer that less code is better. What do i mean by that? A function should do what its name says, no more no less, if you have to add in a global handler like that i would add it in as a wrapper, fabric functions are difficult enough to read, no need to add error handling to the mix. With that said:

    import sys
import traceback

from fabric.api import task, settings, local, abort
from fabric.decorators import _wrap_as_new
from functools import wraps

HOST = 'elo'
PORT = 22


def alert_on_fail(func):
    @wraps(func)
    def decorated(*args, **kwargs):
        try:
            return func(*args, **kwargs)
        except:
            # TODO: add more code here
            exception_type, value, tb_msg = sys.exc_info()
            traceback_msg = traceback.format_exc()
            notify('something went wrong: ' + traceback_msg)
            abort('exiting error!!')
    return _wrap_as_new(func, decorated)


@task
@alert_on_fail
def fail_test(host=HOST, port=PORT):
    notify('fail test', msg_type='info')
    local('''python -c "raise Exception('foobar')"''')
    notify('script ran successfully', msg_type='success')  # this will never run because the function above crashed


@task
@alert_on_fail
def pass_test(host=HOST, port=PORT):
    notify('pass test', msg_type='info')
    local('whoami')
    notify('script ran successfully', msg_type='success')


def notify(msg, **kwargs):
    # DISREGARD THIS
    print 'sent to slack:', msg

    Output:

    $ fab fail_test
sent to slack: fail test
[localhost] local: python -c "raise Exception('foobar')"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
Exception: foobar

Fatal error: local() encountered an error (return code 1) while executing 'python -c "raise Exception('foobar')"'

Aborting.
sent to slack: something went wrong: Traceback (most recent call last):
  File "/private/tmp/fabfile.py", line 21, in decorated
    return func(*args, **kwargs)
  File "/private/tmp/fabfile.py", line 34, in fail_test
    local('''python -c "raise Exception('foobar')"''')
  File "/usr/local/lib/python2.7/site-packages/fabric/operations.py", line 1198, in local
    error(message=msg, stdout=out, stderr=err)
  File "/usr/local/lib/python2.7/site-packages/fabric/utils.py", line 347, in error
    return func(message)
  File "/usr/local/lib/python2.7/site-packages/fabric/utils.py", line 53, in abort
    sys.exit(msg)
SystemExit: local() encountered an error (return code 1) while executing 'python -c "raise Exception('foobar')"'


Fatal error: exiting error!!

Aborting.
exiting error!!

    and:

    $ fab pass_test
sent to slack: pass test
[localhost] local: whoami
buzzi
sent to slack: script ran successfully

Done.

    You'll notice that the functions are now "easy" to read, they are "simple", all the error handling code has been moved over somewhere else.