Supervisor (python supervisord) email alerts

The program supervisor written in python is used to supervise long running processes. In case that a long running process will stop (crash) supervisor will detect it and will restart it, you will get entries into the log files however unless you have a log aggregation tool or you login into the server or have some other monitoring tool you will not know that your process has crashed.

However there is hope :) - you can setup an event listener into supervisor which can email you in case that a process has exit. To do so you will need to install a python package superlance This is how the setup is done.

# install superlance
$ sudo pip install superlance  # if you don't have pip install try easy_install 

# configure supervisor to send events to crashmail

$ sudo vim /etc/supervisor/supervisord.conf  # change according to your setup

[eventlistener:crashmail]
command=crashmail -a -m root@localhost
events=PROCESS_STATE_EXITED

$ sudo supervisor stop && sudo supervisor start
# done :)

In the example above if a process will crash (exit) an event will be sent to crashmail which in turn will email to root@localhost - of course you can change the email address, crashmail uses actually sendmail to send email (postfix and qmail come with a sendmail like program so no worries).
Also the email alert will be sent out for any program that crashed but if you want to filter out you can choose just the program you want by specifying -p program_name instead if -a, for more info you can see Crashmail section on the superlance docs.