Friday, December 26, 2014

Python web application deployment, my setup


There are many options to deploy Python web applications or WSGI-compliant applications in general. Using Apache or Nginx is a popular choice.

Here I will use the following software, which happens to be 100 % Python:

  • Flask, a Python framework for web applications, but other WSGI-compliant applications should work just as well.
  • Tornado, a scalable non-blocking web server written in Python
  • Supervisor, a system for controlling process state similar to Linux's init. Configuring a standalone daemon is another option, although less flexible.

Step 1

Make sure your Flask application is enclosed in a function.
It's basically something like this:

def create_app(any_parameters_you_like):
    app = Flask(__name__)
    app.config['SERVER_NAME'] = 'johndoe.com:1234'
    return app

Outside of any method, you can write something such as:

app = create_app()

There are several things to note here:
  • You need to specify the URL and port that will be used to access the web application.
  • By default, the built-in server started on the last line only listens for local connections, "0.0.0.0" will make your application accessible from the outside (LAN and any other network).

Step 2

Create a file my_tornado.py anywhere you like, for instance in the same directory than the app. Here we suppose the create_app() function is in a file named my_module.py.

The content of this file for servers requiring SSL connections is as following


#!/usr/bin/env python
import sys 
import os
sys.path.insert(0, os.path.dirname(__file__))
sys.path.insert(1, '/any/other/directory/to/add/to/your/path')

from tornado.wsgi import WSGIContainer
from tornado.httpserver import HTTPServer
from tornado.ioloop import IOLoop
from tornado.web import FallbackHandler, RequestHandler, Application
import tornado.web


import my_module
app = my_module.create_app()

application = Application([
    (r"/my_app/.*", FallbackHandler, dict(fallback=WSGIContainer(app))),
    ])  

http_server = HTTPServer(application, ssl_options = { 
    "certfile": "/etc/ssl/certs/my_server.crt",
    "keyfile": "/etc/ssl/certs/private/my_server.key",
})
http_server.listen(1234)

print 'Ready.'

IOLoop.instance().start()


As explained, in this example SSL is activated. Also a non standard port (5000) is used. You can change the options and run in insecure mode. For SSL you need to roll your own certificates.

Launch the Python file. The app should be running fine.
Because I was launching this on a Raspberry Pi it took several seconds to start. Thanks to the "Ready" message you will know when the server has completely started.
See if "/yoursite" is accessible.

Step 4

Make it start at boot using init.d, xinetd or other techniques. Here I am using Supervisor.
You can optionally enable the web interface in the config /etc/supervisord.conf which will look like this:


Add a new section to the file:

[program:my_app]
command = /home/pi/my_webapp/my_tornado.py
startsecs=10
stopsignal=INT
user=pi

Change the appropriate settings if necessary. Note stopsignal is the signal sent to the process so it stops itself, it's the first argument of "kill". Sending SIGINT has the same effect as pressing Ctrl+C after Tornado has been started.
startsecs is a hint for Tornado : the service is considered running if the process is still up after 10 seconds.
As explained you can also use a standalone daemon, but Supervisor is more flexible and doesn't need any configuration and fiddling with boilerplate code.

Ensure Supervisord starts with the system.
UPDATE:  I had a problem recently because one of the services declared with supervisord required another service (started with a regular init.d script) to work. The workaround is to start supervisord after everything elseby appending "$all" at the end of #Required-Start in the init.d script as described at the bottom of this page.

That's it, we're done!

Saturday, December 13, 2014

Tutorial: Linux firewall and IPTV (e.g. Swisscom TV)



Swisscom TV like any IPTV network really, relies on two technologies to deliver its content to its users:
  • Multicast adressing
  • IGMPv3
  • (IGMP snooping)
 I probably don't need to explain what Multicast is. You should only know that this is what has been causing network administrators many headaches!
Netflix and YouTube don't use Multicast but Unicast instead, only YOU are receiving a particular stream. This is the difference between Video-on-Demand and IPTV.
Speaking of which, when you are watching shows from the past days on Swisscom TV, it's actually sending Unicast packets.

IGMP allows clients (your TV or Set Top Box) to register itself for multicast traffic. This is basically the protocol that is used to select the TV channel, so that the STB can receive it.

IGMP snooping prevents your home network (possibly even the Wi-Fi network) to be flooded by the video stream packets even though only one TV
 needs them. This is a Level 2 technology implemented in switches.
We won't deal with IGMP snooping here, only with the first two.

This tutorial is based on Debian 7.7 but can work anywhere where IGMPProxy can run.
  1. Download IGMPProxy, compile and install it.
  2. Swisscom seems to use the following subnets:

    213.3.72.0/24
    195.186.0.0/16

    We'll add them to the igmpproxy configuration in /usr/local/etc/igmpproxy.conf. I have it configured like this:

    quickleave

    phyint eth0 upstream  ratelimit 0  threshold 1
            altnet 192.168.1.0/24 # see below
            altnet 213.3.72.0/24
            altnet 224.0.0.0/4
            altnet 195.186.0.0/16

    phyint eth1 downstream  ratelimit 0  threshold 1
            altnet 172.16.123.128/29 # see below

    phyint lo disabled


    My configuration (adapt to your needs):
    eth0 is the WAN interface
    eth1 is the LAN interface

    192.168
    .1.0/24 is the subnet between the Swisscom modem and the firewall
    172.16.123.128/29 is the subnet I dedicated to STB.
    224.0.0.0/4 identifies multicast addresses. It goes from 224 to 240, hence the /4 network mask.
    See the documentation for the meaning of quickleave.
  3. Configure the firewall to let said subnets in the INPUT chain. Allow FORWARDing between both interfaces.
    (These instructions are vague on purpose. If you are running a firewall you obviously know your policies and rules better than anyone!
    If you want to be precise you will have to allow for the IGMP protocol with the parameter -p igmp of IPTables for instance).
  4. Launch igmpproxy in debug mode, as root:
    igmpproxy -d -v /usr/local/etc/igmpproxy.conf

    See if you can watch TV. Look at the addresses. Try to add them to your configuration, relaunch the proxy and see if it works.
  5. Schedule igmpproxy to start at boot.
    You can use this script and follow the instructions at the bottom of the page. (Discard the explanations in Russian in-between ;-))
  6. Happy TV watching!


    Here's what the traffic looks like while I've been playing with this:
    (This is with pnp4nagios and the stat_net plugin, which is based on /proc/net/dev.)

I am actually quite surprised by this. It seems to need only 3 MBit/s for one TV (and not recording a second channel). I would have expected from 2 to 3 times more bandwidth usage.
I tried nload just to make sure:

Same observation!
(Possibly because /sys/class/net/eth0/statistics/rx_bytes which nload uses, is filled by the same kernel module as /proc/net/dev, in which case this should be accurate as these are the actual frames read by the Linux kernel.)


If you are getting continuous messages in your syslog:

I tried everything. I can either stop the messages or watch TV. For some reason, the STB sends "upstream" IGMP packets. It's very strange and doesn't make much sense. The IGMP proxy doesn't "proxies" them and the TV runs smoothly.
I figured I could safely ignore these. You should also note that IGMPProxy is a very young project (version 0.1).

I added this near the beginning of /etc/rsyslog.conf::msg, contains, "The source address 172.16.123." ~

and restarted the rsyslog daemon. The messages are finally gone. Another solution is to use another file for IGMPProxy logs.



Keywords: linux, unix, firewall, shorewall, bsd, iptables, netfilter, bluewin tv, swisscom, iptv, multicast, 

Getting instant notifications on your smartphone or tablet from Nagios


If you are a network administrator or manage a production system of some sort, you might be interested to be notified as soon as possible in the event of problems.

Nagios is a great tool to watch your hosts and services and even though it's kind of 1990-ish it still does a very good job and stays the most used tool for this task.

It can natively send e-mails. That's great, but I don't know about you, I get more than 40 e-mails a day, and even with SaneBox and other tricks it's easy to miss e-mails and it can literally take days until I notice one that was more important than the others.

Pushover provides instant notifications on handheld devices such as Android, iPhone, and iPad using "push" technology (instead of polling).
It has many advantages:

  • It uses a REST API, meaning you can use it in all scenarios you ever dreamt of.
  • They provide clients for smartphones, tablets and they even support OS X Growl notifications.
  • It's inexpensive at only $5 USD.
  • You don't need to run or install any software on your servers.

Here is a tutorial on how to use it with Nagios, based on Debian 7.7 stable, as of December 2014:

  1. Create an account on Pushover. Mark down your personal key.
  2. Create an application. Mark down the application key.
    I named mine "Nagios" but you can choose anything you like.
  3. Set up your account on one of your devices.
  4. Download Jedda Wignall's script in your plugins directory with your root account:
    wget -O /usr/lib/nagios/plugins/notify_by_pushover.sh https://raw.githubusercontent.com/jedda/OSX-Monitoring-Tools/master/notify_by_pushover.sh
  5. Make it executable:
    chmod +x /usr/lib/nagios/plugins/notify_by_pushover.sh
  6. Create a Nagios command to run it. I like to have a /etc/nagios3/conf.d/custom_handlers.cfg file, but you can name it with any name you want as long as it is in that directory. Copy the following coutent to it:

    define command {
        command_name        handler_pushover
         command_line       $USER1$/notify_by_pushover.sh -u $ARG1$ -a $ARG2$ -t "$ARG3$" -m "$ARG4$"
    }


    $USER$1 stands for /usr/lib/nagios/plugins and $ARGn$ will be replaced with arguments set after the command-name following "!" in the next step.
  7. If you want every host to report a failure (and supposing you used the template everywhere), modify /etc/nagios3/conf.d/generic-host_nagios2.cfg, and add the following line anywhere inside the host definition:

    event_handler                   handler_pushover!PERSONAL_KEY!APP_KEY!Nagios!$HOSTNAME$ : $HOSTSTATE$
    Replace the personal key and the application keys with the ones obtained in step 1 and 2.

    "Nagios" here is the title that will appear in the notification, it doesn't need to match the application name and it can include spaces.
  8. The last parameter of the command (everything after the last "!") is the message. Here I simply chose to show the hostname and the state. The variables enclosed in dollar signs can be found in the Nagios manual.
  9. If you want every service to report a failure (and supposing you used the template everywhere), modify /etc/nagios3/conf.d/generic-service_nagios2.cfg, and add the following line anywhere inside the service definition:

    event_handler                   handler_pushover!up4evfJ7bkP43M5mLvGZ24Gp1aoM14!a6fLnAFT2rNMCXeEeogbghbLbV1o9m!Nagios!$HOSTNAME$ / $SERVICEDESC$ ($SERVICESTATE$) : $SERVICEOUTPUT$

    This will print the status of the service as well as the line shown in the web interface.
    You can include modify "Nagios" (title of the notification) and everything after the last ("!") with macros from the Nagios manual.
  10. Reload Nagios:
    service nagios3 reload
  11. Provoke a "warning" or "critical" message in Nagios. You should receive a notification as soon as the problem is discovered by Nagios.
Life is great!

Now, there may be better ways to do this. You should probably use the notification mechanism from Nagios instead. The handler is supposed to "handle" the problem, not report it. As a result I had to put retry_interval 120 in some of my service definitions to avoid getting the same message again and again.

Tell me what you think and how it worked out for you!