Watch a folder and send files by e-mail

My printer is able to convert faxes to PDF documents and save these documents on a network folder.
From there I could do many things. But I need to watch for new documents in a folder.

But first let's setup the Windows / Samba network folder.

Shared folder with Samba

The comment is shown in the file explorer. browseable tells if the folder is listed, and writeable is useful to write and (more interestingly in this case) delete files.
valid users can contain users or groups (prefixed with @) that have the permission to read and (if applicable) write to the folder.
create mask is required because the folder is meant to be shared to a group instead of belonging to a single user. Use 0750 if you only want the creator of the file to be able to remove the file.

I created a sambausers group on my machine and put the appropriate users in it.. Remember you need a system account for each user, and that you need to configure each account through smbpasswd. By the way you can configure Samba with a database or LDAP if you like.

You need to make a new directory and let the sambausers group own it and have read / write / execute (chmod g+rwx) on it. The execute permission on the folder is needed so you can create files in the folder.

Restart Samba. Check you can access the folder from your file manager with the appropriate credentials.

Watch directories on your file system

One great way to do this is the incron daemon. It lets you setup cron-like tasks to execute a shell script or any executable program whenever a change is detected in the folder.

Install incron on your system.

Login with a user with permissions to read files from the shared folder on your Linux box. You can sudo -i -u theUser.

Open man incrontab in one terminal window.

In another window, fire up incrontab -e to edit the tasks associated with the user.

I have mine configured with /srv/shr/IncomingFax IN_CREATE /home/myuser/somescript.sh

You can watch for other events, just read the manual you just opened!

Now everytime a file is created in the IncomingFax folder the script will be executed.

E-mail the file just added

Here is an example of the shell script that I use. It might not be the smartest way to do what it does (particularly because the information of the file added is lost in the process because of how incron works)

The script holds its "state" by using a ".lastfile" because it might happen that the script is executed several times with the same document. I don't know why it does that, I think it's an issue with the printer. You might not need to do that.

Also I happen to have a log file to log what happens with the script. You might not want that either. What you might like though is to verify the extension of the file. Note that the filesystem is case-sensitive so ".PDF" files won't be matched.

Make sure you have mutt installed. It's a bit complicated to send attachments. Sendmail is not enough. Note you can attach several files at once. The argument to the echo command is the message body, and what comes after "-s" is the message subject.

I purposedly let this example in French as a reminder that it is safer to avoid any other encoding than pure ASCII. There has to be a way to cope with UTF-8 but I didn't have time to investigate that issue.

This script has (at least) one limitation. I should actually rewrite it to make it more robust. I am assuming it takes less than 5 seconds for the printer to transfer the file. After 5 seconds I send the file by e-mail. There would be ways to know if the transfer is finished:

• Use a program to read the file. That program probably knows if it's valid.
• Wait until the file size has been constant for some time. Then we can assume the transfer is finished.
• If there are a lot of transfers, we can assume the before-last file was completely transferred when we detect a new file.

There you go. We can now look at a folder, filter files by extension, and send the new file by e-mail.

Secure e-mail and encrypted files

In the last article I explained that you should encrypt your traffic. The article was focused on web browsing.

With emails it's a another story.

Web traffic is like a phone conversation that can be tapped. Email is like regular mail. You don't even have a direct connection with the recipient.

Like regular mail, emails are "routed". You send them from your computer to your email provider, then your e-mail provider sends it to the mail server responsible for the recipient domain, and from there it may possibly be sent again to a mailbox server (e.g. IMAP server).

While it's always a good idea to send your e-mails using a secure SMTP connection, your email provider might not do so with the recipient email server.

Furthermore, while web traffic is volatile, emails are stored on hard drives and are more likely to contain sensitive information. It's very easy for say anyone with access to these servers (such as your boss, the email providers and the government) to look at your emails.

The solution to this problem can actually be used for almost anything and is not new technology at all.
This technology is Asymmetric cryptography. (It's asymmetric because the encryption and decryption keys are not the same. The decryption key can be sent to anyone and are usually published on opened websites. In fact they are not really encryption or decryption key, we'd better call them private and public keys.)

It can take two forms: PGP (with implementations such as GnuPG), the preferred solution,  or digital signatures / certificates (known as S/MIME).

I will list the advantages of using asymmetric cryptography, and a few drawbacks.

Pros

• Only the sender and the recipient can look at the data. For people in the middle, it's just gibberish.
• You encrypt the data for one recipient only. Even if someone else has the decryption key, it would be useless because the e-mail was encrypted with their public key so only them can decrypt it with their private key and your public key.
• The recipient can verify the identity of the sender and know for sure only someone with the proper private key and the password to that key could have sent the email.
• Same thing, but it's possible to verify while not encrypting if the data is not sensitive except you want to make sure you created it. It's useful for software so you don't download a counterfeit copy with security holes.
• Can be used to encrypt anything, any sequence of bytes, including files or your whole home directory.

Cons

• That you already knew: if you use a weak password for the private key or use it on more than one application / website, then you achieve no security.
• The recipient needs to use PGP or S/MIME so that the message gets encrypted and they can decrypt / verify it. So your recipient needs to be as tech-savvy as much as you are.
• Even though there are some solutions for mobile devices and webmails, the solutions are never as good as say the Enigmail extension for Thunderbird.
• It's very easy to lost your private key but it's risky to make copies. See the problem?

The insecure Web and the Dark Net

Prison guards are looking at your Internet traffic

I cannot stress enough the fact that you should encrypt your traffic as much as possible.

This includes web browsing, e-mail, instant messaging and so on.

You see, unencrypted traffic is like letters sent from prisons. People are looking at it. It can be the police monitoring criminal activities, but also the police monitoring the regular John Doe that did nothing wrong, your Internet provider or your boss.

It's how the Internet works. Packets gets routed from router to router until they reach their destination. Along the way anyone can look at the data if it is sent in clear.

How to protect against these spies

HTTPS

There are several ways (that can be combined) to make your browsing more secure.

Number 1 is HTTPS. While "sniffers" can still know who you are contacting (and block the communication), they don't know what your are sending.

The traffic is encrypted so that only your computer and the final server (such as your banking web application) can see the data in clear.

BEWARE: if you are using a company computer, there are chances that they installed their own certificate and that what you believe is encrypted is actually not. There are ways to check this.

The problem with HTTPS is that not all websites offer it.

Some sites are HTTP-only. You will need something more complicated, but you are basically screwed.

Some sites are HTTPS-only. These are the best kind.

Some sites offer HTTPS but you have to ask for it explicitely. For these sites I strongly suggest using HTTPS Anywhere, a browser extension that connects you to the HTTPS version when one is available.

Proxies

While proxies won't encrypt unencrypted traffic, they can be effective when some firewall is blocking you access to a website. You see, the biggest concern for your network provider, for instance your school WiFi network or the McDonald's WiFi is not really that you are visiting a website they don't like. It's because they know the government is spying on them, and if the spies find something then the company is accountable and will in the newspapers or maybe even involved in a trial.

The idea is that the connection must seem to come from another point on the Internet. This can be achieved using proxies. There are lists of proxies that you can use for free as well as paid solutions providing a greater bandwidth.

However they are easy to detect because then all your traffic will go to one IP address, and some proxies even show your real IP address to the receiver.

If you have a SSH connection to a remote host, you can simply use it to redirect your traffic:

ssh -L 1234:the-website.com:80 user@host

Then you can access the website with http://localhost:1234/

But that sometimes doesn't work because many websites use fully qualified links and use other domains.

Tor

Tor is a very complex network / technology that is not only secure but hides your traffic, making it hard to detect, provided you follow other rules and pay attention to your Internet activity.

The same goes for...

I2P

Another technology, probably even better as it uses different paths for the packets sent and received and this network path can change for every packet if needed.

Tor is the older project. Both protocols / networks are associated with the DarkNet. Until recently a website known as Silk's Road on Tor made millions of drug transactions. Buying weed was as simple as ordering the new iPhone online. You would receive your package at home no question asked in a discrete typical business package.

And this is nothing compared to other things you can do on the dark net: hire hackers, hitmen, buy guns, fake ID cards, stolen credit cards and other info.

Because of course these transactions would be detected with PayPal for instance, all these websites use the Bitcoin peer-to-peer money. (It's peer to peer but there is a log file storing all transactions, but even if the user IDs are supposed to be anonymous, it isn't hard to analyze the data and associate people with the transactions.)

Don't take me wrong, I think Bitcoin is a great idea! But it's a shame it's used like this though...

But they are very effective because the connection is untraceable and encrypted along the way (except if the website at the end is on the regular open Internet and doesn't use HTTPS).

To use Tor all you need is to download the Tor browser.

In another article I will explain you how you can achieve secure e-mail.

VirtualBox headless server setup

UPDATE: For day-to-day operations, you can use these handy aliases.

It might be interesting in some scenarios to have a virtual machine running on a headless server.
If you don't have thousands of dollars to spend on VMware vSphere or similar commercial solutions you can do it for free, if you don't mind the command-line.
And once you've created one you can simply clone it and it's not much longer than clicking buttons on a web interface...
Also, except for Oracle's implementation of the VRDP server (a cousin of the VNC protocol), the solution is entirely based on open source software if that is something important for you.

If you are really allergic to the command-line or you want to let sane human people use your "VM server", you might want to give a look at unofficial solutions such as

• phpVirtualBox (web-based, includes a RDP client as a Java applet)
• RemoteBox (heavy client on your workstation, makes it easier to connect with RDP)

Both solutions try to imitate the official GUI application, so you should be feeling at home with either of them. Note phpVirtualBox might be insecure and dangerous. You should restrict its access using HTTPS and Basic Authentication. See also: Apache's "allow from ..." parameter to restrict by the client IP address or subnet.

My setup

Host: Debian 7.7, 64-bit
Guest: Windows 8.1, 64-bit. It will be named "Windows8.1_x64". The setup is similar for other operating systems.
Packages: VirtualBox 4.3 with guest additions, DKMS

VirtualBox package

Although VirtualBox is available through official Debian repositories, the version there is usually out-of-date and it can make things complicated with newer versions of guest OSes or it might be hard to find older versions of the VirtualBox extension pack.

As root (or using a combination of tee and sudo):

# echo "deb http://download.virtualbox.org/virtualbox/debian wheezy contrib" > /etc/apt/sources.list.d/virtualbox.list
# wget -q https://www.virtualbox.org/download/oracle_vbox.asc -O- | apt-key add -
# apt-get install virtualbox-4.3 dkms

Note: When I tried it APT was so slow I downloaded the file with a web browser directly and installed it with dpkg.

DKMS

Make sure you installed dkms as shown in the previous section. Otherwise you will have to re-compile and re-install the VirtualBox kernel module after every update of your kernel.

DKMS is a general-purpose package maintained by the "Dynamic Kernel Modules Support Team" at Debian. It doesn't come from Oracle / with VirtualBox.

VirtualBox extension pack

The extension pack is required for many features such as the VRDP server (VRDP is similar to VNC) and the guest tools.

Download the pack from the VirtualBox website: https://www.virtualbox.org/wiki/Downloads
The link is titled "All supported platforms". Once you've read the license, you can download with wget:

$ wget "http://download.virtualbox.org/virtualbox/4.3.SOMETHING/Oracle_VM_VirtualBox_Extension_Pack-4.3.DO_NOT_COPY_THIS_LINK.vbox-extpack"

DO NOT COPY THIS LINK. This command is only given for reference. Copy the link from the website instead.

Install the extension pack as root:

# VBoxManage extpack install *.vbox-extpack

Creating a naked virtual machine

The rest of the tutorial can and should be done not as root but as a regular user. It doesn't even seem like membership to the "vboxusers" group is required.

$ VBoxManage createvm --name Windows8.1_x64 --register

or

$ VBoxManage createvm --name Windows8.1_x64
$ VBoxManage registervm "$HOME/VirtualBox VMs/Windows8.1_x64/Windows8.1_x64.vbox"

Both commands will create the directory above. I advise you to put all your files there, except maybe for the installation disk image.

Let's set a few options on the VM:

$ VBoxManage modifyvm Windows8.1_x64 --ostype Windows8_64 --memory 4096 --vram 128 --nic1 nat --nictype1 82540EM

The list of os types can be obtained using VBoxManage list ostypes.

The (RAM) memory is given in MB. VRAM stands for "Video RAM" (I think this parameter's default value is 128 MB). Note the NIC type is "8254 ZERO EM". I need to research on this but although 82540EM is compatible with most guest OSes,  

NIC type virtio is a better choice if the guest OS supports it as it offers better performance.
UPDATE: Windows 8.1 is not compatible with virtio.

Bridged mode

Bridged mode is useful if you want to communicate with the VM from any device on your network.
This mode is particularly useful for SSH connections.
The default NAT mode hides the VM from the network, so it looks like the network traffic is coming from your host machine.
To enable it, type VBoxManage modifyvm Windows8.1_x64 --nic1 bridged --bridgeadapter1 eth0.
eth0 is the interface connected to the network you'd like to access the VM from.
Note modern systems bundled with systemd now use a predictable naming scheme for interface names and the name will be usually something like enp5s0 (ethernet) or wlp3s0 (wireless).
Either way try ip addr if you are not sure...

Hard drive

As you may or may not know, hard drives live independently from virtual machines in VirtualBox, hence the existence of the "Virtual Media Manager" in VirtualBox's GUI.
For this reason we need to create the drive first and attach it to the virtual machine later.
This feature is also nice when you wish to play with a virtual RAID setup by "disconnecting" drives.

I suggest installing the hard drive alongside the other files of the VM:

$ cd VirtualBox\ VMs/Windows8.1_x64

To create a 50 GB hard drive:

$ VBoxManage createhd --filename Windows8.1_x64.vdi --size 50000

Note 1: The file is created in the current directory.

Note 2: The file won't use 50 GB until the virtual disk is actually full. Remember it can be full even if Windows still shows available space due to fragmentation on FAT and NTFS partitions.

When copying virtual hard drives I suggest defragmenting it first and then compacting the VDI (Virtual Disk Image) file with VBoxManage modifyhd ...filename... --compact

To attach the drive to the virtual machine we first need to create a SATA controller:

$ VBoxManage storagectl Windows8.1_x64 --name SATA --add sata --controller IntelAhci --bootable on

Then we can attach the disk:

$ VBoxManage storageattach Windows8.1_x64 --storagectl SATA --port 0 --device 0 --type hdd --medium "$(pwd)"/"Windows8.1_x64.vdi"

Important: The last argument must be an absolute path (you can safely copy and paste the last command if you copied and pasted the first two commands of this section).

Installation medium

Locate your ISO image of the Windows installation disk.

For some reason optical disk drives need to be attached to an IDE controller. Let's add one:

$ VBoxManage storagectl Windows8.1_x64 --name IDE --add ide --controller PIIX4

Then we can attach the image to the controller:

$ VBoxManage storageattach Windows8.1_x64 --storagectl IDE --port 0 --device 0 --type dvddrive --medium /absolute/path/to/your/windows8_disk.iso

Replace the ISO image path with emptydrive to eject the medium.

Make the system boot on the DVD:

$ VBoxManage modifyvm Windows8.1_x64 --boot1 dvd

Configure the VRDP port (optional, 3389 by default)

If you are running several virtual machines at the same time you want to change the VRDP port of the VM with

$ VBoxManage modifyvm Windows8.1_x64 --vrdeport 5012 --vrde on

The option --vrdeaddress 192.168.0.1 (or whatever) is also available and might be useful if you don't use a firewall.

Windows 8 specific configuration

Windows 8 will start but the installer will complain with obscure error codes (0x00000225) with the options set to far. We need a few more:

$ VBoxManage modifyvm Windows8.1_x64 --pae on --ioapic on

In fact you probably want to enable PAE/NX with any guest OS.

There is a problem with the mouse cursors from the Aero theme. To solve it use the "(None)" theme with no shadow.

Starting the VM

It is now time to start the VM. Although you can also use VBoxHeadless that shows errors and warnings, I prefer the following command that forks and runs the VM as a daemon process:

$ VBoxManage startvm Windows8.1_x64 --type headless

Connecting from your workstation

Note for Linux guest OSes: If you want to install a Linux guest you won't be able to connect to it with SSH (some distributions enable this mode of installation) as long as the NIC type is set to NAT. You can change it to another supported mode and configure the interface accordingly or connect to the virtual machine (which will be needed if you intend to run an installer) using the same method as for the Windows installation described here.

Make sure the VRDP server is started and listening for connections on the appropriate port:

# netstat -tlpn | grep VBoxHeadless

Connect from your workstation using your preferred RDP client, for instance for the KDE Remote Desktop Client (krdc):

$ krdc rdp://your-server-ip:3389

Press "Cancel" when asked for a password.
Otherwise install and run rdesktop:

$ rdesktop -a 16 -N your-server-ip:3389

Put IPv6 addresses between brackets [2a00:...]:3389

The Windows 8.1 installer uses a resolution of 1024x768. rdesktop will select it automatically.

rdesktop worked fined on Ubuntu 14.04 but on Arch Linux I could only use FreeRDP:



CAVEAT: My mouse pointer was off. This is corrected with the guest additions but of course you can only install them after Windows... Try to put the pointer at one of the edge of the screen to make matters marginally better. You can also use the keyboard.

Installing the Guest additions

The guest additions is simply a DVD that comes with the extension pack.

But before loading the drive (I already showed you the technique, remember?), power off the machine, either with one of VBoxManage controlvm arguments or from the guest OS.

Afterward we can take a look at the options set on the VM:

$ VBoxManage showvminfo Windows8.1_x64

For instance, near the line "IDE (0, 0)" you will see that the Windows installation DVD was unmounted.

Let's mount the Guest additions DVD:

$ VBoxManage modifyvm Windows8.1_x64 --dvd /usr/share/virtualbox/VBoxGuestAdditions.iso

and start the VM with the method shown earlier.

Install the guest additions from the guest OS by locating the DVD drive and firing up the installer.

Reboot.
The "mouse bug" should be fixed now.

Performance optimizations

If you use KRDC, select 16 bit color depth and select "Connection type: Broadband".

Disable the visual effects using this method:
http://www.thewindowsclub.com/disable-visual-effects-windows
My favorite options:

Windows still looks natural but everything is much smoother.

You can also use a smaller resolution and less colors (8 bit instead of 16 or 24) in your RDP client.

Choosing a solid color desktop background also speeds things up.


Quick startup and shutdown

To make Windows boot or shutdown quickly, do not boot it up or shut it down ;-)

Use Virtualbox "save state" feature instead:

$ VBoxManage controlvm Windows8.1_x64 savestate

$ VBoxManage startvm Windows8.1_x64 --headless

You can also "reset", "poweroff" or "discardstate" if appropriate. These are dangerous options similar to pulling off the plug of a physical computer. Use them only if the guest OS didn't power off the machine after shutdown (80-90s style) or the OS froze.
You can also "pause" or "resume" the guest OS temporarily.

I always "savestate" a Windows guest OS when I am not using it because I've noticed Windows or one app or another is writing and reading on the disk all the time. This reduces performance of the host OS and wears the drive, shortening its life.

To shutdown the system cleanly, supposing there is no unsaved work (if there is, the shutdown will be cancelled):

$ VBoxManage controlvm Windows8.1_x64 acpipowerbutton

Security

See the VirtualBox manual link at the end of this tutorial to set up a proper authentication method.
Here is a quick how-to:

$ VBoxManage setproperty vrdeauthlibrary "VBoxAuthSimple"

$ VBoxManage modifyvm Windows8.1_x64 --vrdeauthtype external

$ VBoxManage setextradata Windows8.1_x64 "VBoxAuthSimple/users/YOUR_RDP_USERNAME" $(VBoxManage internalcommands passwordhash "YOUR_RDP_PASSWORD" | awk '{print $3;}')

Change the user and password with one of your choice.
krdcwill ask for your password, but with rdesktop you need to provide it on the command-line:

$ rdesktop -a 16 -N YOUR_SERVER_IP:3389 -u YOU_RDP_USER -p YOUR_RDP_PASSWORD -r sound:off -x l -z -P

(Here I used a few other options such as optimization for LAN connections, compression, no sound...)
For FreeRDP:

$ xfreerdp /v:YOUR_SERVER_IP:3389 /u:YOUR_RDP_USER /p:YOUR_RDP_PASSWORD -toggle-fullscreen /bpp:16 /w:1808 /h:1017 +auto-reconnect

(Here the options are: 16 bit graphics (65536 colors), and a custom resolution. "+auto-reconnect" is supposed to re-connect automatically, but to be honest it doesn't really work. YMMV...)

With Windows' Remote Desktop Connection utility, you need to click "Show options" and then the option about credentials. After you've connected once, your credentials will be saved.

Cool stuff

There are plenty of cool stuff such as Remote USB that you can do. Make sure to check out the VirtualBox user manual.

Your Feedback counts!

Please comment below if you have any remark, question or suggestion or if you liked this tutorial.

Python web application deployment, my setup

There are many options to deploy Python web applications or WSGI-compliant applications in general. Using Apache or Nginx is a popular choice.

Here I will use the following software, which happens to be 100 % Python:

• Flask, a Python framework for web applications, but other WSGI-compliant applications should work just as well.
• Tornado, a scalable non-blocking web server written in Python
• Supervisor, a system for controlling process state similar to Linux's init. Configuring a standalone daemon is another option, although less flexible.

Step 1

Make sure your Flask application is enclosed in a function.

It's basically something like this:

def create_app(any_parameters_you_like):

    app = Flask(__name__)

    app.config['SERVER_NAME'] = 'johndoe.com:1234'

    return app

Outside of any method, you can write something such as:

app = create_app()

There are several things to note here:

• You need to specify the URL and port that will be used to access the web application.
• By default, the built-in server started on the last line only listens for local connections, "0.0.0.0" will make your application accessible from the outside (LAN and any other network).

Step 2

Create a file my_tornado.py anywhere you like, for instance in the same directory than the app. Here we suppose the create_app() function is in a file named my_module.py.

The content of this file for servers requiring SSL connections is as following

#!/usr/bin/env python

import sys 

import os

sys.path.insert(0, os.path.dirname(__file__))

sys.path.insert(1, '/any/other/directory/to/add/to/your/path')

from tornado.wsgi import WSGIContainer

from tornado.httpserver import HTTPServer

from tornado.ioloop import IOLoop

from tornado.web import FallbackHandler, RequestHandler, Application

import tornado.web

import my_module

app = my_module.create_app()

application = Application([

    (r"/my_app/.*", FallbackHandler, dict(fallback=WSGIContainer(app))),

    ])  

http_server = HTTPServer(application, ssl_options = { 

    "certfile": "/etc/ssl/certs/my_server.crt",

    "keyfile": "/etc/ssl/certs/private/my_server.key",

})

http_server.listen(1234)

print 'Ready.'

IOLoop.instance().start()

As explained, in this example SSL is activated. Also a non standard port (5000) is used. You can change the options and run in insecure mode. For SSL you need to roll your own certificates.

Launch the Python file. The app should be running fine.
Because I was launching this on a Raspberry Pi it took several seconds to start. Thanks to the "Ready" message you will know when the server has completely started.
See if "/yoursite" is accessible.

Step 4

Make it start at boot using init.d, xinetd or other techniques. Here I am using Supervisor.

You can optionally enable the web interface in the config /etc/supervisord.conf which will look like this:

Add a new section to the file:

[program:my_app]

command = /home/pi/my_webapp/my_tornado.py

startsecs=10

stopsignal=INT

user=pi

Change the appropriate settings if necessary. Note stopsignal is the signal sent to the process so it stops itself, it's the first argument of "kill". Sending SIGINT has the same effect as pressing Ctrl+C after Tornado has been started.

startsecs is a hint for Tornado : the service is considered running if the process is still up after 10 seconds.

As explained you can also use a standalone daemon, but Supervisor is more flexible and doesn't need any configuration and fiddling with boilerplate code.

Ensure Supervisord starts with the system.
UPDATE:  I had a problem recently because one of the services declared with supervisord required another service (started with a regular init.d script) to work. The workaround is to start supervisord after everything elseby appending "$all" at the end of #Required-Start in the init.d script as described at the bottom of this page.

That's it, we're done!

Tutorial: Linux firewall and IPTV (e.g. Swisscom TV)

Swisscom TV like any IPTV network really, relies on two technologies to deliver its content to its users:

• Multicast adressing
• IGMPv3
• (IGMP snooping)

 I probably don't need to explain what Multicast is. You should only know that this is what has been causing network administrators many headaches!
Netflix and YouTube don't use Multicast but Unicast instead, only YOU are receiving a particular stream. This is the difference between Video-on-Demand and IPTV.
Speaking of which, when you are watching shows from the past days on Swisscom TV, it's actually sending Unicast packets.

IGMP allows clients (your TV or Set Top Box) to register itself for multicast traffic. This is basically the protocol that is used to select the TV channel, so that the STB can receive it.

IGMP snooping prevents your home network (possibly even the Wi-Fi network) to be flooded by the video stream packets even though only one TV
 needs them. This is a Level 2 technology implemented in switches.
We won't deal with IGMP snooping here, only with the first two.

This tutorial is based on Debian 7.7 but can work anywhere where IGMPProxy can run.

1. Download IGMPProxy, compile and install it.
2. Swisscom seems to use the following subnets:
   
   213.3.72.0/24
   195.186.0.0/16
   
   We'll add them to the igmpproxy configuration in /usr/local/etc/igmpproxy.conf. I have it configured like this:
   
   quickleave
   
   phyint eth0 upstream  ratelimit 0  threshold 1
           altnet 192.168.1.0/24 # see below
           altnet 213.3.72.0/24
           altnet 224.0.0.0/4
           altnet 195.186.0.0/16
   
   phyint eth1 downstream  ratelimit 0  threshold 1
           altnet 172.16.123.128/29 # see below
   
   phyint lo disabled
   
   My configuration (adapt to your needs):
   eth0 is the WAN interface
   eth1 is the LAN interface
   
   192.168.1.0/24 is the subnet between the Swisscom modem and the firewall
   172.16.123.128/29 is the subnet I dedicated to STB.
   224.0.0.0/4 identifies multicast addresses. It goes from 224 to 240, hence the /4 network mask.
   See the documentation for the meaning of quickleave.
3. Configure the firewall to let said subnets in the INPUT chain. Allow FORWARDing between both interfaces.
   (These instructions are vague on purpose. If you are running a firewall you obviously know your policies and rules better than anyone!
   If you want to be precise you will have to allow for the IGMP protocol with the parameter -p igmp of IPTables for instance).
4. Launch igmpproxy in debug mode, as root:
   igmpproxy -d -v /usr/local/etc/igmpproxy.conf
   
   See if you can watch TV. Look at the addresses. Try to add them to your configuration, relaunch the proxy and see if it works.
5. Schedule igmpproxy to start at boot.
   You can use this script and follow the instructions at the bottom of the page. (Discard the explanations in Russian in-between ;-))
6. Happy TV watching!
   
   
   Here's what the traffic looks like while I've been playing with this:
   (This is with pnp4nagios and the stat_net plugin, which is based on /proc/net/dev.)

I am actually quite surprised by this. It seems to need only 3 MBit/s for one TV (and not recording a second channel). I would have expected from 2 to 3 times more bandwidth usage.
I tried nload just to make sure:

Same observation!
(Possibly because /sys/class/net/eth0/statistics/rx_bytes which nload uses, is filled by the same kernel module as /proc/net/dev, in which case this should be accurate as these are the actual frames read by the Linux kernel.)


If you are getting continuous messages in your syslog:

I tried everything. I can either stop the messages or watch TV. For some reason, the STB sends "upstream" IGMP packets. It's very strange and doesn't make much sense. The IGMP proxy doesn't "proxies" them and the TV runs smoothly.
I figured I could safely ignore these. You should also note that IGMPProxy is a very young project (version 0.1).

I added this near the beginning of /etc/rsyslog.conf::msg, contains, "The source address 172.16.123." ~

and restarted the rsyslog daemon. The messages are finally gone. Another solution is to use another file for IGMPProxy logs.

Keywords: linux, unix, firewall, shorewall, bsd, iptables, netfilter, bluewin tv, swisscom, iptv, multicast, 

Getting instant notifications on your smartphone or tablet from Nagios

If you are a network administrator or manage a production system of some sort, you might be interested to be notified as soon as possible in the event of problems.

Nagios is a great tool to watch your hosts and services and even though it's kind of 1990-ish it still does a very good job and stays the most used tool for this task.

It can natively send e-mails. That's great, but I don't know about you, I get more than 40 e-mails a day, and even with SaneBox and other tricks it's easy to miss e-mails and it can literally take days until I notice one that was more important than the others.

Pushover provides instant notifications on handheld devices such as Android, iPhone, and iPad using "push" technology (instead of polling).
It has many advantages:

• It uses a REST API, meaning you can use it in all scenarios you ever dreamt of.
• They provide clients for smartphones, tablets and they even support OS X Growl notifications.
• It's inexpensive at only $5 USD.
• You don't need to run or install any software on your servers.

Here is a tutorial on how to use it with Nagios, based on Debian 7.7 stable, as of December 2014:

1. Create an account on Pushover. Mark down your personal key.
2. Create an application. Mark down the application key.
   I named mine "Nagios" but you can choose anything you like.
3. Set up your account on one of your devices.
4. Download Jedda Wignall's script in your plugins directory with your root account:
   wget -O /usr/lib/nagios/plugins/notify_by_pushover.sh https://raw.githubusercontent.com/jedda/OSX-Monitoring-Tools/master/notify_by_pushover.sh
5. Make it executable:
   chmod +x /usr/lib/nagios/plugins/notify_by_pushover.sh
6. Create a Nagios command to run it. I like to have a /etc/nagios3/conf.d/custom_handlers.cfg file, but you can name it with any name you want as long as it is in that directory. Copy the following coutent to it:
   
   define command {
       command_name        handler_pushover
        command_line       $USER1$/notify_by_pushover.sh -u $ARG1$ -a $ARG2$ -t "$ARG3$" -m "$ARG4$"
   }
   
   $USER$1 stands for /usr/lib/nagios/plugins and $ARGn$ will be replaced with arguments set after the command-name following "!" in the next step.
7. If you want every host to report a failure (and supposing you used the template everywhere), modify /etc/nagios3/conf.d/generic-host_nagios2.cfg, and add the following line anywhere inside the host definition:
   
   event_handler                   handler_pushover!PERSONAL_KEY!APP_KEY!Nagios!$HOSTNAME$ : $HOSTSTATE$
   Replace the personal key and the application keys with the ones obtained in step 1 and 2.
   
   "Nagios" here is the title that will appear in the notification, it doesn't need to match the application name and it can include spaces.
8. The last parameter of the command (everything after the last "!") is the message. Here I simply chose to show the hostname and the state. The variables enclosed in dollar signs can be found in the Nagios manual.
9. If you want every service to report a failure (and supposing you used the template everywhere), modify /etc/nagios3/conf.d/generic-service_nagios2.cfg, and add the following line anywhere inside the service definition:
   
   event_handler                   handler_pushover!up4evfJ7bkP43M5mLvGZ24Gp1aoM14!a6fLnAFT2rNMCXeEeogbghbLbV1o9m!Nagios!$HOSTNAME$ / $SERVICEDESC$ ($SERVICESTATE$) : $SERVICEOUTPUT$
   
   This will print the status of the service as well as the line shown in the web interface.
   You can include modify "Nagios" (title of the notification) and everything after the last ("!") with macros from the Nagios manual.
10. Reload Nagios:
    service nagios3 reload
11. Provoke a "warning" or "critical" message in Nagios. You should receive a notification as soon as the problem is discovered by Nagios.

Life is great!

Now, there may be better ways to do this. You should probably use the notification mechanism from Nagios instead. The handler is supposed to "handle" the problem, not report it. As a result I had to put retry_interval 120 in some of my service definitions to avoid getting the same message again and again.

Tell me what you think and how it worked out for you!