Arch Linux and Arduino with your favorite IDE

You can edit your Arduino code from any IDE and compile it / upload it without using the Arduino IDE. To do that, the ino tool comes to the rescue.

If you are using Arch Linux you might be interested to know I uploaded a package named ino-1.6-git on AUR. It has been patched to work with the latest version of the Arduino SDK.

Installation

yaourt ino-1.6-git
sudo gpasswd -a $USER tty
sudo gpasswd -a $USER uucp
sudo gpasswd -a $USER lock

Log out and in again, or reboot.

Quick Start:

cd my-arduino-project
ino init
$EDITOR src/sketch.ino
ino build
ino upload

Happy Arduino-ing!

Remove audio from a video on Linux

If you are working on a computer vision project, there is a great chance that you have been recording videos with sound, which you have no use of. To save space and processing power, you can strip the audio from the video just like this:

avconv -i input_video.mp4 -an output_video.mp4

(Replace avconv with ffmpeg if necessary.)

Just so you know, there is something dishonest with avconv as they make you believe ffmpeg is dead or obsolete. In fact it's nothing like that. avconv is a fork of ffmpeg and has become the default on Ubuntu. So if you are reading to use a filter that isn't present in avconv, then you need to download the latest version of ffmpeg from source or from a PPA. Read more on this subject here.

Rotate a video on Linux

When you record videos on your smartphone, the video itself is not rotated. The video recorder simply adds a tag telling the player how to rotate the video.
That's why mediainfo reports "Rotation: 180°" or other angles on videos recorded from my Samsung Galaxy S4.

Unfortunately unlike the video player on your phone, the vast majority of video players on desktop computers and TVs don't support this tag.

If the video is upside down:

avconv -i original_video.mp4 -vf "vflip,hflip" output_video.mp4
or
avconv -i original_video.mp4 -vf "transpose=2,transpose=2" output_video.mp4

(You can replace avconv with ffmpeg if avconv is not found on your system.)

Note that the first version is slightly faster (2-5 %).

If the video was recorded in portrait mode:

if the top of the image is on the left of the screen, it needs a +90° rotation:

avconv -i original_video.mp4 -vf "transpose=1" output_video.mp4

if the top of the image is on the right of the screen, it needs a -90° or +270° rotation:

avconv -i original_video.mp4 -vf "transpose=2" output_video.mp4

Improving FabulaTech USB over Network

I've been looking for a solution to share a USB device over the network for years. Unfortunately, as of today, no viable opensource solution exists. usbip is supposed to be that solution, but it's broken and I never managed to make it work on Ubuntu, despite the program being available in universe. The installation on Windows is a very tedious process, even more complicated IMO than the procedure below.

There are however commercial solutions. One of them is Fabulatech USB over Network. It will work on Windows, Mac and Linux.

On Linux, there is no graphical user interface, and they don't provide anything that would make our lives easier. Only a server program and a client program that can be invoked from the command-line.
That's why we'll have to write a daemon and a script to automatically share some devices after they are plugged in.

Step 0 : Unzip the Linux tarball somewhere

Let's say this somewhere is /opt/ftusbnet.

Step 1: Start the daemon (ftusbnetd) at boot

Install supervisord: sudo apt-get install supervisor

Write a configuration file for supervisor by editing /etc/supervisor/conf.d/ftusbnetd.conf:

[program:ftusbnetd]

command=/opt/ftusbnet/sbin/ftusbnetd

autostart=true

autorestart=unexpected

Restart supervisord: sudo service supervisor restart

(You can use supervisorctl if you don't want to restart all the running services.)

If you have never used supervisor before, you really should. Regular init scripts are still useful, but most of the time supervisor does the job much better, with only two lines.

UPDATE: supervisor 3.0b2-1 distributed with Ubuntu 14.04.1 LTS has a little bug. It won't start, after complaining that /var/log/supervisor doesn't exist.
You can either change the line childlogdir=/var/log/supervisor in /etc/supervisor/supervisord.conf, or create the directory before starting supervisord by modifying /etc/init.d/supervisor and adding the line mkdir /var/log/supervisor after case "$1" in start)

Step 2: Run a script when a USB device is connected

Read the previous blog post.

Let's assume the "add" script is /opt/ftusbnet/bin/udev-connect.

Usually, to share a device you would issue the ftusbnetctl command like this:

ftusbnetctl share 1234, where 1234 is an ID generated by ftusbnet and that even changes each time you plug the device.

It would be nice if we could tell udev to run a script only after ftusbnetctl "sees" the USB device. Unfortunately we can't do that. The device won't be accessible until udev has run all the scripts we told him to run. We can't just "sleep" or start a background process. 

The only workaround I found is to run the script 1 to 60 seconds after the USB device is processed by udev. So yes: You might have to wait up to a minute before the device is shared. 

For this we will use the at utility. Unfortunately, at is an old UNIX tool and it only works with minutes, days, and weeks. Tell me if you find a better solution!

/opt/ftusbnet/bin/udev-connect contains this:

#!/bin/sh

echo "/opt/ftusbnet/bin/scansnap-connect" | at 'now + 1 minute'

In my case I am connecting a device that is shown as "ScanSnap" when executing ftusbnetctl list.

Make sure the script is executable.

Step 3: (Cont'd)

Here is the content of /opt/ftusbnet/bin/scansnap-connect:

#!/bin/sh

ftusbclient=/opt/ftusbnet/bin/ftusbnetctl

while ! $ftusbclient list | grep ScanSnap

do  

    sleep 1

done

sleep 1

ftid=$( ${ftusbclient} list | grep ScanSnap | awk '{print $1;}' )

logger -t ftusbnet-udev "Sharing device ID $ftid"

$ftusbclient share $ftid

Make sure the script is executable.

The before-last line will print a message in the syslog. If all you never see the line or the ID is truncated, you have a problem.

Ubuntu: run a script when a USB device is plugged in

There's a lot of stuff you can do with udev. One of them is running a script when a USB device is plugged in.

It's as simple as:

1. figure out the vendor id and product id with lsusb
2. edit /etc/udev/rules.d/85-something.rules:

ACTION=="add", SUBSYSTEM=="usb", ATTR{idVendor}=="04c5", ATTR{idProduct}=="11a2", RUN+="/absolute/path/to/your/script/that/will/be/executed/as/root"

3. Write the script mentioned above and make sure it's executable.
4. Restart udev: service udev restart.

You can do other things with udev such as managing /dev/ and the permissions on the device.
However you should not use udev to mount USB disks, there are other tools for that.

To run a script when detaching a USB device, replace ACTION=="add" with ACTION=="remove". You can put several lines in the same ".rules" file.

Two scripts for the Cisco VPN Client

I recently wrote a post about using openconnect instead of the Cisco VPN Client on Linux. If you really want to use the Cisco VPN Client but miss the handy vpnc and vpnd commands, this article is just for you.
It uses the expect tool from Don Libes, who wrote it in 1990 when he was working for the U.S. government. Therefore, Expect is in the public domain.

Before executing the commands and modifying the script, you should run /opt/cisco/anyconnect/bin/vpn to understand what is going on. The script tries to mimic a user waiting for the console to print things, and then type on the keyboard.

Here's your vpnc script:



And here is vpnd:

Speed up Windows updates

Apparently there's something you can do to speed up Windows updates:

1. Open a command-line prompt with elevated privileges (Win + S, search "prompt", right-click, Run as administrator)
2. sfc /scannow

The Windows System File Checker tool will detect corrupt system files, and some people have experienced a noticeable speed up of the Windows updates.

OpenVPN with two simple commands

In the previous blog post, I wrote about Cisco VPN. Now here is the same thing with OpenVPN.

It also contains a little trick I have to use because I can't manage to get the DNS configuration right after connecting. (I need to access the DNS server internal to the LAN I am connecting to.)

Here's the connect.sh script:



And here's the disconnect.sh script:



To make the scripts work, you have to put them in the same folder than your OpenVPN configuration file (here named "client.conf"). This directory must also contain a valid "resolv.conf" file that will be used when the VPN connection is up and running.

As in the previous article, you can execute connect.sh and then close the terminal.
I advise you to create aliases to these scripts. The scripts are made so that you can run them from any directory.

Cisco VPN on Linux, with free software

Many companies and schools use a VPN solution from Cisco.
While the client on Windows is fine, the Linux version does nasty stuff such as creating copies and aliases of the /etc/hosts file. Also, I don't like running closed-source software with root privileges.
The third reason not to use Cisco VPN Client on Linux: you can't manage it with the package manager.

There's an open source alternative called openconnect.

I've written two aliases:

1) vpnc initiates the session, then goes to the background. You can close the terminal.
2) vpnd disconnects your from the VPN network.

Put this in your ~/.bash_aliases or whatever:

alias vpnc='echo Your_Password | sudo openconnect vpn_server_adress.domain.com --authgroup="AuthGroup as displayed on Cisco VPN client" --user=Your_Username --passwd-on-stdin --background'
alias vpnd='sudo pkill openconnect'

You can also put your password in a file, chmod 400 it, and use "cat my_file" instead of "echo Your_Password".
If you think don't like my pkill solution, you can use openconnect's "--pid-file" argument.

There you have it. Your friends using Cisco VPN client will envy you so much. 4 characters and you are connected!

Google's No reCAPTCHA in PHP

Google bought reCAPTCHA in 2009, and recently they made their small revolution by replacing the usual but at the end craaaaaaaazy skewed texts with a simple checkbox, except in a few cases where there might be a regular captcha or a small popup to ask you to identify pictures of cats :

On my website I was still using the old library with PHP classes and functions. Now they provide a RESTful JSON web service to do the same. Unfortunately Google provides no example and if you are using PHP on a shared hosting you will run into problems if you try to use methods commonly found on the Internet, StackOverflow included.

So here it is.

On the page displaying the captcha:


In the PHP script handling the form:

The public and private keys are provided by Google for each reCAPTCHA instance.

The Internet tells you to use the one-liner file_get_contents(url) to retrieve the JSON response, unfortunately on shared hosting this is not an option and you have to fallback to cURL instead.

The 1st generation

The story behind reCAPTCHA is an interesting one.
Originally the slogan was "Stop spam. Read books" because each day millions of captchas were solved to help digitizing books were OCR technologies were unable to do so.

They had this system with two different OCR engines and if neither of them was able to find a word from the dictionary, then the world would be presented to users. Words from OCR get 0.5 point, and answers from humans get 1.0 point. When the image gets 2.5 points with the same deciphered word, it is considered as accepted.
Words considered valid are then shown next to unknown words in order to tell humans and machines apart (this is called a Turing test). These valid words can be words that were decoded correctly by the two OCR engines or words resulting from the process described in the previous paragraph.

The New York Times Archive

At some point, the slogan disappeared because reCAPTCHA stopped reading books... Apparently it helped digitizing 13 million articles from The New York Times.

The 2nd generation

On websites not using the new API demonstrated in the animated GIF above, the captcha image looks like this:

It is only today that I realized the number on the right comes right from Google Street View and made the link between Google buying reCAPTCHA and these house numbers. So instead of helping culture millions of people are now helping Google with their business and working for free for the company. (Yet this blog is hosted by Google, I know ;-))

Like me you are probably amazed how the guys at Google are able to automatically detect house numbers in all images the cameras take, so you would ask yourself: Is it really that complicated to identify a 3 digit house number?
No, it is not! But Google wants to make sure its OCR engine is right before filling its database with wrong information to provide its users accurate information and make more money.

The 3rd generation

It's great news for us humans only having to click a textbox instead of trying to decipher impossibly complex CAPTCHAs as found on downloading websites. Remember the one with the cats and dogs, and I am not talking of Microsoft latest finding?

Alternatives

TextCaptcha is another good alternative you should checkout.

Good reads

At Stanford they studied how CAPTCHAs have become hard to solve for humans.

Last but not least, I didn't resist posting this, a Turing test letting only computers in:

(Source: http://www.smbc-comics.com/?id=2999)

More ditching: OpenSSL

That thing must go in the trash, for real.
It turns out the Heartbleed vulnerability everybody has been talking about in April 2014 was just the tip of the iceberg.
The reasons? The whole thing is buggy, very badly written, opened to NSA backdoors, and having tested it myself, the library (that is, the C interface) is impossible to use without losing one's temper...

Let's welcome LibreSSL, a fork of OpenSSL by the OpenBSD community. They have already developed a more user-friendly library (libtls) and they have been actively fixing the codebase since May.

LibreSSL has replaced OpenSSL in OpenBSD 5.6, released in November 2014. It is now production ready and can be trusted. [1]
In my opinion, people should move away from OpenSSL, and even more from Microsoft CryptoAPI which is closed-source. [2]

[1] http://www.openbsd.org/papers/eurobsdcon2014-libressl.html
[2] http://blog.cryptographyengineering.com/2013/09/on-nsa.html (An excellent blog for security lovers by the way! Love what this guy writes.)

Swisscom PLC device password Asoka PL7667-ETH

I have been looking to configure Swisscom's PLC / HomePlug AV device (Asoka PL7667-ETH) for quite some time now, but I couldn't find any manual mentioning the password of the web interface.

Here it is:

Username: admin
Password: welcome

(found in this document)

You might want to use the interface to:

• switch the attached device on and off, thus reducing the electrical bill (I'll probably write a script for that ;-))
• change the IP address and other settings

Python traps

Short post today: I love Python. Unfortunately, like all programming languages, it has some traps that you might fall into if you don't pay attention. Read this blog post for more information.

For instance

def myMethod(lst=[]):
    lst += 'a'
    return lst

doesn't do what you think it does. Indeed:

> myMethod()
['a']
> myMethod()
['a', 'a']
> myMethod()
['a', 'a', 'a']

Suprise!
For once, C++ is more user-friendly, as it will create a new instance for the value of default parameters every time the method is called. That is not the case in Python.

Downloading subtitles

Subliminal is a command-line utility written in Python that downloads subtitles for TV episodes or movies.

While media players such as Kodi (formerly known as XBMC) offer this feature, it's a painful process to pause the video, open the subtitles menu, download the subtitles file, wait until the download completes and so on.
I stream videos directly from my NAS, so that wasn't even an option because the smart TV is unable to store the subtitles file anywhere, it has to be at the same location as the video.

With one little command, Subliminal fetches subtitles from various sources and finds what it thinks is the best subtitles track for the media files you throw at it.

Here's how to install and use it:

1. Try to find if you can install Subliminal directly from your package manager. If so, skip to step 4.
2. Install Python PIP.
3a. Recommended method: using virtualenv(wrapper): mkvirtualenv subtitles; pip install subliminal colorlog
3b. Alternate method: directly on the system: sudo pip install subliminal colorlog
4. Download the subtitles: subliminal -l en -s --color videoOrDirectory anotherVideoOrDirectory...
You can use other flags. These will download a subtitle track in English. The "-s" flag means the file will be named something.srt instead of something.en.srt. I use this option because some players are not able to associate the ".en.srt" subtitles file with the video. "--color" is just for eye candy. You can use "-v" for more messages.

To my knowledge subliminal won't browse directories recursively. You can achieve that effect with:
find . -type d -exec sh -c 'subliminal -l en -s  "{}"' \;

Subliminal is smart enough to detect if the file is a video, and if it needs subtitles.
I was confused with the message "No video to download subtitles for". It will be printed if there was no video in the path(s) you provided and if the video(s) already contain(s) a subtitles track. You can check that with mediainfo.

As with Kodi, you should follow certain naming conventions. It is best to include the full name of the show or movie in the title, and describe the episode as "S01E02" (Season 1 Episode 2) or "1x02". For best compatibility avoid spaces and non-ASCII characters. If you downloaded the video from the Internet, you should however try to keep the filename as it was, including the uploader's nickname and the video quality (HDTV, 720p, 1080p...) There might be a subtitles track for the exact video file you have.

Ditch scp (SSH copy) and use rsync instead

That is a strong statement, I know.

You have been living happily and using scp to transfer files between machines is a no brainer. But here is the thing: HTTP, FTP, SMTP, and SSH are bad file transfer protocols. Why you ask? Because they offer no guarantee you have got yourself a carbon copy.

The solution? Checksums! And not MD5 or SHA1 which are both broken, but SHA256 or SHA512, both already available on your system (except for Windows...)

So you could sha256sum the file before transfer, transfer, then sha256sum the file after the transfer. But that's rather painful and you'd have to write a complicated script to automate this. There's a much simpler solution.

For some reason, people think rsync is a complicated tool. Maybe because it's got so many flags and options. But it's actually a very complete tool that does what you want without having to write complicated shell scripts.
Four amazing things it can do:

• Syncing directories (great for backup), transferring only what needs to be transferred, deleting things that aren't present on one side if that's what you want
• Resuming directory transfers where it left, copying only what's needed
• On-the-fly compression
• Checksums

The last feature is a very interesting one, because you will know immediately if the transfer failed, whereas scp won't even tell you anything and might act like everything was fine. It's only 6 months later when you try to restore a backup that you realize all your data is lost forever...

So, replace this:

scp myfile.tar.gz remote-user@remote-host:/remote/directory/

with this:

rsync -avPe ssh myfile.tar.gz remote-user@remote-host:/remote/directory/

Of course you can transfer files the other way around. There is no additional flag needed for directories, but you have to understand the importance of trailing slashes. They play an important role.

Here I used the SSH protocol to perform the transfer but there are other options, including the rsync protocol, that is commonly used to mirror distribution repositories accross the world, amongst other things.

Parallel Gzip with a progress bar

Everybody knows how to create a "tar.gz", probably using something like this:

tar zcvf foo.tar.gz bar-directory/

This is fine to compress something relatively small (a few megs).
However, if you have to archive hundreds of gigabytes, you probably want features such as:

• Speeding things up
• Displaying a progress bar
• Displaying an ETA

This command* offers all of these:

tar cf - bar-directory/ -P | pv -s $(du -sb bar-directory/ | awk '{print $1}') | pigz > foo.tar.gz

You'll probably have to install pigz, which is a parallelized version of Gzip. You can substitute it with pbzip2 if you want a "tar.bz2" archive.

Supposing the CPU was the bottleneck and not the I/O, which is probably the case if you are working on the local filesystem, this will speed things up by a factor of "number of CPU cores". For instance, instead of 1 hour with gzip, it took only 15 minutes with pigz on my machine with 4 CPU cores.
In fact it was almost as fast as copying the directory without archiving / compression.

As a bonus you get a nice progress bar in your terminal:

*On OS X and *BSD, du uses another unit, use awk to convert the size into bytes.

Anybody can look at your Instagram

Geek version: I just think everybody should know that Instagram uses unencrypted connections. So not only it's very easy to sniff the traffic it sends or receives from a REST JSON web service, but also that the links to the pictures in the JSON messages are publicly accessible for anyone with the right link...

Non-geek version: Anybody can look at your Instagram pictures, what's in your inbox, what you send and what you receive, especially if you use a company / school / shared WiFi network. Use a mobile connection or avoid using Instagram altogether.
Instagram automatically downloads data without your knowledge, so even if the app is not opened, people can look at your data.

Also, never use a password on more than one website (use a password manager instead) and use a code on your phone.