VirtualBox headless server setup

UPDATE: For day-to-day operations, you can use these handy aliases.

It might be interesting in some scenarios to have a virtual machine running on a headless server.
If you don't have thousands of dollars to spend on VMware vSphere or similar commercial solutions you can do it for free, if you don't mind the command-line.
And once you've created one you can simply clone it and it's not much longer than clicking buttons on a web interface...
Also, except for Oracle's implementation of the VRDP server (a cousin of the VNC protocol), the solution is entirely based on open source software if that is something important for you.

If you are really allergic to the command-line or you want to let sane human people use your "VM server", you might want to give a look at unofficial solutions such as

• phpVirtualBox (web-based, includes a RDP client as a Java applet)
• RemoteBox (heavy client on your workstation, makes it easier to connect with RDP)

Both solutions try to imitate the official GUI application, so you should be feeling at home with either of them. Note phpVirtualBox might be insecure and dangerous. You should restrict its access using HTTPS and Basic Authentication. See also: Apache's "allow from ..." parameter to restrict by the client IP address or subnet.

My setup

Host: Debian 7.7, 64-bit
Guest: Windows 8.1, 64-bit. It will be named "Windows8.1_x64". The setup is similar for other operating systems.
Packages: VirtualBox 4.3 with guest additions, DKMS

VirtualBox package

Although VirtualBox is available through official Debian repositories, the version there is usually out-of-date and it can make things complicated with newer versions of guest OSes or it might be hard to find older versions of the VirtualBox extension pack.

As root (or using a combination of tee and sudo):

# echo "deb http://download.virtualbox.org/virtualbox/debian wheezy contrib" > /etc/apt/sources.list.d/virtualbox.list
# wget -q https://www.virtualbox.org/download/oracle_vbox.asc -O- | apt-key add -
# apt-get install virtualbox-4.3 dkms

Note: When I tried it APT was so slow I downloaded the file with a web browser directly and installed it with dpkg.

DKMS

Make sure you installed dkms as shown in the previous section. Otherwise you will have to re-compile and re-install the VirtualBox kernel module after every update of your kernel.

DKMS is a general-purpose package maintained by the "Dynamic Kernel Modules Support Team" at Debian. It doesn't come from Oracle / with VirtualBox.

VirtualBox extension pack

The extension pack is required for many features such as the VRDP server (VRDP is similar to VNC) and the guest tools.

Download the pack from the VirtualBox website: https://www.virtualbox.org/wiki/Downloads
The link is titled "All supported platforms". Once you've read the license, you can download with wget:

$ wget "http://download.virtualbox.org/virtualbox/4.3.SOMETHING/Oracle_VM_VirtualBox_Extension_Pack-4.3.DO_NOT_COPY_THIS_LINK.vbox-extpack"

DO NOT COPY THIS LINK. This command is only given for reference. Copy the link from the website instead.

Install the extension pack as root:

# VBoxManage extpack install *.vbox-extpack

Creating a naked virtual machine

The rest of the tutorial can and should be done not as root but as a regular user. It doesn't even seem like membership to the "vboxusers" group is required.

$ VBoxManage createvm --name Windows8.1_x64 --register

or

$ VBoxManage createvm --name Windows8.1_x64
$ VBoxManage registervm "$HOME/VirtualBox VMs/Windows8.1_x64/Windows8.1_x64.vbox"

Both commands will create the directory above. I advise you to put all your files there, except maybe for the installation disk image.

Let's set a few options on the VM:

$ VBoxManage modifyvm Windows8.1_x64 --ostype Windows8_64 --memory 4096 --vram 128 --nic1 nat --nictype1 82540EM

The list of os types can be obtained using VBoxManage list ostypes.

The (RAM) memory is given in MB. VRAM stands for "Video RAM" (I think this parameter's default value is 128 MB). Note the NIC type is "8254 ZERO EM". I need to research on this but although 82540EM is compatible with most guest OSes,  

NIC type virtio is a better choice if the guest OS supports it as it offers better performance.
UPDATE: Windows 8.1 is not compatible with virtio.

Bridged mode

Bridged mode is useful if you want to communicate with the VM from any device on your network.
This mode is particularly useful for SSH connections.
The default NAT mode hides the VM from the network, so it looks like the network traffic is coming from your host machine.
To enable it, type VBoxManage modifyvm Windows8.1_x64 --nic1 bridged --bridgeadapter1 eth0.
eth0 is the interface connected to the network you'd like to access the VM from.
Note modern systems bundled with systemd now use a predictable naming scheme for interface names and the name will be usually something like enp5s0 (ethernet) or wlp3s0 (wireless).
Either way try ip addr if you are not sure...

Hard drive

As you may or may not know, hard drives live independently from virtual machines in VirtualBox, hence the existence of the "Virtual Media Manager" in VirtualBox's GUI.
For this reason we need to create the drive first and attach it to the virtual machine later.
This feature is also nice when you wish to play with a virtual RAID setup by "disconnecting" drives.

I suggest installing the hard drive alongside the other files of the VM:

$ cd VirtualBox\ VMs/Windows8.1_x64

To create a 50 GB hard drive:

$ VBoxManage createhd --filename Windows8.1_x64.vdi --size 50000

Note 1: The file is created in the current directory.

Note 2: The file won't use 50 GB until the virtual disk is actually full. Remember it can be full even if Windows still shows available space due to fragmentation on FAT and NTFS partitions.

When copying virtual hard drives I suggest defragmenting it first and then compacting the VDI (Virtual Disk Image) file with VBoxManage modifyhd ...filename... --compact

To attach the drive to the virtual machine we first need to create a SATA controller:

$ VBoxManage storagectl Windows8.1_x64 --name SATA --add sata --controller IntelAhci --bootable on

Then we can attach the disk:

$ VBoxManage storageattach Windows8.1_x64 --storagectl SATA --port 0 --device 0 --type hdd --medium "$(pwd)"/"Windows8.1_x64.vdi"

Important: The last argument must be an absolute path (you can safely copy and paste the last command if you copied and pasted the first two commands of this section).

Installation medium

Locate your ISO image of the Windows installation disk.

For some reason optical disk drives need to be attached to an IDE controller. Let's add one:

$ VBoxManage storagectl Windows8.1_x64 --name IDE --add ide --controller PIIX4

Then we can attach the image to the controller:

$ VBoxManage storageattach Windows8.1_x64 --storagectl IDE --port 0 --device 0 --type dvddrive --medium /absolute/path/to/your/windows8_disk.iso

Replace the ISO image path with emptydrive to eject the medium.

Make the system boot on the DVD:

$ VBoxManage modifyvm Windows8.1_x64 --boot1 dvd

Configure the VRDP port (optional, 3389 by default)

If you are running several virtual machines at the same time you want to change the VRDP port of the VM with

$ VBoxManage modifyvm Windows8.1_x64 --vrdeport 5012 --vrde on

The option --vrdeaddress 192.168.0.1 (or whatever) is also available and might be useful if you don't use a firewall.

Windows 8 specific configuration

Windows 8 will start but the installer will complain with obscure error codes (0x00000225) with the options set to far. We need a few more:

$ VBoxManage modifyvm Windows8.1_x64 --pae on --ioapic on

In fact you probably want to enable PAE/NX with any guest OS.

There is a problem with the mouse cursors from the Aero theme. To solve it use the "(None)" theme with no shadow.

Starting the VM

It is now time to start the VM. Although you can also use VBoxHeadless that shows errors and warnings, I prefer the following command that forks and runs the VM as a daemon process:

$ VBoxManage startvm Windows8.1_x64 --type headless

Connecting from your workstation

Note for Linux guest OSes: If you want to install a Linux guest you won't be able to connect to it with SSH (some distributions enable this mode of installation) as long as the NIC type is set to NAT. You can change it to another supported mode and configure the interface accordingly or connect to the virtual machine (which will be needed if you intend to run an installer) using the same method as for the Windows installation described here.

Make sure the VRDP server is started and listening for connections on the appropriate port:

# netstat -tlpn | grep VBoxHeadless

Connect from your workstation using your preferred RDP client, for instance for the KDE Remote Desktop Client (krdc):

$ krdc rdp://your-server-ip:3389

Press "Cancel" when asked for a password.
Otherwise install and run rdesktop:

$ rdesktop -a 16 -N your-server-ip:3389

Put IPv6 addresses between brackets [2a00:...]:3389

The Windows 8.1 installer uses a resolution of 1024x768. rdesktop will select it automatically.

rdesktop worked fined on Ubuntu 14.04 but on Arch Linux I could only use FreeRDP:



CAVEAT: My mouse pointer was off. This is corrected with the guest additions but of course you can only install them after Windows... Try to put the pointer at one of the edge of the screen to make matters marginally better. You can also use the keyboard.

Installing the Guest additions

The guest additions is simply a DVD that comes with the extension pack.

But before loading the drive (I already showed you the technique, remember?), power off the machine, either with one of VBoxManage controlvm arguments or from the guest OS.

Afterward we can take a look at the options set on the VM:

$ VBoxManage showvminfo Windows8.1_x64

For instance, near the line "IDE (0, 0)" you will see that the Windows installation DVD was unmounted.

Let's mount the Guest additions DVD:

$ VBoxManage modifyvm Windows8.1_x64 --dvd /usr/share/virtualbox/VBoxGuestAdditions.iso

and start the VM with the method shown earlier.

Install the guest additions from the guest OS by locating the DVD drive and firing up the installer.

Reboot.
The "mouse bug" should be fixed now.

Performance optimizations

If you use KRDC, select 16 bit color depth and select "Connection type: Broadband".

Disable the visual effects using this method:
http://www.thewindowsclub.com/disable-visual-effects-windows
My favorite options:

Windows still looks natural but everything is much smoother.

You can also use a smaller resolution and less colors (8 bit instead of 16 or 24) in your RDP client.

Choosing a solid color desktop background also speeds things up.


Quick startup and shutdown

To make Windows boot or shutdown quickly, do not boot it up or shut it down ;-)

Use Virtualbox "save state" feature instead:

$ VBoxManage controlvm Windows8.1_x64 savestate

$ VBoxManage startvm Windows8.1_x64 --headless

You can also "reset", "poweroff" or "discardstate" if appropriate. These are dangerous options similar to pulling off the plug of a physical computer. Use them only if the guest OS didn't power off the machine after shutdown (80-90s style) or the OS froze.
You can also "pause" or "resume" the guest OS temporarily.

I always "savestate" a Windows guest OS when I am not using it because I've noticed Windows or one app or another is writing and reading on the disk all the time. This reduces performance of the host OS and wears the drive, shortening its life.

To shutdown the system cleanly, supposing there is no unsaved work (if there is, the shutdown will be cancelled):

$ VBoxManage controlvm Windows8.1_x64 acpipowerbutton

Security

See the VirtualBox manual link at the end of this tutorial to set up a proper authentication method.
Here is a quick how-to:

$ VBoxManage setproperty vrdeauthlibrary "VBoxAuthSimple"

$ VBoxManage modifyvm Windows8.1_x64 --vrdeauthtype external

$ VBoxManage setextradata Windows8.1_x64 "VBoxAuthSimple/users/YOUR_RDP_USERNAME" $(VBoxManage internalcommands passwordhash "YOUR_RDP_PASSWORD" | awk '{print $3;}')

Change the user and password with one of your choice.
krdcwill ask for your password, but with rdesktop you need to provide it on the command-line:

$ rdesktop -a 16 -N YOUR_SERVER_IP:3389 -u YOU_RDP_USER -p YOUR_RDP_PASSWORD -r sound:off -x l -z -P

(Here I used a few other options such as optimization for LAN connections, compression, no sound...)
For FreeRDP:

$ xfreerdp /v:YOUR_SERVER_IP:3389 /u:YOUR_RDP_USER /p:YOUR_RDP_PASSWORD -toggle-fullscreen /bpp:16 /w:1808 /h:1017 +auto-reconnect

(Here the options are: 16 bit graphics (65536 colors), and a custom resolution. "+auto-reconnect" is supposed to re-connect automatically, but to be honest it doesn't really work. YMMV...)

With Windows' Remote Desktop Connection utility, you need to click "Show options" and then the option about credentials. After you've connected once, your credentials will be saved.

Cool stuff

There are plenty of cool stuff such as Remote USB that you can do. Make sure to check out the VirtualBox user manual.

Your Feedback counts!

Please comment below if you have any remark, question or suggestion or if you liked this tutorial.