More sandboxing

When working on untrusted code or data it's impossible to predict what happens when one does a:

bundle install --path=vendor

or

npm install

Does this phone out your private SSH and GPG keys? Does a

evince Downloads/justdownloaded.pdf

try to exploit the PDF viewer? While you can run stuff in separate virtual machines this can get cumbersome. libvirt-sandbox to the rescue! It allows to sandbox applications using libvirt's virtualization drivers. It took us a couple of years (The ITP is from 2012) but we finally have it in Debian's NEW queue. When libvirt-sandbox creates a sandbox it uses your root filesystem mounted read only by default so you have access to all installed programs (this can be changed with the --root option though). It can use either libvirt's QEMU or LXC drivers. We're using the later in the examples below:

So in order to make sure the above bundler call has no access to your $HOME you can use:

sudo virt-sandbox \
   -m ram:/tmp=10M \
   -m ram:$HOME=10M \
   -m ram:/var/run/screen=1M \
   -m host-bind:/path/to/your/ruby-stuff=/path/to/your/ruby-stuff \
   -c lxc:/// \
   -S $USER \
   -n rubydev-sandbox \
   -N dhcp,source=default \
   /bin/bash

This will make your $HOME unaccessible by mounting a tmpfs over it and using separate network, ipc, mount, pid and utc namespaces allowing you to invoke bundler with less worries. /path/to/your/ruby-stuff is bind mounted read-write into the sandbox so you can change files there. Bundler can fetch new gems using libvirt's default network connection.

And for the PDF case:

sudo virt-sandbox \
  -m ram:$HOME=10M \
  -m ram:/dev/shm=10M \
  -m host-bind:$HOME/Downloads=$HOME/Downloads \
  -c lxc:/// \
  -S $USER \
  -n evince-sandbox \
  --env="DISPLAY=:0" \
  --env="XAUTHORITY=$XAUTHORITY" \
  /usr/bin/evince Downloads/justdownloaded.pdf

Note that the above example shares /tmp with the sandbox in order to give it access to the X11 socket. A better isolation can probably be achieved using xpra or xvnc but I haven't looked into this yet.

Besides the command line program virt-sandbox there's also the library libvirt-sandbox which makes it simpler to build new sandboxing applications. We're not yet shipping virt-sandbox-service (a tool to provision sandboxed system services) in the Debian packages since it's RPM distro specific. Help on porting this to Debian is greatly appreciated.

Interactively installing Debian virtual machines with virt-install without having to download anything in advance can already be done by pointing it to a Debian mirror via --location. But you can also add files to the initrd after downloading using --initrd-inject. Upstreams intended use is for kickstart files but we can also feed it a preseed.cfg to automate the whole installation:

virt-install --connect=qemu:///system \
             --location=http://ftp.us.debian.org/debian/dists/stable/main/installer\-i386 \
             --initrd-inject=/path/to/preseed.cfg \
             --extra-args="auto" \
             --name d-i --ram=512 \
             --disk=pool=default,size=5,format=qcow2,bus=virtio

preseed.cfg is a regular preseed file (as described in the Debian Wiki) in your local filesystem. I'm using this one for Squeeze and Wheezy VMs. It must be named preseed.cfg in order for d-i to pick it up from the initrd. This also works for URLs like qemu+ssh://<remotehost>/system/ since virt-install uses libvirt's streaming API to upload the kernel and modified initrd to the remote host.

In case you're running this on an i386 you'll need this fix which already sits in experimental.

This blog is flattr enabled.

Eric Blake and myself had the chance to talk about libvirt at FLOSS Weekly hosted by Randall Schwartz and Simon Phipps. It's available as video and audio stream.

Debian's libvirt in unstable finally passes the Technology Compatibility Kit (libvirt-tck) for qemu:///system:

Files=59, Tests=1579, 315 wallclock secs ( 0.77 usr  1.02 sys + 31.40 cusr 10.14 csys = 43.33 CPU)
Result: PASS

We're running libvirt's internal test suite since 0.9.0 but this doesn't launch any real virtual machines to check things like suspend, resume, snapshotting, migration and it doesn't create storage pools and volumes or networks. This means a lot of testing was done manually with each release.

libvirt-tck provides a framework to perform these kinds of integration testing between libvirt and it's drivers, comes with hundreds of testcases already and it's easy to set up since it's available in experimental:

apt-get install -t experimental libvirt-tck

Edit /etc/libvirt-tck/default.cfg if you don't want to use the default connection URI. Then just run libvirt-tck:

libvirt-tck

Since this is supposed to stress libvirtd and the hypervisor and since it creates and deletes storage pools, networks and virtual machines it's recommended to run this on a dedicated system to make sure the tests don't blow away any precious config.

If a test fails one can debug that single test using:

LIBVIRT_TCK_CONFIG=/etc/libvirt-tck/default.cfg prove --verbose /usr/share/libvirt-tck/tests/hooks/051-daemon-hook.t

This already revealed errors in our package like a missing parted build dependency breaking the creation of disk based storage pools, bugs in libvirt, crashes in QEMU and minor errors in the test suite itself. We're currently cheating a bit since the tests for hooks and nwfilter are currently disabled due to license problems with the used Perl modules.

libvirt-tck can easily be integrated into Jenkins since it can now use libtap-formatter-junit-perl to generate JUnit XML. To do so simply configure a free-style software project to execute these commands:

rm -f libvirt-tck.xml
sudo libvirt-tck --timer --format junit --force > libvirt-tck.xml

allow Jenkins to run libvirt-tck as root

jenkins ALL=(ALL) NOPASSWD: /usr/bin/libvirt-tck --timer --format junit --force

and configure it to publish a JUnit test result report.

Currently I'm only running the tests for qemu:///system so any help running and debugging this for LXC, VirtualBox or XEN is very welcome.

Many thanks go to Salvatore Bonaccorso for packaging the missing Perl modules needed by libvirt-tck.

Here are some more examples of libguestfs usage:

What kernels, modules and packages are in a (currently powered off) libvirt based virtual machine vmfoo:

virt-inspector -c qemu:///system vmfoo

Show free space in virtual machine vmfoo:

virt-df -c qemu:///system vmfoo

List file systems:

virt-list-filesystems -c qemu:///system -l vmfoo

Edit /etc/passwd in vmfoo:

virt-edit -c qemu:///system lenny-base-clone /etc/passwd

Tell guestfish to mount all filesystems of virtual machine vmfoo for more complex inspection and manipulation tasks like LVM operations, partitioning, resizing or filesystem check or creation:

guestfish $(virt-inspector -c qemu:///system --ro-fish vmfoo)

Debian packages are still on alioth.

I've finally updated the libguestfs Debian packages to 1.0.84 getting them back in sync with upstream. Download instructions are on the pkg-libvirt's wiki page.

Fetching files from a vm image with guestfish is as simple as:

guestfish --ro -a lenny-base-clone.img -m /dev/debian/root download /etc/passwd /tmp/passwd

After hacking around #561991 by running debirf as root we now do pass the testsuite again. The new version includes fuse support as well as the hivex tools to manipulate windows registries.

The source for the Debian package is available at git.debian.org. The next step will be to split out the appliance so we can fetch that one via http and upload libguestfs to contrib.

As already mentioned on <pkg-libvirt-discuss@lists.alioth.debian.org> libvirt 0.7.4-2 (currently in experimental) runs qemu and kvm processes as libvirt-qemu:kvm instead of root allowing you to use features like bridged networking without having to run the emulator itself with full privileges.

Kvm 88+dfsg-3 supports ksm to safe memory when running several similar virtual machines. On a recent linux kernel just do a

echo 1 > /sys/kernel/mm/ksm/run

to activate it. Then run two similar virtual machines and see how /sys/kernel/mm/ksm/pages_shared increases.

Thanks to work of Jan and Micheal Debian now also ships kvm's stable branch qemu-kvm.

The slides of my talk about libvirt at dc9 are now available.

I've worked a bit on Richard's libguestfs Debian packages They now also provide perl and python bindings as well as a debug package and the appliance is built using debootstrap/debirf. The wiki has the necessary apt/sources.list (packages are currently available for i386 only).

The source for the Debian package is available at git.debian.org so please send patches to wrap more of the available language bindings into Debian packages.

If you're interested in updates on this as well as other virtualization related stuff in Debian what about joining the pkg-libvirt-discuss list.

New upstream versions of virtinst and virt-manager are available via Debian's experimental distribution. All the bugfixes we carried as patches have been applied upstream by Cole Robinson, which is great. New features include disk- and newtork-I/O graphs and direct support for installing paravirtualized Debian Lenny Xen Guests via:

   virt-install -n xenfoo1 -r 96 --disk path=/path/to/file,device=disk,size=1 --location=http://ftp.nl.debian.org/debian/dists/testing

Updated libvirt packages for Debian that fix CVE-2008-5086 are available:

• 0.4.6 (>= 0.4.6-10) in Unstable - these work for Lenny and Unstable
• 0.5.1 (>= 0.5.1-4) in Experimental

A version for etch-backports is available here. I'll upload this version to backports.org as soon as 0.4.6-10 moved into Lenny/Testing.

The libvirt munin plugins improved quiet a bit since their initial release:

• I added two more plugins:
  • libvirt-cputime: monitor cpu time used by virtual machines
  • libvirt-mem: monitor the amount of memory used by virtual machines
• these two plugins have adjustable limits for their warning and critical values, see the README for more details
• the plugins now work properly with Xen by special casing Domain-0
• they also ship as a Debian package

Now that Debian Installer supports Xen guests (many thanks to the d-i team for that) it was about time that virtinst/virt-manager could make use of it. Debian's virtinst package in experimental now supports this (the patch is also already included in upstream's mercurial repository). To install a paravirtialized guest on an existing block device use:

virt-install -n xenfoo1 -r 96 --disk path=/dev/blockdevice,device=disk --location=http://people.debian.org/~joeyh/d-i

To install into an image file you additionally have to specify the file size in GB:

virt-install -n xenfoo1 -r 96 --disk path=/path/to/file,device=disk,size=1 --location=http://people.debian.org/~joeyh/d-i

if you don't want to use a sparse file also add sparse=false. In all cases the disk will become /dev/xvda in the domU and the device will be partitioned during installation (so no need to pass in single partitions, booting will happen using pygrub).

--location above points to the Debian Installer daily builds since there hasn't been a d-i release with Xen support yet (this should happen really soon now). Once this has happened you can use any Debian mirror to install. (--location=http://ftp.nl.debian.org/debian/dists/testing/main/installer-i386).

My top three remaining "annoyances" in libvirt when it comes to KVM are:

1. missing hotplug for disks
2. daemon restart kills VMs
3. missing migration support

Since KVM now supports PCI hotadd since a couple of releases I fixed up 1 today. I already sent a partial patch for 2 some time ago which I need to finish and clean up. Hopefully 3 is resolved upstream until then.

Once this is done libvirt plus kvm will be a great tandem for virtualization. On the desktop virt-manager does a great job while for H/A purposes it integrates with Redhat's cluster suite - you can use the vm.sh resource script for this after removing the xen specific code and replacing it with calls to ''virsh''.

Here are two munin plugins that monitor disk and network I/O of virtual machines using libvirt. These are also two simple examples on using the libvirt python bindings. You need python-libvirt and python-libxml2 installed in order to use them. If you don't want to use the default uri, try:

[libvirt-*]
env.uri qemu:///system

Munin-node needs to have read access to libvirt's socket. This is the case in Debian's default installation.

Libvirt provides a lot of help to debug problems:

• it creates per virtual machine logfiles in /var/log/libvirt/ (when running as root) and ~/.libvirt/qemu/logs/ (when running as user). These contain the verbatim calls to QEMU/KVM so you can copy/paste them into the shell - you usually want to drop the -monitor pty argument then so you can access the QEMU/KVM monitor interactively (Ctrl-Alt-F2).

• running it verbosely in foreground mode is as easy as simply running libvirtd -v.

• if you do export LIBVIRT_DEBUG=1 before doing the above you'll get the very useful debugging output. I usally drop the -v then.

• also use the above export statement when debugging problems in virt-manager, virt-viewer or virsh but note that in the case of QEMU/KVM you'll end up calling into libvirtd with these tools so watching the daemon's debug output is more important.

• when debugging testcase failures in the source code export VIR_TEST_DEBUG=1 is your friend. Debugging a XML testcase failure then looks like:

  abs_srcdir=/your/path/to/libvirt.cvs/tests/ VIR_TEST_DEBUG=1 ./qemuxml2xmltest