Debian's libvirt in unstable finally passes the Technology Compatibility Kit (libvirt-tck) for qemu:///system:
Files=59, Tests=1579, 315 wallclock secs ( 0.77 usr 1.02 sys + 31.40 cusr 10.14 csys = 43.33 CPU)
Result: PASS
We're running libvirt's internal test suite since 0.9.0 but this doesn't launch any real virtual machines to check things like suspend, resume, snapshotting, migration and it doesn't create storage pools and volumes or networks. This means a lot of testing was done manually with each release.
libvirt-tck provides a framework to perform these kinds of integration testing between libvirt and it's drivers, comes with hundreds of testcases already and it's easy to set up since it's available in experimental:
apt-get install -t experimental libvirt-tck
Edit /etc/libvirt-tck/default.cfg if you don't want to use the default connection URI. Then just run libvirt-tck:
libvirt-tck
Since this is supposed to stress libvirtd and the hypervisor and since it creates and deletes storage pools, networks and virtual machines it's recommended to run this on a dedicated system to make sure the tests don't blow away any precious config.
If a test fails one can debug that single test using:
LIBVIRT_TCK_CONFIG=/etc/libvirt-tck/default.cfg prove --verbose /usr/share/libvirt-tck/tests/hooks/051-daemon-hook.t
This already revealed errors in our package like a missing parted build dependency breaking the creation of disk based storage pools, bugs in libvirt, crashes in QEMU and minor errors in the test suite itself. We're currently cheating a bit since the tests for hooks and nwfilter are currently disabled due to license problems with the used Perl modules.
libvirt-tck can easily be integrated into Jenkins since it can now use libtap-formatter-junit-perl to generate JUnit XML. To do so simply configure a free-style software project to execute these commands:
rm -f libvirt-tck.xml
sudo libvirt-tck --timer --format junit --force > libvirt-tck.xml
allow Jenkins to run libvirt-tck as root
jenkins ALL=(ALL) NOPASSWD: /usr/bin/libvirt-tck --timer --format junit --force
and configure it to publish a JUnit test result report.
Currently I'm only running the tests for qemu:///system so any help running and debugging this for LXC, VirtualBox or XEN is very welcome.
Many thanks go to Salvatore Bonaccorso for packaging the missing Perl modules needed by libvirt-tck.
