Wednesday, March 26, 2014

python-bugzilla 1.0.0 released

I released python-bugzilla 1.0.0 yesterday. Since Arun led the charge to get python 3 support merged, it seemed like as good a time as any to go 1.0 :)

python-bugzilla provides the /usr/bin/bugzilla command line tool that is heavily used in Fedora and internally at Red Hat. The library itself is used by parts of Fedora infrastructure like pkgdb and bodhi.

This major changes in this release:

- Python 3 support (Arun Babu Neelicattu)
- Port to python-requests (Arun Babu Neelicattu)
- bugzilla: new: Add --keywords, --assigned_to, --qa_contact (Lon
- bugzilla: query: Add --quicksearch, --savedsearch
- bugzilla: query: Support saved searches with --from-url
- bugzilla: --sub-component support for all relevant commands

The sub component stuff is a recent extension that hasn't been used much yet. I believe the next update will add sub components for the Fedora kernel package, so bugs can be assigned to things like 'Kernel->USB', 'Kernel->KVM", etc. Quite helpful for complicated packages.

But I don't know if it's going to be opened up as an option for any Fedora package, I guess we'll just have to wait and see. I assume there will be an announcement about it at some point.

Tuesday, March 25, 2014

virt-manager: Improved CPU model default

In virt-manager 1.0 we improved many of the defaults we set when creating a new virtual machine. One of the major changes was to how we choose the CPU model that's exposed to the VM OS.

CPU model here means something like 'Pentium 3' or 'Opteron 4' and all the CPU flags that go along with that. For KVM, every CPU flag that we expose to the VM has to be something provided by the host CPU, so we can't just unconditionally tell the VM to use the latest and greatest features. The newer the CPU that's exposed to the guest, the more features the guest kernel and userspace can use, which improves performance.

There's a few trade offs however: if live migrating your VM, the destination host CPU needs to be able to represent all the features that have been exposed to the VM. So if your VM is using 'Opteron G5', but your destination host is only an 'Opteron G4', the migration will fail. And changing the VM CPU after OS installation can cause Windows VMs to require reactivation, which can be problematic. So in some instances you are stuck with the CPU model the VM was installed with.

Prior to virt-manager 1.0, new VMs received the hypervisor default CPU model. For qemu/kvm, that's qemu64, a made up CPU with very few feature flags. This leads to less suboptimal guest OS performance but maximum migration compatibility.

But the reality is that cross host migration is not really a major focus of virt-manager. Migration is all about preserving uptime of a server VM, but most virt-manager users are managing VMs for personal use. It's a bigger win to maximize out of the box performance.

For virt-manager 1.0, we wanted new VMs to have an identical copy of the host CPU. There's two ways to do that via libvirt:
  1. mode=host-passthrough: This maps to the 'qemu -cpu host' command line. However, this option is explicit not recommended for libvirt usage. libvirt wants to fully specify a VM's hardware configuration, to insulate the VM from any hardware layout changes when qemu is updated on the host. '-cpu host' defers to qemu's detection logic, which violates that principle.
  2. mode=host-model: This is libvirt's attempted reimplementation of '-cpu host', and is the recommended solution in this case. However the current implementation has quite a few problems. The issues won't affect most users, and they are being worked on, but for now host-model isn't safe to use as a general default.
So for virt-manager 1.0, we compromised to using the nearest complete CPU model of the host CPU. This requires a bit of explanation. There are multiple CPUs on the market that are labelled as 'core2duo'. They all share a fundamental core of features that define what 'core2duo' means. But some models also have additional features. virt-manager 1.0 will ignore those extra bits and just pass 'core2duo' to your VM. This is the best we can do for performance at the moment without hitting the host-model issues mentioned above.

However this default is configurable: if you want to return to the old method that maximizes migration compatibility, or you want to try out host-model, you can change the default for new VMs in Edit->Preferences:

Saturday, March 22, 2014

virt-manager 1.0.1 release

I've just released virt-manager 1.0.1. This was mostly a bug fix release to gather up the sizeable number of bug fixes that accumulated since virt-manager 1.0.0 was released last month.

Though there were a few mini features added:
  • - virt-install/virt-xml: New --memorybacking option (Chen Hanxiao)
  • - virt-install/virt-xml: New --memtune option (Chen Hanxiao)
  • - virt-manager: UI for LXC (Chen Hanxiao)
  • - virt-manager: gsettings key to disable keygrab (Kjö Hansi Glaz)
  • - virt-manager: Show domain state reason in the UI (Giuseppe Scrivano)
  • - Support for many more device live update operations, like changing the network source for a running VM, etc.
Builds inprogress for F20 and rawhide

Monday, March 17, 2014

Snapshot support in virt-manager

The biggest feature we added in virt-manager 1.0 is VM snapshot support. Users have been asking us to expose this in the UI for quite a long time. In this post I'll walk you through the new UI.

Let's start with some use cases for VM snapshots:
  1. I want to test some changes to my VM, and either throw them away, or use them permanently.
  2. I want to have a single Fedora 20 VM, but multiple snapshots with mutually exclusive OS changes in each. One snapshot might have F20 libvirt installed, but another snapshot will have libvirt.git installed. I want to switch between them for development, bug triage, etc.
  3. I encountered a bug in the VM. I want to save the running state of the VM incase developers want further debugging information, but I also want to restart the VM and continue to use it in the meantime.
The libvirt APIs support two different types of snapshots with qemu/kvm.

Internal snapshots

Internal snapshots are the snapshots that QEMU has supported for a long time. Libvirt refers to them as 'internal' because all the data is stored as part of the qcow2 disk image: if you have a VM with a single qcow2 disk image and take 10 snapshots, you still have only one file to manage. This is the default snapshot mode if using the 'virsh snapshot-*' commands.

These snapshots can be combine disk and VM memory state for 'checkpointing', so you can jump back and forth between a saved running VM state. A snapshot of an offline VM can also be performed, and only the disk contents will be saved.

  • Only works with qcow2 disk images. Since virt-manager has historically used raw images, pre-existing VMs may not be able to work with this type.
  • They are non-live, meaning the VM is paused while all the state is saved. For end users this likely isn't a problem, but if you are managing a public server, minimizing downtime is essential.
  • Historically they were quite slow, but this has improved quite a bit with QEMU 1.6+

External snapshots

External snapshots are about safely creating copy-on-write overlay files for a running VM's disk images. QEMU has supported copy-on-write overlay files for a long time, but the ability to create them for a running VM is only a couple years old. They are called 'external' because every snapshot creates a new overlay file.

While the overlay files have to be qcow2, these snapshots will work with any base disk image. They can also be performed with very little VM downtime, at least under a second. The external nature also allows different use cases like live backup: create a snapshot, back up the original backing image, when backup completes, merge the overlay file changes back into the backing image.

However that's mostly where the benefits end. Compared to internal snapshots, which are an end to end solution with all the complexity handled in QEMU, external snapshots are just a building block for handling the use cases I described above... and the many of the pieces haven't been filled in yet. Libvirt still needs a lot of work to reach feature parity with what internal snapshots already provide. This is understandable, as the main driver for external snapshot support was for features like live backup that internal snapshots weren't suited for. Once that point was reached, there hasn't been much of a good reason to do the difficult work of filling in the remaining support when internal snapshots already fit the bill.

virt-manager support

Understandably we decided to go with internal snapshots in virt-manager's UI. To facilitate this, we've changed the default disk image for new qemu/kvm VMs to qcow2.

The snapshot UI is reachable via the VM details toolbar and menu:

That button will be disabled with an informative tool tip if snapshots aren't supported, such as if the the disk image isn't qcow2, or using a libvirt driver like xen which doesn't have snapshot support wired up.

Here's what the main screen looks like:

It's pretty straight forward. The column on the left lists all the snapshots. The 'VM State' means the state the VM was in when the snapshot was taken. So running/reverting to a 'Running' snapshot means the VM will end up in a running state, a 'Shutoff' snapshot will end up with the VM shutoff, etc.

The check mark indicates the last applied snapshot, which could be the most recently created snapshot or the most recently run/reverted snapshot. The VM disk contents are basically 'the snapshot contents' + 'whatever changes I've made since then'. It's just an indicator of where you are.

Internal snapshots are all independent of one another. You can take 5 successive snapshots, delete 2-4, and snapshot 1 and 5 will still be completely independent. Any notion of a snapshot hierarchy is really just metadata, and we decided not to expose it in the UI. That may change in the future.

Run/revert to a snapshot with the play button along the bottom. Create a new snapshot by hitting the + button. The wizard is pretty simple:

That's about it. Give it a whirl in virt-manager 1.0 and file a bug if you hit any issues.

Monday, March 10, 2014

Extending the virt-xml command line

As previously explained, virt-manager 1.0.0 shipped with a tool called virt-xml, which enables editing libvirt XML from the command line in one shot. This post will walk through an example of patching virt-xml to support a new libvirt XML value.

A bit of background: libvirt VM configuration is in XML format. It has quite an extensive XML schema. For QEMU/KVM guests, most of the XML attributes map to qemu command line values. QEMU is always adding new emulated hardware and new features, which in turn require the XML schema to be extended. Example: this recent libvirt change to allow turning off Spice drag + drop support with a <filetransfer enable='no'/> option.

For this example, we are going to expose a different property: defaultMode, also part of the graphics device. defaultMode can be used to tell qemu to open all spice channels in secure TLS mode. But for the purpose of this example, what defaultMode actually does and how it works isn't important. For virt-xml, the only important bit is getting the value for the command line, writing it correctly as XML, and unit testing the XML generation.

You can see the completed virt-xml git commit over here.

Step 0: Run the test suite

The virt-manager test suite aims to always 100% pass, but depending on your host libvirt version things can occasionally be broken. Run 'python test' and note if any tests fail. The important bit here is that after we make all the following changes, the test suite shouldn't regress at all.

Step 1: XML generation

 diff --git a/virtinst/ b/virtinst/  
 index 37f268a..a87b71c 100644  
 --- a/virtinst/  
 +++ b/virtinst/  
 @@ -204,6 +204,7 @@ class VirtualGraphics(VirtualDevice):  
    passwdValidTo = XMLProperty("./@passwdValidTo")  
    socket = XMLProperty("./@socket")  
    connected = XMLProperty("./@connected")  
 +  defaultMode = XMLProperty("./@defaultMode")

    listens = XMLChildProperty(_GraphicsListen)  
    def remove_listen(self, obj):  

Starting with virt-manager git, first we extend the internal API to map a python class property to its associated XML value.

The virtinst/ directory contains the internal XML building API used by all the tools shipped with virt-manager. There's generally a single file and class per XML object, examples
  • <graphics> device
  • <cpu> block
  • <os> block
  • And so on
If you aren't sure what file or class you need to alter, try grepping for a property you know that virt-install already supports. So, for example, using virt-install --graphics=? I see that there's a property named passwdValidTo. Doing 'git grep passwdValidTo' will point to virtinst/

'XMLProperty' is some custom glue that maps a python class property to an XML value, for both reading and writing. The value passed to XMLProperty is an XML xpath. If you don't know how xpaths work, google around, or try to find an existing example in the virtinst code.

Notice that this doesn't do much else, like validate that the value passed to defaultMode is actually valid. The general rule is to leave this up to libvirt to complain.

Step 2: Command line handling

 diff --git a/virtinst/ b/virtinst/  
 index 826663a..41d6a8c 100644  
 --- a/virtinst/  
 +++ b/virtinst/  
 @@ -1810,6 +1810,7 @@ class ParserGraphics(VirtCLIParser):  
      self.set_param("passwd", "password")  
      self.set_param("passwdValidTo", "passwordvalidto")  
      self.set_param("connected", "connected")  
 +    self.set_param("defaultMode", "defaultMode")  
    def _parse(self, opts, inst):  
      if opts.fullopts == "none":  

The next step is to set up command line handling. In this case we are adding a sub option to the --graphics command. Open up virtinst/ and search for '--graphics', you'll find a comment with a ParserGraphics class defined after it. That's where we plug in new sub options.

The 'self.set_param' registers the sub option: first argument is the name on the cli, second argument is the property name we defined above. In this case they are the same.

Some options do extra validation or need to do special handling. If you need extra functionality, look at examples that pass setter_cb to set_param.

After this bit is applied, you'll see defaultMode appear in the --graphics=? output, and everything will work as expected. But we need to add a unit test to validate the XML generation.

An easy way to test that this is working is with a command line like:

./virt-install --connect test:///default --name foo --ram 64 \
               --nodisks --boot network --print-xml \
               --graphics spice,defaultMode=secure

That will use libvirt's 'test' driver, which is made for unit testing, and doesn't affect the host at all. The --print-xml command will output the new XML. Verify that your new command line option works as expected before continuing. See the HACKING file for additional tips for using the test driver.

Step 3: Unit test XML generation

 diff --git a/tests/ b/tests/  
 index a2448d2..397da45 100644  
 --- a/tests/  
 +++ b/tests/  
 @@ -559,6 +559,7 @@ class XMLParseTest(unittest.TestCase):  
      check("channel_cursor_mode", "any", "any")  
      check("channel_playback_mode", "any", "insecure")  
      check("passwdValidTo", "2010-04-09T15:51:00", "2011-01-07T19:08:00")  
 +    check("defaultMode", None, "secure")  
      self._alter_compare(guest.get_xml_config(), outfile)  

tests/ tests reading and writing, so it will test the change we made in virtinst/ Before you make any tests/, run 'python test --testfile', you should see a new error: this is because will emit a test failure if a new XML property in virtinst/ that isn't explicitly tested!

Similar to how you found what virtinst/ file to edit by grepping for a known graphics property like passwdValidTo, do the same in to find the pre-existing graphics test function. The check() invocation is a small wrapper for setting and reading a value: the first argument is the python property name we are poking, the second argument is what the initial value should be, and the final argument is the new value we are setting.

The initial XML comes from tests/xmlparse-xml/*, and is initialized at the start of the function. But in our case, we don't need to manually alter that. So make the change, and rerun 'python test --testfile' and...

Things broke! That's because the generated XML output changed, and contains our new defaultMode value. So we need to update the known-good XML files we compare against. The easiest way to do that is to run 'python test --testfile --regenerate-output'. Run 'git diff' afterwards to ensure that only the graphics file was changed.

Finally, run 'python test' and ensure the rest of the test suite doesn't regress compared to the initial run you did in Step 0.

For cases where you added non-trivial command line handling, take a look at tests/, where we run a battery of command line parsing tests. You likely want to extend this to verify your command line works as expected.

Also, if you want to add an entirely new command line option that maps to an XML block, this commit adding the --memtune option is a good reference.

Step 4) Documentation?

For each new option sub property, the general rule is we don't need to explicitly list it in the man page or virt-install/virt-xml --help output. The idea is that command line introspection and libvirt XML documentation should be sufficient. However, if your command line option has some special behavior, or is particularly important, consider extending man/virt-install.pod.

Step 5) Submit the patch!

So your patch is done! git commit -a && git send-email -1 --to or simply drop it in a bug report. If you have any questions or need any assitance, drop us a line.

(update 2015-09-04: Point git links to github)

Tuesday, March 4, 2014

virt-xml: Edit libvirt XML from the command line

We shipped a new tool with virt-manager 1.0.0 called virt-xml. virt-xml uses virt-install's command line options to allow building and editing libvirt domain XML. A few basic examples:

Change the <description> of domain 'example':
# virt-xml example --edit --metadata description="my new description"  

Enable the boot device menu for domain 'example':
# virt-xml example --edit --boot bootmenu=on  

Hotplug host USB device 001.003 to running domain 'fedora19':
# virt-xml f19 --add-device --host-device 001.003 --update

The virt-xml man page also has a comprehensive set of examples.

While I doubt anyone would call it sexy, virt-xml fills a real need in the libvirt ecosystem. Prior to virt-xml, a question like 'how do I change the cache mode for my VM disk' had two possible answers:

1) Use 'virsh edit'

'virsh edit' drops you into $EDITOR and allows you to edit the XML manually. Now ignoring the fact that editing XML by hand is a pain, 'virsh edit' requires the user to know the exact XML attribute or property name, and where to put it. And if its in the wrong place or mis-named, in most cases libvirt will happily ignore it with no feedback (this is actually useful at the API level but not very friendly for direct user interaction).

But more than that, have you ever seen what happens when you drop a less than savvy user into vim for the first time? It doesn't end well :) And this happens more than you might expect.

2) Use virt-manager

The more newbie friendly option for sure, the UI is intuitive enough that people can usually find the XML bit they want to twiddle... provided it actually exists in virt-manager.

And that's the problem: over time these types of requests put pressure on virt-manager to expose many kind-of-obscure-but-not-so-obscure-that-virsh-edit-is-an-acceptable-answer XML properties in the UI. It was unclear where to draw the line on what should be in the UI and what shouldn't, and we ended up with various UI bits that very few people were actually interacting with.

Enter virt-xml

So here's virt-xml, that allows us to easily make these types of XML changes with a single command. This takes the pressure off virt-manager, and provides a friendly middle ground between the GUI and 'virsh edit'. It also greatly simplifies documentation and wiki pages (like fedora test day test cases).

The CLI API surface is huge compared to virt-manager's UI. There's no reason that virt-xml can't expand to support every XML property exposed by libvirt. And we've worked on making it trivially easy to to extend the tool to handle new XML options: in many cases, it's only 3 lines of code to add a new --disk/--network/... sub option, including unit testing, command line introspection, and virt-install support.