Wednesday, June 29, 2016

UEFI virt roms now in official Fedora repos

Kamil got to it first, but just a note that UEFI roms for x86 and aarch64 virt are now shipped in the standard Fedora repos, where previously the recommended place to grab them was an external nightly repo. Kamil has updated the UEFI+QEMU wiki page to reflect this change.

On up to date Fedora 23+ these roms will be installed automatically with the relevant qemu packages, and libvirt is properly configured to advertise the rom files to applications, so enabling this with tools like virt-manager is available out of the box.

For the curious, the reason we can now ship these binaries in Fedora is because the problematic EDK2 'FatPkg' code, which had a Fedora incompatible license, was replaced with an implementation with a less restrictive (and more Fedora friendly) license.

Saturday, June 18, 2016

virt-manager 1.4.0 release

I've just released virt-manager 1.4.0. Besides the spice GL bits that I previously talked about, nothing too much exciting in this release except a lot of virt-install/virt-xml command line extensions.

The changelog highlights:
  • virt-manager: spice GL console support (Marc-André Lureau, Cole Robinson)
  • Bump gtk and pygobject deps to 3.14
  • virt-manager: add checkbox to forget keyring password (Pavel Hrdina)
  • cli: add --graphics gl= (Marc-André Lureau)
  • cli: add --video accel3d= (Marc-André Lureau)
  • cli: add --graphics listen=none (Marc-André Lureau)
  • cli: add --transient flag (Richard W.M. Jones)
  • cli: --features gic= support, and set a default for it (Pavel Hrdina)
  • cli: Expose --video heads, ram, vram, vgamem
  • cli: add --graphics listen=socket
  • cli: add device address.type/address.bus/...
  • cli: add --disk seclabelX.model (and .label, .relabel)
  • cli: add --cpu (and .cpus, and .memory)
  • cli: add --network rom_bar= and rom_file=
  • cli: add --disk backing_format=

Friday, June 10, 2016

check-pylint: mini tool for running pylint anywhere

pylint and pep8 are indispensable tools for python development IMO. For projects I maintain I've long ago added a 'setup pylint' sub-command to run both commands, and I've documented this as a necessary step in the contributor guidelines.

But over the years I've accumulated many repos for small bits of python code that never have need for a script, but I still want the convenience of being able to run pylint and pep8 with a single command and a reasonable set of options.

So, a while back I wrote this tiny 'check-pylint' script which does exactly that. The main bit it adds is automatically searching the current directory for python scripts and modules and passing them to pylint/pep8. From the README:

Simple helper script that scoops up all python modules and scripts beneath the current directory, and passes them through pylint and pep8. Has a bit of smarts to ignore .git directory, and handle files that don't end in .py

The point is that you can just fire off 'check-pylint' in any directory containing python code and get a quick report.

Tuesday, June 7, 2016

python-bugzilla API changes in git

I've made a number of API changes to python-bugzilla in git. Most of it is removing old mis-designed features which I don't think anyone is actually using. I went through all apps I know of that use python-bugzilla (fedora bits like bodhi pkgdb infra scripts, bugwarrior, to confirm as much and I don't think anything below will affect them. Even if there's no complaints I won't cut the release for 4-6 weeks.

The list is long but most changes are fairly straight forward. If any of this sounds controversial, or if I misjudged one of the 'unused' bits and should reconsider, please leave your thoughts here or on the python-bugzilla mailing list.

The interesting change is:
  • Bugzilla.bug_autorefresh now defaults to False. Previously if trying to access, and 'foo' wasn't cached, the Bug() object would basically do a Bug.refresh() and fetch all the bug contents from the server. This default sucks, it can lead to poorly performing scripts and unnecessary bugzilla load. If you have code that uses include_fields, exclude_fields, or extra_fields anywhere, this change may affect you. You can set Bugzilla().bug_autorefresh = False before doing any bug lookup to force this off (with current versions) and catch any errors. If you hit issues you probably need to extend your include_fields specifications. The reason for this change is that the old pattern made it too easy for people's scripts to unintentionally start requiring a much higher number of XMLRPC calls, thus completely negating the usage of include_fields in the first place. Some more details over here:

These are slightly interesting and may impact a few people:
  • Credentials are now cached in ~/.cache/python-bugzilla/. If the old ~/.bugzilla{cookies,token} files are in place we will continue to update and use them. * RHBugzilla.rhbz_back_compat __init__ attribute is gone. If the user manually set this to true, we would alter some Bug fields returned via query() to convert python lists into a single comma separated string, to match rhbugzilla output from before the 2012 upgrade. Convert your code to use the modern bugzilla list output.
  • bin/bugzilla was converted to argparse, which has a bug on python 2.7 that affects some possible command lines: If you use a command like 'bugzilla modify --cc' to remove that email from the CC list, you now need to ensure there's an equals sign in there, like That is backwards compatible with old python-bugzilla too, so update your scripts now. A few other options expect that format too.
  • bugzilla query --boolean_chart option is removed. It provided a custom specification for crafting complex queries, similar to what the bugzilla UI can do. I don't think anyone is really using this, but if you are, generate a web query URL and pass it to bugzilla query --from-url '$URL' which is much easier to deal with.
  • bugzilla query 'boolean' options, where you could say 'bugzilla query --keywords 'foo & bar' to match both substrings, are no longer supported. If you need logic like this, use the --from-url technique mentioned above.
  • Bug.get_history() is now Bug.get_history_raw() Bugzilla.bugs_history is now Bugzilla.bugs_history_raw() This API is only a year old and unwisely returns raw output from bugzilla, which has some formatting oddities. I renamed it to *_raw so we have the future opportunity to add an API with better output.
  • The getbugsimple and getbugssimple APIs were removed. They were basically just wrappers around standard getbug() at this point. Use getbug/getbugs instead.
  • The simplequery API was removed. This basically matches the basic query from the bugzilla front page. Just use the standard query methods instead, see examples/ from git. I didn't find any users of this function.
  • Various whiteboard editing functions from the Bug object were removed. - getwhiteboard: this is just a wrapper around standard Bug attributes - appendwhiteboard, prependwhiteboard, setwhiteboard: these were just wrappers around standard build_update/update_bugs - addtag/gettags/deltag which minor wrappers around the *whiteboard functions, and were poorly named, given that there's an actual bug 'tags' field at this point. See examples/ from git for examples of using the standard update APIs. That said I've never heard of anyone actually using these, and they were designed around really old RHBZ APIs.

These I doubt will actually affect anyone:
  • Bugzilla.initcookie() dropped... use Bugzilla.cookiefile = X instead 
  • Bugzilla.adduser() dropped... it was the same as Bugzilla.createuser 
  • RHBugzilla.multicall __init__ attribute is gone. It's been a no-op and raises a warning for a long time. 
  • Bugzilla.version string is gone... this was meant to describe the python-bugzilla API version but was never used. 
  • Initing like Bugzilla() with no options was previously allowed, but now requires an explicit Bugzilla(url=None). 
  • We no longer handle cookies in LWP format. We've been silently converting them to mozilla format for 2 years so I assume this doesn't affect anyone, but if it does, you might need to delete your caches in ~/.bugzilla{cookies,token} 
  • Bug.setstatus(), Bug.close() args 'private_in_it' and 'nomail' were removed: they have been no-ops for years 
  • Bug.addcomment() args 'timestamp', 'bz_gid' and 'worktime' were removed: they've been no-ops for years
  • Bug.setassignee() arg 'reporter' was removed: it's thrown an error for years

Sunday, May 22, 2016

spice OpenGL/virgl acceleration on Fedora 24

New in Fedora 24 virt is 3D accelerated SPICE graphics, via Virgl. This is kinda-sorta OpenGL passthrough from the VM up to the host machine. Much of the initial support has been around since qemu 2.5, but it's more generally accessible now that SPICE is in the mix, since that's the default display type used by virt-manager and gnome-boxes.

I'll explain below how you can test things on Fedora 24, but first let's cover the hurdles and caveats. This is far from being something that can be turned on by default and there's still serious integration issues to iron out. All of this is regarding usage with libvirt tools.

Caveats and known issues

Because of these issues, we haven't exposed this as a UI knob in any of the tools yet, to save us some redundant bug reports for the above issues from users who are just clicking a cool sounding check box :) Instead you'll need to explicitly opt in via the command line.

Testing it out

You'll need the following packages (or later) to test this:
  • qemu-2.6.0-2.fc24
  • libvirt-
  • virt-manager-1.3.2-4.20160520git2204de62d9.fc24
  • At least F24 beta on the host
  • Fedore 24 beta in the guest. Anything earlier is not going to actually enable the 3D acceleration. I have no idea about the state of other distributions. And to make it abundantly clear this is linux only and likely will be for a long time at least, I don't know if Windows driver support is even on the radar.
Once you install a Fedora 24 VM through the standard methods, you can enable spice GL for your VM with these two commands:

$ virt-xml --connect $URI $VM_NAME --confirm --edit --video clearxml=yes,model=virtio,accel3d=yes
$ virt-xml --connect $URI $VM_NAME --confirm --edit --graphics clearxml=yes,type=spice,gl=on,listen=none

The first command will switch the graphics device to 'virtio' and enable the 3D acceleration setting. The second command will set up spice to listen locally only, and enable GL. Make sure to fully poweroff the VM afterwards for the settings to take effect. If you want to make the changes manually with 'virsh edit', the XML specifics are described in the spice GL documentation.

Once your VM has started up, you can verify that everything is working correctly by checking glxinfo output in the VM, 'virgl' should appear in the renderer string:

$ glxinfo | grep virgl
    Device: virgl (0x1010)
OpenGL renderer string: Gallium 0.4 on virgl

And of course the more fun test of giving supertuxkart a spin :)

Credit to Dave Airlie, Gerd Hoffman, and Marc-André Lureau for all the great work that got us to this point!

Wednesday, January 20, 2016

Tips for querying git tags

With package maintenance, bug triage, and email support, I often need to look at a project's git tags to know about the latest releases, when they were released, and what releases contain certain features. Here's a couple workflow tips that make my life easier.

Better git tag listing

Based on Peter Hutterer's 'git bi' alias for improved branch listing (which is great and highly recommended), I made one for improved tags output that I mapped as 'git tags'. Output looks like:

  • Shows tag name, commit message, commit ID, and date, all colorized. Commit message is redundant for many projects that tag the release commit, but it's interesting in some cases.
  • Tags are listed by date rather than alphabetically. Some projects change tag string formats, or versioning schemes, that then don't sort correctly when listed alphabetically. Sorting by date makes it easy to see the latest tag. Often I just want to know what the latest tag or the latest stable release is, this makes it easy.
The alias code is:
   tags = "!sh -c ' \  
 git for-each-ref --format=\"%(refname:short)\" refs/tags | \  
 while read tag; do \  
   git --no-pager log -1 --format=format:\"$tag %at\" $tag; echo; \  
 done | \  
 sort -k 2 | cut -f 1 --delimiter=\" \" | \  
 while read tag; do \  
   fmt=\"%Cred$tag:%Cblue %s %Cgreen%h%Creset (%ai)\"; \  
   git --no-pager log -1 --format=format:\"$fmt\" $tag; echo; \  

Find the first tag that contains a commit

This seems to come up quite a bit for me. An example is here; a user was asking about a virt-install feature, and I wanted to tell them what version it appeared in. I grepped git log, found the commit, then ran:

 $ git describe --contains 87a611b5470d9b86bf57a71ce111fa1d41d8e2cd  

That shows me that v1.0.0 was the first release with the feature they wanted, just take whatever is to the left of the tilde.

This often comes in handy with backporting as well: a developer will point me at a bug fix commit ID, I run git describe to see what upstream version it was released in, so I know what fedora package versions are lacking the fix.

Another tip here is to use the --match option to only search tags matching a particular glob. I've used this to filter out matching against a maintenance or bugfix release branch, when I only wanted to search major version releases.

Don't pull tags from certain remotes

For certain repos like qemu.git, I add a lot of git remotes pointing to individual developer's trees for occasional patch testing. However if trees have lots of non-upstream tags, like for use with pull-requests, they can interfere with my workflow for backporting patches. Use the --no-tags option for this:

 $ git remote add --no-tags $repo

Friday, January 15, 2016

Using CPU host-passthrough with virt-manager

I described virt-manager's CPU model default in this post. In that post I explained the difficulties of using either of the libvirt options for mirroring the host CPU: mode=host-model still has operational issues, and mode=host-passthrough isn't recommended for use with libvirt over supportability concerns.

Unfortunately since writing that post the situation hasn't improved any, and since host-passthrough is the only reliably way to expose the full capabilities of the host CPU to the VM, users regularly want to enable it. This is particularly apparent if trying to do nested virt, which often doesn't work on Intel CPUs unless host-passthrough is used.

However we don't explicitly expose this option in virt-manager since it's not generally recommended for libvirt usage. You can however still enable it in virt-manager:
  • Navigate to VM Details->CPU
  • Enter host-passthrough in the CPU model field
  • Click Apply

Wednesday, January 13, 2016

github 'hub' command line tool

I don't often need to contribute patches to code hosted on github; most of the projects I contribute to are either old school and don't use github for anything but mirroring their main git repo, or are small projects I entirely maintain so I don't submit pull-requests.

But when I do need to submit patches, github's hub tool makes my life a lot simpler, which allows forking repositories and submitting pull-requests very easily from the command line.

The 'hub' tool wants to be installed as an alias for 'git'. I originally tried that, but it made my bash prompt insanely slow since I show the current git branch and dirty state in my bash prompt. When I first encountered this, I filed a bug against the hub tool (with a bogus workaround), and nowadays it seems they have a disclaimer in their README.

Their recommended fix is to s/git/command git/g in, which doesn't work too well if you use the linked fedora suggestion of pointing at the package installed file in /usr/share, so I avoid the alias. You can run 'hub' standalone, but instead I like to do:

 sudo dnf install hub  
 ln -s /usr/bin/hub /usr/libexec/git-core/git-hub  

Then I can git hub fork and git hub pull-request all I want :)

Monday, January 11, 2016

qemu:///system vs qemu:///session

If you've spent time using libvirt apps like virt-manager, you've likely seen references to libvirt URIs. The URI is how users or apps tell libvirt what hypervisor (qemu, xen, lxc, etc) to connect to, what host it's on, what authentication method to use, and a few other bits. 

For QEMU/KVM (and a few other hypervisors), there's a concept of system URI vs session URI:
  • qemu:///system: Connects to the system libvirtd instance, the one launched by systemd. libvirtd is running as root, so has access to all host resources. qemu VMs are launched as the unprivileged 'qemu' user, though libvirtd can grant the VM selective access to root owned resources. Daemon config is in /etc/libvirt, VM logs and other bits are stored in /var/lib/libvirt. virt-manager and big management apps like Openstack and oVirt use this by default.
  • qemu:///session: Connects to a libvirtd instance running as the app user, the daemon is auto-launched if it's not already running. libvirt and all VMs run as the app user. All config and logs and disk images are stored in $HOME. This means each user has their own qemu:///session VMs, separate from all other users. gnome-boxes and libguestfs use this by default.

That describes the 'what', but the 'why' of it is a bigger story. The privilege level of the daemon and VMs have pros and cons depending on your usecase. The easiest way to understand the benefit of one over the other is to list the problems with each setup.

qemu:///system runs libvirtd as root, and access is mediated by polkit. This means if you are connecting to it as a regular user (like when launching virt-manager), you need to enter the host root password, which is annoying and not generally desktop usecase friendly. There are ways to work around it but it requires explicit admin configuration.

Desktop use cases also suffer since VMs are running as the 'qemu' user, but the app (like virt-manager) is running as your local user. For example, say you download an ISO to $HOME and want to attach it to a VM. The VM is running as unprivileged user=qemu and can't access your $HOME, so libvirt has to change the ISO file owner to qemu:qemu and virt-manager has to give search access to $HOME for user=qemu. It's a pain for apps to handle, and it's confusing for users, but after dealing with it for a while in virt-manager we've made it generally work. (Though try giving a VM access to a file on a fat32 USB drive that was automounted by your desktop session...)

qemu:///session runs libvirtd and VMs as your unprivileged user. This integrates better with desktop use cases since permissions aren't an issue, no root password is required, and each user has their own separate pool of VMs.

However because nothing in the chain is privileged, any VM setup tasks that need host admin privileges aren't an option. Unfortunately this includes most general purpose networking options.

The default qemu mode in this case is usermode networking (or SLIRP). This is an IP stack implemented in userspace. This has many drawbacks: the VM can not easily be accessed by the outside world, the VM can access talk to the outside world but only over a limited number of networking protocols, and it's very slow.

There is an option for qemu:///session VMs to use a privileged networking setup, via the setuid qemu-bridge-helper. Basically the host admin sets up a bridge, adds it to a whitelist at /etc/qemu/bridge.conf, then it's available for unprivileged qemu instances. By default on Fedora this contains 'virbr0' which is the default virtual network bridge provided by the system libvirtd instance, and what qemu:///system VMs typically use.

gnome-boxes originally used usermode networking, but switched around Fedora 21 timeframe to use virbr0 via qemu-bridge-helper. But that's dependent on virbr0 being set up correctly by the host admin, or via package install (libvirt-daemon-config-network package on Fedora).

qemu:///session also misses some less common features that require host admin privileges, like host PCI device assignment. Also VM autostart doesn't work as expected because the session daemon itself isn't autostarted.

Apps have to decide for themselves which libvirtd mode to use, depending on their use case.

qemu:///system is completely fine for big apps like oVirt and Openstack that require admin access to the virt hosts anyways.

virt-manager largely defaults to qemu:///system because that's what it has always done, and that default long precedes qemu-bridge-helper. We could switch but it would just trade one set of issues for another. virt-manager can be used with qemu:///session though (or any URI for that matter).

libguestfs uses qemu:///session since it avoids all the permission issues and the VM appliance doesn't really care about networking.

gnome-boxes prioritized desktop integration from day 1, so qemu:///session was the natural choice. But they've struggled with the networking issues in various forms.

Other apps are in a pickle: they would like to use qemu:///session to avoid the permission issues, but they also need to tweak the network setup. This is the case vagrant-libvirt currently finds itself in.

Friday, January 8, 2016

Running KVM arm 32 on AArch64

Just a little tip: libvirt 1.2.17 fixed the last bits necessary to run 32bit arm VMs on AArch64 hosts with KVM acceleration. We just needed to make sure that libvirt advertised the capability, all the lower level qemu and kernel bits were already in place.

Just select armv7l in virt-manager's UI when creating a new VM, or pass --arch armv7l to virt-install, if on an aarch64 host, and KVM will be used if it's available.

In my (very brief) testing the VM seems to be much faster than 32-on-32 KVM, but I don't think that's a surprise given the speed difference between the host machines.

Update: Marcin posted some virt-manager screenshots and performance info.

Wednesday, January 6, 2016

tip: Launch a new terminal with F1

Back when I first switched to gnome-shell with Fedora 16, I read a lot of blog posts and wiki pages about keyboard shortcuts, extensions that people were using, praise and complaints, etc. Somewhere in all that (I can't remember where), I picked up a tip that I now use dozens of times a day and couldn't live without: remapping F1 to launch a new terminal.

It didn't even take much training to get used to it, because the method is just so much quicker than anything involving the mouse, or super key, or alt+f2, that I took to it immediately. And from there it was a small step to get used to using exit to actually quit a terminal, leading to drastically less mouse usage (yes my habits were quite crappy on gnome2).

It also has the added benefit of unmapping the help dialog from F1, which I've never once used intentionally, and every unintentional usage (of which there were many) would make my machine churn pretty hard for a good few seconds before popping up the dialog.

It's pretty easy to make the change on F23:

  • From gnome-shell desktop, go to Activities->Keyboard
  • Navigate to Shortcuts->Custom Shortcuts
  • Under launchers, disable 'Launch help browser'
  • Under 'Custom shortcuts', click '+', map F1 to gnome-terminal

I'm sure if you use a single terminal + tmux, or guake, this isn't interesting, but maybe it will help someone like it helped me.

Monday, January 4, 2016

Polkit password-less access for the 'libvirt' group

Many users, who admin their own machines, want to be able to use tools like virt-manager without having to enter a root password. Just google 'virt-manager without password' and see all the hits. I've seen many blogs and articles over the years describing various ways to work around it.

The password prompting is via libvirt's polkit integration. The idea is that we want the applications to run as a regular unprivileged user (running GUI apps as root is considered a no-no), and only use the root authentication for talking to system libvirt instance. Most workarounds suggest installing a polkit rule to allow your user, or a particular user group, to access libvirt without needing to enter the root password.

In libvirt v1.2.16 we finally added official support for this (and backported to Fedora22+). The group is predictably called 'libvirt'. This matches polkit rules that debian and suse were already shipping too.

So just add your user to the 'libvirt' group and enjoy passwordless virt-manager usage:

$ usermod --append --groups libvirt `whoami`

Saturday, January 2, 2016

UEFI support in virt-install and virt-manager

One of the new features in virt-manager 1.2.0 (from back in May) is user friendly support for enabling UEFI.

First a bit about terminology: When UEFI is packaged up to run in an x86 VM, it's often called OVMF. When UEFI is packaged up to run in an AArch64 VM, it's often called AAVMF. But I'll just refer to all of it as UEFI.

Using UEFI with virt-install and virt-manager

The first step to enable this for VMs is to install the binaries. UEFI still has some licensing issues that make it incompatible with Fedora's policies, so the bits are hosted in an external repo. Details for installing the repo and UEFI bits are over here.

Once the bits are installed (and you're on Fedora 22 or later), virt-manager and virt-install provide simple options to enable UEFI when creating VMs.

Marcin has a great post with screenshots describing this for virt-manager (for aarch64, but the steps are identical for x86 VMs).

For virt-install it's as simple as doing:

$ sudo virt-install --boot uefi ...

virt-install will get the binary paths from libvirt and set everything up with the optimal config. If virt-install can't figure out the correct parameters, like if no UEFI binaries are installed, you'll see an error like: ERROR    Error: --boot uefi: Don't know how to setup UEFI for arch 'x86'

See 'virt-install --boot help' if you need to tweak the parameters individually.

Implementing support in applications

Libvirt needs to know about UEFI<->NVRAM config file mapping, so it can advertise it to tools like virt-manager/virt-install. Libvirt looks at a hardcoded list of known host paths to see if any firmware is installed, and if so, lists those paths in domain capabilities output (virsh domcapabilities). Libvirt in Fedora 22+ knows to look for the paths provided by the repo mentioned above, so just installing the firmware is sufficient to make libvirt advertise UEFI support.

The domain capabilities output only lists the firmware path and the associated variable store path. Notably lacking is any indication of what architecture the binaries are meant for. So tools need to determine this mapping themselves.

virt-manager/virt-install and libguestfs use a similar path matching heuristic. The libguestfs code is a good reference:

    match guest_arch with
    | "i386" | "i486" | "i586" | "i686" ->
       [ "/usr/share/edk2.git/ovmf-ia32/OVMF_CODE-pure-efi.fd",
         "/usr/share/edk2.git/ovmf-ia32/OVMF_VARS-pure-efi.fd" ]
    | "x86_64" ->
       [ "/usr/share/OVMF/OVMF_CODE.fd",
         "/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd" ]
    | "aarch64" ->
       [ "/usr/share/AAVMF/AAVMF_CODE.fd",
         "/usr/share/edk2.git/aarch64/vars-template-pflash.raw" ]
    | arch ->
       error (f_"don't know how to convert UEFI guests for architecture %s")
             guest_arch in

Having to track this in every app is quite crappy, but it's the only good solution at the moment. Hopefully long term libvirt will grow some solution that makes this easier for applications.