Monday, January 11, 2016

qemu:///system vs qemu:///session

If you've spent time using libvirt apps like virt-manager, you've likely seen references to libvirt URIs. The URI is how users or apps tell libvirt what hypervisor (qemu, xen, lxc, etc) to connect to, what host it's on, what authentication method to use, and a few other bits. 

For QEMU/KVM (and a few other hypervisors), there's a concept of system URI vs session URI:
  • qemu:///system: Connects to the system libvirtd instance, the one launched by systemd. libvirtd is running as root, so has access to all host resources. qemu VMs are launched as the unprivileged 'qemu' user, though libvirtd can grant the VM selective access to root owned resources. Daemon config is in /etc/libvirt, VM logs and other bits are stored in /var/lib/libvirt. virt-manager and big management apps like Openstack and oVirt use this by default.
  • qemu:///session: Connects to a libvirtd instance running as the app user, the daemon is auto-launched if it's not already running. libvirt and all VMs run as the app user. All config and logs and disk images are stored in $HOME. This means each user has their own qemu:///session VMs, separate from all other users. gnome-boxes and libguestfs use this by default.

That describes the 'what', but the 'why' of it is a bigger story. The privilege level of the daemon and VMs have pros and cons depending on your usecase. The easiest way to understand the benefit of one over the other is to list the problems with each setup.

qemu:///system runs libvirtd as root, and access is mediated by polkit. This means if you are connecting to it as a regular user (like when launching virt-manager), you need to enter the host root password, which is annoying and not generally desktop usecase friendly. There are ways to work around it but it requires explicit admin configuration.

Desktop use cases also suffer since VMs are running as the 'qemu' user, but the app (like virt-manager) is running as your local user. For example, say you download an ISO to $HOME and want to attach it to a VM. The VM is running as unprivileged user=qemu and can't access your $HOME, so libvirt has to change the ISO file owner to qemu:qemu and virt-manager has to give search access to $HOME for user=qemu. It's a pain for apps to handle, and it's confusing for users, but after dealing with it for a while in virt-manager we've made it generally work. (Though try giving a VM access to a file on a fat32 USB drive that was automounted by your desktop session...)

qemu:///session runs libvirtd and VMs as your unprivileged user. This integrates better with desktop use cases since permissions aren't an issue, no root password is required, and each user has their own separate pool of VMs.

However because nothing in the chain is privileged, any VM setup tasks that need host admin privileges aren't an option. Unfortunately this includes most general purpose networking options.

The default qemu mode in this case is usermode networking (or SLIRP). This is an IP stack implemented in userspace. This has many drawbacks: the VM can not easily be accessed by the outside world, the VM can access talk to the outside world but only over a limited number of networking protocols, and it's very slow.

There is an option for qemu:///session VMs to use a privileged networking setup, via the setuid qemu-bridge-helper. Basically the host admin sets up a bridge, adds it to a whitelist at /etc/qemu/bridge.conf, then it's available for unprivileged qemu instances. By default on Fedora this contains 'virbr0' which is the default virtual network bridge provided by the system libvirtd instance, and what qemu:///system VMs typically use.

gnome-boxes originally used usermode networking, but switched around Fedora 21 timeframe to use virbr0 via qemu-bridge-helper. But that's dependent on virbr0 being set up correctly by the host admin, or via package install (libvirt-daemon-config-network package on Fedora).

qemu:///session also misses some less common features that require host admin privileges, like host PCI device assignment. Also VM autostart doesn't work as expected because the session daemon itself isn't autostarted.

Apps have to decide for themselves which libvirtd mode to use, depending on their use case.

qemu:///system is completely fine for big apps like oVirt and Openstack that require admin access to the virt hosts anyways.

virt-manager largely defaults to qemu:///system because that's what it has always done, and that default long precedes qemu-bridge-helper. We could switch but it would just trade one set of issues for another. virt-manager can be used with qemu:///session though (or any URI for that matter).

libguestfs uses qemu:///session since it avoids all the permission issues and the VM appliance doesn't really care about networking.

gnome-boxes prioritized desktop integration from day 1, so qemu:///session was the natural choice. But they've struggled with the networking issues in various forms.

Other apps are in a pickle: they would like to use qemu:///session to avoid the permission issues, but they also need to tweak the network setup. This is the case vagrant-libvirt currently finds itself in.


  1. Thanks for this blog post, it makes quite a few things clearer :)

    However, it raises some questions.

    GNOME Boxes uses the session libvirt, but because of the network, it requires the system libvirt to be running, and the virbr0 interface to have been setup.

    On a laptop, it turns out that having the system libvirtd running and the virbr0 interface up consumes quite a bit of battery. A friend who actually measured it said it was significant and he improved his battery life quite a bit by disabling it.

    However, that means he now has to start libvirtd manually before using Boxes.

    Could that all be automatic, though?

    Ideally libvirtd wouldn't be running, the virbr0 interface wouldn't be setup, until you start a VM in Boxes. libvirtd would probably need to be socket-/dbus-activated, I guess?

    1. Hmm, that's interesting that system libvirtd consumes battery just by running, I would have assumed it was sitting idle most of the time. Did the person who tested battery usage post their results anywhere? Might be interesting to see if it's libvirtd's fault, or virbr0+iptables.

      Re: socket activation, that's a nice idea, but currently libvirt needs a bit of work to use socket activation, at least the VM autostart feature is implemented in such a way that it always expects the daemon to be run at machine startup, but that could be fixed I imagine.

      Though really boxes implicitly depending on system libvirtd to have setup networking is not the ideal layout. IMO NetworkManager should provide support for sharing the host network with VMs, which could be implemented in different ways. But I don't know if that will ever work, and it's a lot more effort

  2. Awesome post, thanks.
    I have system libvirtd daemon enabled just to have virbr0 up and running, so VMs started from qemu:///session can tap into it.
    I would love to configure the bridge myself (NetworkManager or otherwise) and get rid of system libvirtd daemon. Not sure how exactly libvirtd does that.
    Would it be possible to fill brctl section on this wiki page?