Cole Robinson

Better output with /usr/bin/bugzilla --json

2020-10-03T00:00:00-04:00

In python-bugzilla 2.4.0 I added a --json output switch to /usr/bin/bugzilla.

IMO this is the optimal way to process large amounts of bugzilla query output from the command laine. so if you are already doing just that with the old output options, consider switching to --json. That is, if you aren't ready to make the full leap to using the library directly ;)

Replace usage of `--raw`

The older --raw output mode will print bug contents in a strange custom array-like output format that is poorly specified and impossible to parse reliably. Please use --json instead. --raw may be removed in the future.

Replace usage of `--outputformat`

The older --outputformat option processes bug contents and prints them out in the specified string format. Variables are referenced with something akin to RPM macro format. Example:

$ bugzilla query --outputformat '%{id}::%{summary}' --id 1234567
1234567::Package should not ship a separate emacs sub-package

This has served us pretty well, but besides being a custom and poorly specified format, it has limitations with formatting subvalues of non-string bugzilla fields.

With --json you can make use of the powerful jq tool which gives you the ability to do all manner of querying and processing of the output.

This example duplicates the --outputformat example above:

$ bugzilla query --json --id 1234567 | jq -r '.bugs[0] | "\(.id)::\(.summary)"'
1234567::Package should not ship a separate emacs sub-package

This example demonstrates accessing a structured subfield like comments. We print the first comment's creation time for a list of bugs:

$ bugzilla query --json \
    --product Fedora --component virt-manager --status OPEN | \
    jq -r ".bugs[].comments[0].creation_time"
2020-07-08T16:35:14Z
2020-08-30T14:22:42Z
2020-04-12T20:28:24Z
2020-02-05T09:57:48Z
2020-09-09T15:17:32Z
2019-12-28T22:29:05Z
2020-08-15T21:20:02Z
2020-09-21T13:59:37Z
2020-09-18T08:50:28Z

Optimizing `--json` queried data

One downside of --json compared to --outputformat is that it will return all bug data even if you only plan to parse a subset of it. For big queries, or ones performed repeatedly, this can cause unnecessary load on the bugzilla instance.

--outputformat knows what fields you need, so will only fetch those from the bugzilla instance, transferring the minimally required amount of data. You can achieve the same thing manually with --json by also specifying the option --includefield field1 --includefield field2 .. for every field you need to process. Example:

$ bugzilla query --json --includefield summary --id 1234567
{
  "bugs": [
    {
      "id": 1234567,
      "summary": "Package should not ship a separate emacs sub-package"
    }
  ]
}

As a more general tip, use the bugzilla web UI to generate a query that only checks for changes after a certain timestamp that is relevant to you, and pass that to query --from-url $URL, so you can limit the time window for the data you are fetching.

Using `--json` to find bugzilla API field names

This applies to --raw as well, but you can use --json to find out the API name of custom fields in the bugzilla instance. Many times I've been asked questions like: 'How do I access bugzilla.redhat.com field FOOBAR from python-bugzilla?'. The simplest way to figure it out is to find a bug that you know has data filled in for that field, pass that bug ID to bugzilla query --json --id XXX and examine the output to find the field name. If they are custom fields they usually have a prefix like cf_, at least for bugzilla.redhat.com.

virt-install --cloud-init support

2020-09-26T00:00:00-04:00

As part of GSOC 2019, Athina Plaskasoviti implemented --cloud-init support for virt-install. This post provides a bit more info about the feature.

Why cloud-init

For a long while, most mainstream Linux distros have shipped 'cloud images': raw or qcow2 formatted disk images with the distro minimally pre-installed on it. These images typically have cloud-init set to run on VM bootup. cloud-init can do a variety of things, like add users, change passwords, register ssh keys, and generally perform any desired action on the VM OS. This only works when cloud-init is passed the right configuration by whatever platform is starting the VM, like OpenStack or virt-install. cloud-init supports many different datasources for getting configuration outside the VM.

Historically though the problem is that slapping these images into virt-install or virt-manager gives crappy results, because these tools were not providing any datasource. In this case, cloud-init reverts to its distro default configured behavior, which in most cases is unusable. For example on Fedora, the result was: hang waiting for cloud-init data, time out, drop to login prompt with all accounts locked.

Prior to virt-install --cloud-init support, the simplest workaround was to use libguestfs, specifically virt-customize, to set a root password and disable cloud-init:

$ virt-customize -a MY-CLOUD-IMAGE.qcow2 \
    --root-password password:SUPER-SECRET-PASSWORD \
    --uninstall cloud-init

--cloud-init option

The --cloud-init option will tell virt-install to set up a nocloud datasource via a specially formatted .iso file that is generated on the fly, and only used for the first VM bootup.

The default behavior when --cloud-init is specified with no suboptions will do the following:

Generate a random root password and print it to the terminal.
Default to VM serial console access rather than graphical console access. Makes it easier to paste the password and gives more ssh-like behavior.
Sets the root password to expire on first login. So the temporary password is only used once.
Disables cloud-init for subsequent VM startups. Otherwise on the next VM boot you'd face locked accounts again.

The --cloud-init also has suboptions to specify your own behavior, like transfer in a host ssh public key, pass in raw cloud-init user-data/meta-data, etc. See the virt-install --cloud-init man page section for the specifics.

Room for improvement

This is all a step in the right direction but there's lots more we can do here:

Extend virt-install's --install option to learn how to fetch cloud images for the specified distro. libosinfo and osinfo-db already track cloud image download links for many distros so the info we need is already in place. We could make virt-install --install fedora32,cloud=yes a single way to pull down a cloud image, generate a cloud-init datasource, and create the VM in one shot.
Use --cloud-init by default when the user passes us a cloud-init enabled disk image. virt-customize has a lot of disk image detection smarts already, but we aren't using that in virt-install yet.
virt-manager UI support. There's an issue tracking this with some more thoughts in it.
Similar support for CoreOS Ignition which fulfills a similar purpose as cloud-init for distros like Fedora CoreOS. There's an issue tracking this too.

I personally don't have plans to work on these any time soon, but I'm happy to provide guidance if anyone is interested in helping out!

virt-manager 3.0.0 released!

2020-09-16T00:00:00-04:00

Yesterday I released virt-manager 3.0.0. Despite the major version number bump, things shouldn't look too different from the previous release. For me the major version number bump reflects certain feature removals (like dropping virt-convert), and the large amount of internal code changes that were done, though there's a few long awaited features sprinkled in like virt-install --cloud-init support which I plan to write more about later.

This release includes:

virt-install --cloud-init support (Athina Plaskasoviti, Cole Robinson)
The virt-convert tool has been removed. Please use virt-v2v instead
A handful of UI XML configuration options have been removed, in an effort to reduce maintenance ongoing maintenance burden, and to be more consistent with what types of UI knobs we expose. The XML editor is an alternative in most cases. For a larger discussion see this thread and virt-manager's DESIGN.md file.
The 'New VM' UI now has a 'Manual Install' option which creates a VM without any required install media
In the 'New VM' UI, the network/pxe install option has been removed. If you need network boot, choose 'Manual Install' and set the boot device after initial VM creation
'Clone VM' UI has been reworked and simplified
'Migrate VM' UI now has an XML editor for the destination VM
Global and per-vm option to disable graphical console autoconnect. This makes it easier to use virt-manager alongside another client like virt-viewer
virt-manager: set guest time after VM restore (Michael Weiser)
virt-manager: option to delete storage when removing disk device (Lily Nie)
virt-manager: show warnings if snapshot operation is unsafe (Michael Weiser)
Unattended install improvements (Fabiano Fidêncio)
cli: new --xml XPATH=VAL option for making direct XML changes
virt-install: new --reinstall=DOMAIN option
virt-install: new --autoconsole text|graphical|none option
virt-install: new --os-variant detect=on,require=on suboptions
cli: --clock, --keywrap, --blkiotune, --cputune additions (Athina Plaskasoviti)
cli: add --features kvm.hint-dedicated.state= (Menno Lageman)
cli: add --iommu option (Menno Lageman)
cli: Add --graphics websocket= support (Petr Benes)
cli: Add --disk type=nvme source.* suboptions
cli: Fill in all --filesystem suboptions
Translation string improvements (Pino Toscano)
Convert from .pod to .rst for man pages
Switch to pytest as our test runner
Massively improved unittest and uitest code coverage
Now using github issues as our bug tracker

virt-manager libvirt XML editor UI

2020-07-12T00:00:00-04:00

virt-manager 2.2.0 was released in June of last year. It shipped with a major new feature: libvirt XML viewing and editing UI for new and existing domain, pools, volumes, networks.

Every VM, network, and storage object page has a XML tab at the top. Here's an example with that tab selected from the VM Overview section:

Here's an example of the XML view when just a disk is selected. Note it only shows that single device's libvirt XML:

By default the XML is not editable; notice the warning at the top of the first image. After editing is enabled, the warning is gone, like in the second image. You can enable editing via Edit->Preferences from the main Manager window. Here's what the option looks like:

A bit of background: We are constantly receiving requests to expose libvirt XML config options in virt-manager's UI. Some of these knobs are necessary for <1% but uninteresting to the rest. Some options are difficult to set from the command line because they must be set at VM install time, which means switch from virt-manager to virt-install which is not trivial. And so on. When these options aren't added to the UI, it makes life difficult for those affected users. It's also difficult and draining to have these types of justification conversations on the regular.

The XML editing UI was added to relieve some of the pressure on virt-manager developers fielding these requests, and to give more power to advanced virt users. The users that know they need an advanced option are usually comfortable editing the libvirt XML directly. The XML editor doesn't detract from the existing UI much IMO, and it is uneditable by default to prevent less knowledgeable users from getting into trouble. It ain't gonna win any awards for great UI, but the feedback has been largely positive so far.

virt-convert tool removed in virt-manager.git

2020-07-11T00:00:00-04:00

The next release of virt-manager will not ship the virt-convert tool, I removed it upstream with this commit.

Here's the slightly edited quote from my original proposal to remove it:

virt-convert takes an ovf/ova or vmx file and spits out libvirt XML. It started as a code drop a long time ago that could translate back and forth between vmx, ovf, and virt-image, a long dead appliance format. In 2014 I changed virt-convert to do vmx -> libvirt and ovf -> libvirt which was a CLI breaking change, but I never heard a peep of a complaint. It doesn't do a particularly thorough job at its intended goal, I've seen 2-3 bug reports in the past 5 years and generally it doesn't seem to have any users. Let's kill it. If anyone has the desire to keep it alive it could live as a separate project that's a wrapper around virt-install but there's no compelling reason to keep it in virt-manager.git IMO

That mostly sums it up. If there's any users of virt-convert out there, you likely can get similar results by extracting the relevant disk image from the .vmx or .ovf config, pass it to virt-manager or virt-install, and let those tools fill in the defaults. In truth that's about all virt-convert did in to begin with.

Please see virt-v2v for an actively maintained tool that can covert OVA/OVF appliances to libvirt + KVM. This redhat.com article describes an example conversion.

python-bugzilla REST API support

2020-06-29T13:00:00-04:00

I just released python-bugzilla 2.4.0. The main interesting bit it adds is support for Bugzilla's REST API.

All previous versions of python-bugzilla and /usr/bin/bugzilla only used the XMLRPC API, but that is deprecated in Bugzilla 5.0+ and all new API development is taking place on the REST API.

In practice there isn't any released bugzilla version that has big differences between the two API versions. On bugzilla.redhat.com specifically the XMLRPC API is still recommended, because some custom features are not available over REST yet. Note though that bugzilla.mozilla.org is looking at disabling the XMLRPC API entirely, but they are usually ahead of the Bugzilla curve.

By default, python-bugzilla will use some URL parsing heuristics to try to guess if the passed URL is for XMLRPC or REST.

If URL contains /xmlrpc, assume XMLRPC
If URL contains /rest, assume REST
If the URL does not contain a path:
- Try appending /xmlrpc.cgi and if the URL exists, use it
- Try appending /rest and if the URL exists, use it
Otherwise just attempt to initialize the XMLRPC backend

In practice this should mean previously used URLs will continue to use XMLRPC.

The Bugzilla API class also force_rest and force_xmlrpc init arguments to force use of a specific API for the passed URL. Whether REST or XMLRPC is used the existing API should continue to work as expected, it's simply a backend detail.

From /usr/bin/bugzilla there aren't any explicit command line options to force use of one API or the other. If you pass a URL with an explicit /rest or /xmlrpc.cgi then the API will pick the correct backend based on the heuristic above. If your URL has a weirdly named REST API endpoint you can probably trick the heuristic with a funky URL like https://fakebz.example.com/weird-rest-endpoint?/rest

virt-manager is deprecated in RHEL (but only RHEL)

2020-06-16T13:00:00-04:00

TL;DR: I'm the primary author of virt-manager. virt-manager is deprecated in RHEL8 in favor of cockpit, but ONLY in RHEL8 and future RHEL releases. The upstream project virt-manager is still maintained and is still relevant for other distros.

Google 'virt-manager deprecated' and you'll find some discussions suggesting virt-manager is no longer maintained, Cockpit is replacing virt-manager, virt-manager is going to be removed from every distro, etc. These conclusions are misinformed.

The primary source for this confusion is the section 'virt-manager has been deprecated' from the RHEL8 release notes virtualization deprecation section. Relevant quote from the RHEL8.2 docs:

The Virtual Machine Manager application, also known as virt-manager, has been deprecated. The RHEL 8 web console, also known as Cockpit, is intended to become its replacement in a subsequent release.

What that means:

virt-manager is in RHEL8 and will be there for the lifetime of RHEL8.
Red Hat engineering effort assigned to the virt-manager UI has been reduced compared to previous RHEL versions.
The tentative plan is to not ship the virt-manager UI in RHEL9.

Why is this happening? As I understand it, RHEL wants to roughly standardize on Cockpit as their host admin UI tool. It's a great project with great engineers and great UI designers. Red Hat is going all in on it for RHEL and aims to replace the mismash of system-config-X tools and project specific admin frontends (like virt-manager) with a unified project. (Please note: this is my paraphrased understanding, I'm not speaking on behalf of Red Hat here.)

Notice though, this is all about RHEL. virt-manager is not deprecated upstream, or deprecated in other distros automatically just because RHEL has made this decision. The upstream virt-manager project continues on and Red Hat continues to allocate resources to maintain it.

Also, I'm distinguishing virt-manager UI from the virt-manager project, which includes tools like virt-install. I fully expect virt-install to be shipped in RHEL9 and actively maintained (FWIW Cockpit uses it behind the scenes).

And even if the virt-manager UI is not in RHEL9 repos, it will likely end up shipped in EPEL, so the UI will still be available for install, just through external repos.

Overall my personal opinion is that as long as libvirt+KVM is in use on linux desktops and servers, virt-manager will be relevant. I don't expect anything to change in that area any time soon.

Blog moved to Pelican and GitHub Pages

2019-07-30T15:30:00-04:00

I've moved my blog from blogger.com to a static site generated with Pelican and hosted on GitHub Pages. This is a dump of some of the details.

The content is hosted in three branches across two repos:

blog/gh-pages: https://blog.wikichoon.com HTML content
crobinso.github.io/master: https://wikichoon.com front page HTML content
crobinso.github.io/pelican: website source content and theme

The motivation for the split is that according to this pelican SEO article, master branches of GitHub repos are indexed by google, so if you store HTML content in a master branch your canonical blog might be battling your GitHub repo in the search results. And since you can only put content in the master branch of a $username.github.io repo, I added a separate blog.git repo. Maybe I could shove all the content into the blog/gh-pages branch I think dealing with multiple subdomains prevents it. I've already spent too much timing playing with all this stuff though so that's for another day to figure out. Of course, suggestions welcome, blog comments are enabled with Disqus.

One issue I hit is that pushing updated content to blog/gh-pages doesn't consistently trigger a new GitHub Pages deployment. There's a bunch of hits about this around the web (this stackoverflow post in particular) but no authoritative explanation about what criteria GitHub Pages uses to determine whether to redeploy. The simplest 'fix' I found is to tweak the index.html content via the GitHub web UI and commit the change which seems to consistently trigger a refresh as reported by the repo's deployments page.

You may notice the blog looks a lot like stock Jekyll with its minima theme. I didn't find any Pelican theme that I liked as much as minima, so I grabbed the CSS from a minima instance and started adapting the Pelican simple-bootstrap theme to use it. The end result is basically a simple reimplementation of minima for Pelican. I learned a lot in the process but it likely would have been much simpler if I just used Jekyll in the first place, but I'm in too deep to switch now!

Host 'Network Interfaces' panel removed from virt-manager

2019-04-09T14:01:00-04:00

I released virt-manager 2.0.0 in October 2018. Since the release contained the full port to python3, it seemed like a good opportunity to drop some baggage from the app.

The biggest piece we removed was the UI for managing host network interfaces. This is the Connection Details->Network Interfaces panel, and the New Interface wizard for defining host network definitions for things like bridges, bonds, and vlan devices. The main screen of the old UI looked like this:

Some history

Behind the scenes, this UI was using libvirt's Interface APIs, which also power the virsh iface-* commands. These APIs are little more than a wrapper around the netcf library.

netcf aimed to be a linux distro independent API for network device configuration. On Red Hat distros this meant turning the API's XML format into an /etc/sysconfig/network script. There were even pie-in-the-sky ideas about NetworkManager one day using netcf.

In practice though the library never really took off. It was years before a debian backend showed up, contributed by a Red Hatter in the hope of increasing library uptake, though it didn't seem to help. netcf basically only existed to serve the libvirt Interface APIs, yet those APIs were never really used by any major libvirt consuming app, besides virt-manager. And in virt-manager's case it was largely just slapping some UI over the XML format and lifecycle operations.

For virt-manager's usecases we hoped that netcf would make it trivial to bridge the host's network interface, which when used with VMs would give them first class IP addresses on the host network setup, not NAT like the 'default' virtual network. Unfortunately though the UI would create the ifcfg files well enough, behind the scenes nothing played well with NetworkManager for years and years. The standard suggestion for was to disable NetworkManager if you wanted to bridge your host NIC. Not very user friendly. Some people did manage to use the UI to that effect but it was never a trivial process.

The state today

Nowadays NetworkManager can handle bridging natively and is much more powerful than what virt-manager/libvirt/netcf provide. The virt-manager UI was more likely to shoot you in the foot than make things simple. And it had become increasingly clear that virt-manager was not the place to maintain host network config UI.

So we made the decision to drop all this from virt-manager in 2.0.0. netcf and the libvirt interface APIs still exist. If you're interested in some more history on the interface API/netcf difficulties, check out Laine's email to virt-tools-list.

python-bugzilla + bugzilla 5.0 API keys

2019-01-09T17:58:00-05:00

For many uses of /usr/bin/bugzilla and python-bugzilla, it's necessary to actually be logged in to a bugzilla server. Creating bugs, editing bugs, querying private data, etc.

Up until now anyone that's used the command line tool has periodically had to do a bugzilla login to refresh their authentication cache. In older bugzilla versions this was an HTTP cookie, more recently it's a bugzilla API token. Generally login calls were needed infrequently on a single machine as tokens would remain valid for a long time.

Recently, bugzilla.redhat.com received a big update to bugzilla 5.0. However with that update it seems like API tokens now expire after a week, which has necessitated lots more bugzilla login calls than I'm used to.

Thankfully with bugzilla 5.0 and later there's a better option: API keys. Here's how to to use them transparently with /usr/bin/bugzilla and all python-bugzilla library usage. Here's steps for enabling API keys with bugzilla.redhat.com, but the same process should roughly apply to other bugzilla instances too.

Login to the bugzilla web UI
Click on your email
Select Preferences
Select API Keys
Generate an API key with an optional comment like python-bugzilla

Afterwards the screen will look something like this (updated in June 2020):

MY-FAKE-KEY is not my actual key, I replaced it for demo purposes. The actual key is a long string of characters and numbers.

On bugzilla.redhat.com the key is only visible in the red box when the key is initially created; refreshing the page will only show the numeric API key ID which is not what you want. So if you missed copying that value the first time, revoke the key and create a new one.

Once you've copied your key, write a bugzillarc file like this:

$ cat ~/.config/python-bugzilla/bugzillarc
[bugzilla.redhat.com]
api_key=MY-FAKE-KEY

That's it, /usr/bin/bugzilla and python-bugzilla using tools should pick it up automagically. Note, API keys are as good as passwords in certain ways, so treat it with the same secrecy you would treat a password.