Monday, December 15, 2014

Setting up a minimal rbd/ceph server for libvirt testing

In my last post I talked about setting up a minimal gluster server. Similarly this will describe how I set up a minimal single node rbd/ceph server in a VM for libvirt network storage testing.

I pulled info from a few different places and a lot of other reading, but things just weren't working on F21; trying systemctl start ceph just wasn't producing any output, and all the ceph cli commands just hung. I had better success with F20.

The main difficulty was figuring out a working ceph.conf. My VM's IP address is 1902.168.124.101, and its hostname is 'localceph', so here's what I ended up with:

auth cluster required = none
auth service required = none
auth client required = none
osd crush chooseleaf type = 0
osd pool default size = 1

mon data = /data/$name
mon addr =
host = localceph

keyring = /data/keyring.$name
host = localceph

osd data = /data/$name
osd journal = /data/$name/journal
osd journal size = 1000
host = localceph

Ceph setup steps:
  • Cloned an existing F20 VM I had kicking around, using virt-manager's clone wizard. I called it f20-ceph.
  • In the VM, disable firewalld and set selinux to permissive. Not strictly required but I wanted to make this as simple as possible.
  • Setup the ceph server:
    • yum install ceph
    • I needed to set a hostname for my VM, ceph won't accept 'localhost': hostnamectl set-hostname localceph
    • mkdir -p /data/mon.0 /data/osd.0
    • Overwrite /etc/ceph/ceph.conf with the content listed above.
    • mkcephfs -a -c /etc/ceph/ceph.conf
    • service ceph start
    • Prove it works from my host with: sudo mount -t ceph $VM_IPADDRESS:/ /mnt
  • Add some storage for testing:
    • Libvirt only connects to Ceph's block device interface, RBD. The above mount example is _not_ what libvirt will see, it just proves we can talk to the server.
    • Import files within the VM like: rbd import $filename
    • List files with: rbd list
Notable here is that no ceph auth is used. Libvirt supports ceph auth but at this stage I didn't want to deal with it for testing. This setup doesn't match what a real deployment would ever look like.

Here's the pool definition I passed to virsh pool-define on my host:

<pool type='rbd'>
    <host name='$VM_IPADDRESS'/>


  1. Annoyingly the 'mkcephfs' command has been deleted from the ceph packages, with people told to use 'ceph-deploy' instead. I've written a new blog post showing how todo a single node deploy with the new tool

    1. Cool, thanks. Though I've been meaning to poke at this again to set up ceph auth since I never wired up support for that in virt-install/virt-manager