Openstack 2024.1 Caracal and Ceph Reef Integration

In this article I will show you how to integrate OpenStack and Ceph using Kolla-Ansible. OpenStack will store service data like Glance, Nova, and Cinder inside Ceph, this allows for high availability as Ceph will replicate the data. I used Kolla-Ansible to automate the integration of openstack and ceph.

So, let's get started…

Environment

• OpenStack Environment

• Ceph Environment

OpenStack and Ceph Integration

1. Mapping hosts

nano /etc/hosts
---
10.10.11.11 at-openstack-aio
10.10.11.21 at-ceph-aio

2. Create and init pools

# exec on at-ceph-aio
ceph osd pool create volumes
ceph osd pool create images
ceph osd pool create vms
rbd pool init volumes
rbd pool init images
rbd pool init vms

3. Create keyrings

# exec on at-ceph-aio
ceph auth get-or-create client.glance mon 'profile rbd' osd 'profile rbd pool=images' mgr 'profile rbd pool=images'
ceph auth get-or-create client.cinder mon 'profile rbd' osd 'profile rbd pool=volumes, profile rbd pool=vms, profile rbd-read-only pool=images' mgr 'profile rbd pool=volumes, profile rbd pool=vms'

4. Copy ceph keyrings and configuration to openstack node

# exec on at-openstack-aio
mkdir /etc/ceph

# exec on at-ceph-aio
ssh at-openstack-aio tee /etc/ceph/ceph.conf </etc/ceph/ceph.conf
ceph auth get-or-create client.glance | ssh at-openstack-aio sudo tee /etc/ceph/ceph.client.glance.keyring
ceph auth get-or-create client.cinder | ssh at-openstack-aio sudo tee /etc/ceph/ceph.client.cinder.keyring

Remove tabs indentation on ceph.conf, issue

# exec on at-openstack-aio
nano /etc/ceph/ceph.conf
---
# minimal ceph.conf for 62a54df8-c3f7-11ef-8cc8-cf6b30fc4ac0
[global]
fsid = 62a54df8-c3f7-11ef-8cc8-cf6b30fc4ac0
mon_host = [v2:10.10.11.21:3300/0,v1:10.10.11.21:6789/0]

5. Copy ceph keyrings and configuration to openstack directory

# exec on at-openstack-aio
mkdir -p /etc/kolla/config/{glance,cinder,nova}
mkdir /etc/kolla/config/cinder/cinder-volume
cp /etc/ceph/ceph.conf /etc/kolla/config/cinder/ 
cp /etc/ceph/ceph.conf /etc/kolla/config/nova/ 
cp /etc/ceph/ceph.conf /etc/kolla/config/glance/ 
cp /etc/ceph/ceph.client.glance.keyring /etc/kolla/config/glance/ 
cp /etc/ceph/ceph.client.cinder.keyring /etc/kolla/config/cinder/cinder-volume/
cp /etc/ceph/ceph.client.cinder.keyring /etc/kolla/config/nova/

6. Update kolla main configuration

# exec on at-openstack-aio
nano /etc/kolla/globals.yml
---
## BASE
kolla_base_distro: "ubuntu"
kolla_install_type: "source"
openstack_release: "2024.1"

## NETWORK
kolla_internal_vip_address: "10.10.11.100"
kolla_internal_fqdn: "internal.at.lab"
kolla_external_vip_address: "10.10.11.200"
kolla_external_fqdn: "public.at.lab"
network_interface: "ens3"
neutron_external_interface: "ens4"
enable_openstack_core: "yes"
enable_haproxy: "yes"
neutron_plugin_agent: "ovn"
enable_neutron_provider_networks: "yes"

## SERVICE
enable_cinder: "yes"
enable_cinder_backup: "no"

## CEPH
glance_backend_ceph: "yes"
cinder_backend_ceph: "yes"
nova_backend_ceph: "yes"

## TLS
kolla_enable_tls_internal: "yes"
kolla_enable_tls_external: "yes"
kolla_copy_ca_into_containers: "yes"
kolla_enable_tls_backend: "yes"
openstack_cacert: "/etc/ssl/certs/ca-certificates.crt"
kolla_admin_openrc_cacert: "/etc/ssl/certs/ca-certificates.crt"

workaround_ansible_issue_8743: yes

7. Reconfigure openstack service

# exec on at-openstack-aio
source os-venv/bin/activate
kolla-ansible -i ./all-in-one reconfigure --tags glance,cinder,nova

8. Remove lvm volume service

# exec on at-openstack-aio
docker exec -it  cinder_api bash
cinder-manage service list
cinder-manage service remove cinder-volume at-openstack-aio@lvm-1

Operational Test

1. Volume service verification

# exec on at-openstack-aio
openstack volume service list
docker exec -it  cinder_api bash
cinder-manage service list

2. Glance verification

# exec on at-openstack-aio
wget https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img
openstack image create --disk-format qcow2   --container-format bare --public   --file ./jammy-server-cloudimg-amd64.img ubuntu-jammy
openstack image list

# exec on at-ceph-aio
ceph osd lspools
rbd ls -p images
rbd info images/<image id>

3. Cinder verification

# exec on at-openstack-aio
openstack volume create volume-test --size 1
openstack volume list

# exec on at-ceph-aio
rbd ls -p volumes
rbd info volumes/volume-<volume id>

3. Nova verification

# exec on at-openstack-aio
openstack server create --image cirros-0.5.2-image --flavor tiny --key-name at-openstack-aio-key --network internal-net --security-group allow-all vm-cirros-2
openstack server list

# exec on at-ceph-aio
rbd ls -p vms
rbd info vms/<instance id>_disk

4. Operational test

# exec on at-openstack-aio
openstack floating ip create --floating-ip-address 10.10.12.35 public-net 
openstack server add floating ip vm-cirros-2 10.10.12.35
openstack server add volume vm-cirros-2 volume-test --device /dev/vdb
ssh -o 'PubkeyAcceptedKeyTypes +ssh-rsa' cirros@10.10.12.35

echo "GG BANG" > gg.txt
mkfs.ext4 /dev/vdb
mkdir /mnt/data
mount /dev/vdb /mnt/data
lsblk
echo "MANTAP" > /mnt/data/mantap.txt

References

• https://docs.ceph.com/en/reef/rbd/rbd-openstack/

• https://docs.openstack.org/kolla-ansible/2024.1/reference/storage/external-ceph-guide.html