Skip to content

configure_hypervisor: add rbd helpers for quadlet containers#986

Open
insatomcat wants to merge 1 commit into
mainfrom
rbdhelpers
Open

configure_hypervisor: add rbd helpers for quadlet containers#986
insatomcat wants to merge 1 commit into
mainfrom
rbdhelpers

Conversation

@insatomcat

@insatomcat insatomcat commented Jul 23, 2026

Copy link
Copy Markdown
Member

Context

SEAPATH containers (Podman quadlets) that use Ceph RBD for persistent storage must today embed a long sequence of ExecStartPre= and ExecStopPost= steps directly in the quadlet file. A typical unit needs to:

  1. Detect and clean up dirty state from a previous interrupted run (unmount stale mountpoints, unmap all stale RBD device mappings).
  2. Create the RBD image on first use (rbd create).
  3. Map the block device (rbd map).
  4. Create the mountpoint directory if absent.
  5. Detect and format the device on first use (mkfs.ext4).
  6. Mount the device.
  7. On stop: unmount all stacked mounts and unmap all device mappings.

This boilerplate is identical across every container that uses RBD, only the image name changes. Writing it by hand is error-prone (especially the dirty-state cleanup) and produces quadlet files that are hard to read and maintain.

What this commit adds

Two bash scripts installed to /usr/local/bin/ by the configure_hypervisor role. Both take only the image name; pool (rbd) and mountpoint (/mnt/rbd/) are fixed by convention.

seapath-rbd-mount []

Idempotent mount script.  Handles the full lifecycle:

- Dirty-state cleanup: loops umount until the mountpoint is fully clear (interrupted previous runs can stack multiple mounts at the same path), then enumerates ALL device nodes mapped to the image via rbd device list and unmaps them.  The stable symlink /dev/rbd/<pool>/<image> points only to the most recent mapping; earlier stale mappings must be found by iterating the device list.

- First-use image creation: calls rbd create with --image-feature layering and the caller-supplied size (default 1G) only if rbd info reports the image does not exist.

- Device mapping: rbd map creates both /dev/rbdN and the stable symlink /dev/rbd/rbd/<image>.

- First-use filesystem: blkid is used to detect whether a filesystem already exists.  If not (freshly created image), mkfs.ext4 -q formats it silently.  Subsequent runs skip this step.

- Mount.

seapath-rbd-unmount

Best-effort unmount/unmap script for ExecStopPost=.  Loops umount until
the mountpoint is fully clear (handles stacked mounts from repeated
interrupted starts), then force-unmaps (-o force) all device nodes still
mapped to the image.  Force-unmapping handles devices held by udevd,
blkid, or other kernel references after a container crash.  Exits 0
regardless.

Usage in a quadlet

The ExecStartPre / ExecStopPost block in the quadlet file is reduced to:

ExecStartPre=/usr/local/bin/seapath-rbd-mount mycontainer
ExecStopPost=/usr/local/bin/seapath-rbd-unmount mycontainer

Any app-specific subdirectories that the container needs inside the mountpoint are the caller's responsibility (mkdir -p in a separate ExecStartPre= line) — they are inherently application-specific and out of scope for a generic helper.

Deployment

The two scripts are installed by a new rbd_helpers.yml task file included from configure_hypervisor/tasks/main.yml, following the same pattern as the other helpers in that role (ansible.builtin.copy, mode 0755).

The task is skipped on Yocto (seapath_distro != "Yocto"): the root filesystem is read-only there, so the scripts must be shipped by the image itself. Porting them to meta-seapath is required before quadlets relying on these helpers can be used on Yocto.

Context
-------
SEAPATH containers (Podman quadlets) that use Ceph RBD for persistent storage
must today embed a long sequence of ExecStartPre= and ExecStopPost= steps
directly in the quadlet file.  A typical unit needs to:

  1. Detect and clean up dirty state from a previous interrupted run
     (unmount stale mountpoints, unmap all stale RBD device mappings).
  2. Create the RBD image on first use (rbd create).
  3. Map the block device (rbd map).
  4. Create the mountpoint directory if absent.
  5. Detect and format the device on first use (mkfs.ext4).
  6. Mount the device.
  7. On stop: unmount all stacked mounts and unmap all device mappings.

This boilerplate is identical across every container that uses RBD — only
the image name changes.  Writing it by hand is error-prone (especially the
dirty-state cleanup) and produces quadlet files that are hard to read and
maintain.

What this commit adds
---------------------
Two bash scripts installed to /usr/local/bin/ by the configure_hypervisor role.
Both take only the image name; pool (rbd) and mountpoint (/mnt/rbd/<name>)
are fixed by convention.

  seapath-rbd-mount <image-name> [<size>]

    Idempotent mount script.  Handles the full lifecycle:

    - Dirty-state cleanup: loops umount until the mountpoint is fully clear
      (interrupted previous runs can stack multiple mounts at the same path),
      then enumerates ALL device nodes mapped to the image via rbd device list
      and unmaps them.  The stable symlink /dev/rbd/<pool>/<image> points only
      to the most recent mapping; earlier stale mappings must be found by
      iterating the device list.

    - First-use image creation: calls rbd create with --image-feature
      layering and the caller-supplied size (default 1G) only if rbd info
      reports the image does not exist.

    - Device mapping: rbd map creates both /dev/rbdN and the stable symlink
      /dev/rbd/rbd/<image>.

    - First-use filesystem: blkid is used to detect whether a filesystem
      already exists.  If not (freshly created image), mkfs.ext4 -q formats
      it silently.  Subsequent runs skip this step.

    - Mount.

  seapath-rbd-unmount <image-name>

    Best-effort unmount/unmap script for ExecStopPost=.  Loops umount until
    the mountpoint is fully clear (handles stacked mounts from repeated
    interrupted starts), then force-unmaps (-o force) all device nodes still
    mapped to the image.  Force-unmapping handles devices held by udevd,
    blkid, or other kernel references after a container crash.  Exits 0
    regardless.

Usage in a quadlet
------------------
The ExecStartPre / ExecStopPost block in the quadlet file is reduced to:

  ExecStartPre=/usr/local/bin/seapath-rbd-mount mycontainer
  ExecStopPost=/usr/local/bin/seapath-rbd-unmount mycontainer

Any app-specific subdirectories that the container needs inside the
mountpoint are the caller's responsibility (mkdir -p in a separate
ExecStartPre= line) — they are inherently application-specific and out of
scope for a generic helper.

Deployment
----------
The two scripts are installed by a new rbd_helpers.yml task file included
from configure_hypervisor/tasks/main.yml, following the same pattern as
the other helpers in that role (ansible.builtin.copy, mode 0755).

The task is skipped on Yocto (seapath_distro != "Yocto"): the root
filesystem is read-only there, so the scripts must be shipped by the image
itself.  Porting them to meta-seapath is required before quadlets relying
on these helpers can be used on Yocto.

Signed-off-by: Florent Carli <florent.carli@rte-france.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant