seapath-exporter

A comprehensive Prometheus exporter for monitoring various infrastructure components that lack proper metrics exposure in existing exporters. It is part of the SEAPATH project.

🎯 Purpose

This exporter fills the gaps left by standard exporters (libvirt-exporter, ceph-exporter, etc.) by providing additional metrics that are needed for production monitoring but aren't available elsewhere.

Rather than creating multiple small exporters for each missing metric, this project consolidates all the "missing pieces" needed for complete infrastructure monitoring.

📊 Current Metrics

Vhost Thread Monitoring (libvirt/KVM)

Monitors CPU usage of vhost threads for QEMU/KVM virtual machines.

Metric:

virsh_vhost_cpu_time_seconds{domain="vm-name", thread="vhost-12345"} 123.45

• Labels:
  • domain: VM domain name
  • thread: vhost thread identifier
• Type: Gauge
• Unit: Seconds (cumulative CPU time)

Scrape Health

virsh_exporter_libvirt_up 1

Set to 0 when the exporter could not reach libvirt during the last scrape. Without it, a broken libvirt connection would look exactly like a host running no VM, since both produce no virsh_vhost_cpu_time_seconds series.

🚀 Planned Metrics

• Additional libvirt metrics not covered by prometheus-libvirt-exporter
• Ceph metrics missing from ceph-exporter
• Pacemaker cluster metrics
• Debian RT (Real-Time) kernel metrics
• Other infrastructure metrics as needed

This list is actively developed based on real production monitoring needs.

📦 Installation

Container Image

The image is published on the GitHub Container Registry:

podman pull ghcr.io/seapath/seapath-exporter:latest

Quick Start with Podman

podman run -d \
  --name seapath-exporter \
  --restart unless-stopped \
  -p 9184:9184 \
  -v /var/run/libvirt/libvirt-sock:/var/run/libvirt/libvirt-sock:ro \
  -v /var/run/libvirt/qemu:/var/run/libvirt/qemu:ro \
  --pid=host \
  ghcr.io/seapath/seapath-exporter:latest

Quick Start with Docker

docker run -d \
  --name seapath-exporter \
  --restart unless-stopped \
  -p 9184:9184 \
  -v /var/run/libvirt/libvirt-sock:/var/run/libvirt/libvirt-sock:ro \
  -v /var/run/libvirt/qemu:/var/run/libvirt/qemu:ro \
  --pid=host \
  ghcr.io/seapath/seapath-exporter:latest

Systemd with Podman Quadlet (Recommended)

Create /etc/containers/systemd/seapath-exporter.container:

[Unit]
Description=Prometheus SEAPATH Exporter
After=network-online.target libvirtd.service
Wants=network-online.target

[Container]
Image=ghcr.io/seapath/seapath-exporter:latest
PublishPort=9184:9184
Volume=/var/run/libvirt/libvirt-sock:/var/run/libvirt/libvirt-sock:ro
Volume=/var/run/libvirt/qemu:/var/run/libvirt/qemu:ro
PodmanArgs=--pid=host
SecurityLabelDisable=true

[Service]
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable --now seapath-exporter.service

docker-compose

version: '3.8'

services:
  seapath-exporter:
    image: ghcr.io/seapath/seapath-exporter:latest
    container_name: seapath-exporter
    restart: unless-stopped
    ports:
      - "9184:9184"
    volumes:
      - /var/run/libvirt/libvirt-sock:/var/run/libvirt/libvirt-sock:ro
      - /var/run/libvirt/qemu:/var/run/libvirt/qemu:ro
    pid: host

🔧 Configuration

Prometheus Configuration

Add to your prometheus.yml:

scrape_configs:
  - job_name: 'seapath-exporter'
    static_configs:
      - targets: ['localhost:9184']
    scrape_interval: 30s

Environment Variables

Everything is optional, the defaults reproduce the historical behaviour.

Variable	Default	Description
LISTEN_ADDRESS	0.0.0.0	Address the metrics endpoint binds to
LISTEN_PORT	9184	Port the metrics endpoint binds to
LIBVIRT_URI	qemu:///system	libvirt connection URI
QEMU_PID_DIR	/var/run/libvirt/qemu	Where the per-domain QEMU pid files are read
LOG_LEVEL	INFO	DEBUG also logs the domains and threads that could not be inspected
TLS_CERT_FILE	unset	Server certificate, enables TLS when set
TLS_KEY_FILE	unset	Server private key, mandatory together with TLS_CERT_FILE
TLS_CLIENT_CA_FILE	unset	CA used to verify client certificates, enables mutual TLS
TLS_MIN_VERSION	1.3	Minimum accepted TLS version, 1.2 or 1.3

TLS and Authentication

Setting TLS_CERT_FILE and TLS_KEY_FILE switches the endpoint to HTTPS. Adding TLS_CLIENT_CA_FILE additionally requires every scraper to present a client certificate signed by that CA, which authenticates Prometheus without any shared secret to distribute:

podman run -d \
  --name seapath-exporter \
  -p 9184:9184 \
  -v /var/run/libvirt/libvirt-sock:/var/run/libvirt/libvirt-sock:ro \
  -v /var/run/libvirt/qemu:/var/run/libvirt/qemu:ro \
  -v /etc/prometheus/exporters/tls:/etc/prometheus/exporters/tls:ro \
  -e TLS_CERT_FILE=/etc/prometheus/exporters/tls/server.crt \
  -e TLS_KEY_FILE=/etc/prometheus/exporters/tls/server.key \
  -e TLS_CLIENT_CA_FILE=/etc/prometheus/exporters/tls/ca.crt \
  --pid=host \
  ghcr.io/seapath/seapath-exporter:latest

The matching scrape configuration:

scrape_configs:
  - job_name: 'seapath-exporter'
    scheme: https
    tls_config:
      ca_file: /etc/prometheus/tls/ca.crt
      cert_file: /etc/prometheus/tls/prometheus.crt
      key_file: /etc/prometheus/tls/prometheus.key
    static_configs:
      - targets: ['hypervisor1:9184']

This mirrors what the Prometheus exporter-toolkit offers on the Go exporters through --web.config.file, so a fleet can be secured the same way end to end. Note that basic authentication is deliberately not implemented: prometheus_client provides no server side support for it, whereas client certificates are native.

✅ Verification

Check that the exporter is running:

curl http://localhost:9184/metrics

Expected output (when VMs are running):

# HELP virsh_vhost_cpu_time_seconds CPU time (user+system) of vhost threads on the host related to this domain, in seconds
# TYPE virsh_vhost_cpu_time_seconds gauge
virsh_vhost_cpu_time_seconds{domain="vm1",thread="vhost-12345"} 123.45
virsh_vhost_cpu_time_seconds{domain="vm2",thread="vhost-67890"} 67.89
# HELP virsh_exporter_libvirt_up Whether the last scrape managed to query libvirt
# TYPE virsh_exporter_libvirt_up gauge
virsh_exporter_libvirt_up 1.0

With mutual TLS enabled, the same check needs the client material:

curl --cacert ca.crt --cert prometheus.crt --key prometheus.key \
  https://localhost:9184/metrics

🏗️ Building from Source

Prerequisites

• Python 3.11+
• libvirt development libraries
• Podman or Docker

Build the Container

# Clone the repository
git clone https://github.com/seapath/seapath-exporter.git
cd seapath-exporter

# Build with Podman
podman build -t seapath-exporter:latest .

# Or with Docker
docker build -t seapath-exporter:latest .

Run Locally (without container)

# Install dependencies
pip install -r requirements.txt

# Run the exporter
python seapath_exporter.py

./buildtest.sh builds the image and checks that the metrics endpoint answers.

Releasing

Pushing a version tag publishes the image through the Container image workflow:

git tag -s v0.1.0 -m v0.1.0
git push origin v0.1.0

The tag v0.1.0 yields the image tags 0.1.0, 0.1 and latest on ghcr.io/seapath/seapath-exporter. Pull requests build the image without publishing it.

📋 Requirements

Current Requirements (vhost metrics)

• Host with libvirt/QEMU installed
• Access to the libvirt socket (/var/run/libvirt/libvirt-sock, read-write because the exporter connects to qemu:///system)
• Access to QEMU PID files (/var/run/libvirt/qemu)
• Host PID namespace access (--pid=host)
• The container runs as root: reading /proc/<pid>/task/*/comm of the QEMU processes and opening the libvirt socket both require it

System Dependencies

• libvirt-dev / libvirt-devel
• python3-dev / python3-devel
• gcc
• pkg-config

🐛 Troubleshooting

No metrics appearing

1. Check that VMs are running:

   virsh list

2. Check exporter logs:

   # Podman
   podman logs seapath-exporter
   
   # Systemd
   sudo journalctl -u seapath-exporter.service -f

3. Verify the container can access libvirt:

   podman exec seapath-exporter python3 -c "import libvirt; print(libvirt.open('qemu:///system').listDomainsID())"

Permission denied errors

Ensure the container has:

• Access to libvirt socket (check file permissions)
• Host PID namespace (--pid=host)
• SELinux labels if applicable (SecurityLabelDisable=true in Quadlet)

📈 Use Cases

Current:

• Monitor CPU usage of vhost threads for KVM virtual machines
• Track performance of virtio-net network interfaces
• Identify VMs with high vhost CPU consumption
• Create alerts when vhost thread CPU usage exceeds thresholds

Upcoming:

• Monitor specific libvirt domain states and performance metrics
• Track Ceph cluster health metrics not exposed by standard exporters
• Monitor Pacemaker cluster resource states
• Track Debian RT kernel performance metrics

🤝 Contributing

Contributions are welcome! If you need a specific metric that's missing from standard exporters:

1. Open an issue describing the metric and use case
2. Submit a pull request with the implementation
3. Update documentation and tests

Development Guidelines

• Follow existing code structure
• Add metrics that complement (not duplicate) existing exporters
• Include clear documentation and examples
• Test in a real environment before submitting
• Sign off every commit (git commit -s), as required by the SEAPATH Developer Certificate of Origin policy

📜 License

Apache License 2.0, see LICENSE.

🔗 Links

• Container image
• Report Issues

📞 Support

For issues, questions, or feature requests:

• Open an issue on GitHub
• Check existing documentation
• Review closed issues for solutions

🙏 Acknowledgments

This exporter complements existing excellent exporters:

• prometheus-libvirt-exporter
• ceph-exporter
• Other Prometheus exporters in the ecosystem

---

Philosophy: One exporter for all the missing pieces, not one exporter per missing piece.