kubegressd

kubegressd is a privileged Kubernetes DaemonSet that attaches an eBPF BPF_CGROUP_INET_EGRESS program to the cgroup v2 kubepods parent:

/sys/fs/cgroup/kubepods.slice

The BPF program handles IPv4 and IPv6 egress. It reads the current cgroup id with bpf_get_current_cgroup_id(), parses the destination IP, skips traffic matching the configured internal IPv4/IPv6 CIDR LPM tries, and increments coarse per-cgroup external egress byte and packet counters. Per-destination cardinality is intentionally avoided on the hot path.

Build

On Linux with LLVM clang BPF target support:

make all

On macOS, Apple clang usually cannot compile -target bpf; use the Dockerfile or set BPF_CLANG to a real LLVM clang that supports the BPF backend.

Run Configuration

Environment variables:

Name	Default	Purpose
NODE_NAME	required	Node name used to watch local pods.
BPF_OBJECT_PATH	/opt/kubegressd/egress.bpf.o	Compiled BPF object.
BPF_PIN_PATH	/sys/fs/bpf/kubegressd	bpffs path for pinned maps/program.
CGROUP_PATH	/sys/fs/cgroup/kubepods.slice	cgroup v2 parent hook target.
INTERNAL_CIDRS	empty	Comma-separated IPv4/IPv6 CIDRs to skip as internal. Include loopback, link-local, cluster, service, pod, and node CIDRs for external-only accounting.
EXTERNAL_COUNTER_MAX_ENTRIES	8192	Max active cgroup counter keys per node. Size from node max pods, containers per pod, and cleanup headroom.
LIFECYCLE_RINGBUF_SIZE	262144	Cgroup lifecycle ring buffer size in bytes. Must be a power of two. Increase if kubegressd_lifecycle_ringbuf_full_events_total rises during pod churn.
SAMPLE_INTERVAL	15s	Cgroup attribution sampling interval. This can be shorter than the emit interval to catch short-lived pods.
EMIT_INTERVAL	1m	Counter delta emit interval.
METRICS_HIGH_WATER_WINDOWS	5	Number of emit intervals to retain before resetting lifecycle ring buffer high-water stats.
UNATTRIBUTED_STALE_GRACE_READS	10	Emit intervals to hold stale unattributed cgroups before emitting them under the unattributed sentinel identity and deleting the counter.
SPOOL_DB_PATH	/var/lib/kubegressd/kubegressd.db	Host-backed SQLite spool path.
SPOOL_PERIOD	1h	Period table duration; table suffix is the period start timestamp in milliseconds.
SPOOL_RETENTION	6h	How long to keep local period tables.
HTTP_LISTEN_ADDR	:8080	Local collector API listen address.

Userspace Flow

1. Load and pin the BPF program and maps.
2. Populate internal_cidrs_v4 and internal_cidrs_v6.
3. Attach the program to CGROUP_PATH as cgroup_skb/egress.
4. Watch pods scheduled to NODE_NAME.
5. Walk the cgroup tree and map cgroup inode id to pod UID.
6. Prefer cgroup lifecycle tracepoints to bind cgroup id to path/UID at creation time; keep inotify/walk as fallback.
7. Join pod UID to namespace, pod name, workload, tenant, and labels.
8. Mark host-network pods with attribution=host-network-pod; mark root cgroup node traffic with attribution=node-root-cgroup.
9. Store pod egress deltas in a host-backed SQLite spool.
10. After a final stale read, delete counters for cgroups that disappeared.

If the external counter map reaches EXTERNAL_COUNTER_MAX_ENTRIES, traffic is still passed but the packet is counted in skip_counters with skip_reason=external_map_full so undercounting is visible. If the cgroup lifecycle ring buffer fills during pod churn, the lifecycle event is dropped and counted as skip_reason=lifecycle_ringbuf_full. Internal CIDR matches are not emitted as production metrics.

Local Spool API

kubegressd stores only external pod egress rows locally:

id, node_name, window_start_ms, window_end_ms, namespace, pod_name, pod_uid, tx_bytes, tx_packets

Period tables are named with their period start timestamp:

egress_1781672400000

Collectors can discover and page records:

GET /v1/periods
GET /v1/egress?period_start_ms=1781672400000&after_id=123&limit=1000
GET /metrics

Retention drops old period tables instead of deleting large ranges from an active table.

Operational pressure and event totals are exposed at /metrics in Prometheus text exposition format:

kubegressd_external_map_keys
kubegressd_external_map_max_entries
kubegressd_lifecycle_ringbuf_high_watermark_bytes
kubegressd_lifecycle_ringbuf_size_bytes
kubegressd_lifecycle_ringbuf_high_watermark_windows
kubegressd_lifecycle_ringbuf_samples_total
kubegressd_external_map_full_bytes_total
kubegressd_external_map_full_packets_total
kubegressd_lifecycle_ringbuf_full_events_total
kubegressd_unattributed_cgroup_expired_total
kubegressd_unattributed_cgroup_expired_bytes_total
kubegressd_unattributed_cgroup_expired_packets_total

The lifecycle ring buffer high-water mark is read on the configured collection schedule and reset after METRICS_HIGH_WATER_WINDOWS emit intervals, so the metric reports the max seen during the current high-water window.

If an unattributed cgroup expires, its bytes are also written to the normal egress stream under the sentinel identity:

namespace=__kubegressd_unattributed
pod_name=unattributed-cgroup
pod_uid=unattributed:<cgroup_id>

License

The Go userspace daemon and repository code are licensed under Apache-2.0: see LICENSE and NOTICE.

The eBPF sources under bpf/ are dual-licensed under BSD-2-Clause OR GPL-2.0-only and retain the kernel-facing Dual BSD/GPL BPF license string for Linux helper compatibility.