Tenant-Managed Site Prefixes

[chet]

Before this Initiative, operator-configured site_fabric_prefixes define the address space Core accepts for VPC overlays, and Core's database constraints make every VpcPrefix and NetworkPrefix unique across the site. Tenants need to register their own private IPv4 roots, carve exact VpcPrefixes from them, and reuse the same CIDR in separate isolated FNN VPCs.

The first part is fairly direct: make SitePrefix a Core-owned resource, link each VpcPrefix to its exact authority, and expose tenant lifecycle and exact carving through REST and nicocli. The important part is making duplicate addresses safe everywhere that treats an IP as site-unique. That includes routing imports, database constraints, reverse lookup/deletion, reverse DNS, PXE/cloud-init identity, and DPU route withdrawal.

General idea

• Persist configuration-derived and tenant-managed SitePrefixes in Core with stable IDs, ownership, source, and lifecycle state.
• Keep OperatorManaged roots configuration-controlled while allowing a tenant to manage only its own TenantManaged roots. The authority is derived by Core rather than selected by a caller.
• Link every VpcPrefix in the new path to one exact SitePrefix and scope child allocation by that ID.
• Permit overlap only at explicitly enabled sites where every active tenant-serving FNN path passes the site-wide safe-policy predicate, each participating profile opts in, and the physical fabric has no shared external route consumer for the overlap-enabled route targets.
• Validate the complete address space imported by each receiving VPC under one site-scoped mutation lock, including sibling peers and resources being withdrawn.
• Replace raw-IP uniqueness assumptions in Core address operations, PTR handling, and the boot/cloud-init lookup path before the global constraints are removed.
• Reuse the existing Core-to-agent desired/applied network versions. Make prefix and peering mutations advance those versions and wait for affected DPUs before declaring policy ready or deleting route state.
• Project Core SitePrefixes into source-aware REST resources, add tenant-authorized lifecycle endpoints, and use a VPC-and-SitePrefix IPAM namespace for exact VpcPrefix leases.
• Keep the Core overlap gate and REST exposure gate off until cross-stack qualification passes. If either is later disabled, preserve reads, protection, safe contraction, and cleanup while blocking new or expanding mutations.

Ratified starting contract

• Accept canonical RFC1918 IPv4 /8 through /31; a full-root /31 is one generated linknet per VPC-and-SitePrefix namespace, while direct /31 NetworkSegments remain invalid.
• Start with eight non-deleted tenant roots per tenant/site and a configurable 64-effective-rule SitePrefix rendering guardrail. These are rollout limits, not a hardware claim.
• Require MutualIsolation, unique FNN routing domains, safe policy on every active tenant-serving path at the site, and physical-fabric verification that no shared external VRF imports multiple overlap-enabled route targets.
• Preserve current NXDOMAIN behavior for an ambiguous address-only PTR query while guaranteeing that it never returns one tenant's record.
• Do not admit duplicate addresses until the production PXE/cloud-init identity path is proved safe; ambiguity returns no tenant-specific material or discovery fallback.
• Support only server-derived OperatorManaged and TenantManaged authorities. Configured remains only the deprecated protobuf alias and configuration-source vocabulary. The admin CLI may act on a named tenant's resource but cannot create a database-backed operator-global root.
• Require the site gate, resolved-profile opt-in, computed policy checks, fabric qualification, and boot qualification. Gate-off with live duplicates is a freeze and safe-drain mode, not rollback.

This is the starting implementation path, not a substitute for reading current main. Each subissue names the known files and invariants so an engineer or agent can resume without this planning session.

MVP boundaries

• Included: RFC1918 IPv4 /8 through /31, FNN, DatacenterOnly, exact CIDR allocation, overlap-safe peering, Core/admin CLI, REST/OpenAPI/generated clients, nicocli, DPU convergence, qualification, rollout, and rollback documentation.
• Deferred: tenant IPv6, ETV, Flat, direct tenant SitePrefix-to-NetworkSegment creation, public/BYOIP announcement, tenant-specific PTR views, arbitrary shared routing profiles, and a new web UI.

Completion criteria

• Two tenants can register the same root and allocate the same exact VpcPrefix into separate eligible FNN VPCs.
• The same VPC cannot contain overlapping address space.
• A direct or sibling routing relationship that would make one receiver import overlapping space is rejected.
• Overlap cannot activate unless MutualIsolation, the site-wide on-box policy checks, each participating profile opt-in, physical-fabric attestation, and boot-identity qualification pass.
• Duplicate addresses do not cause cross-VPC deletion, lookup, PTR, PXE, or cloud-init disclosure.
• SitePrefix activation and route withdrawal wait for the affected DPUs to acknowledge the existing desired network version.
• The eight-root tenant quota and 64-rule initial site guardrail fail atomically with actionable, tenant-safe diagnostics, and lowering either limit never removes existing protection.
• Disabling a gate with live duplicates blocks expansion but leaves existing resources visible and safely drainable.
• Version-based lifecycle barriers remain disabled until every Core replica serving DPU configuration runs the same rendering semantics.
• Existing configured roots, provider IPBlocks, Allocation flows, non-FNN networks, and sites with both gates off retain their pre-Initiative behavior.
• Operator and tenant workflows, upgrade order, canary checks, failure signals, and rollback limits are documented.

Delivery

Implementation tasks are linked from their parent tracking issues. Routing-safety work under #3890 is intentionally delivered through smaller parallel slices rather than one large PR; #4940 is closed and retained only as a reference implementation and test corpus. Partner request #2446 is tracked separately and blocked by this Initiative; it does not imply another PR. Each issue's coding, merge, and enablement dependency fields define the path; titles intentionally have no numeric prefix.