Enforce overlap eligibility and route-consumer safety atomically

[chet]

Allowing two VPCs to use the same CIDR is safe only while their routing domains remain separate. Checking the two endpoints of a new peering is not enough: if A imports routes from B and C, then B and C must also be disjoint from A's point of view. Prefix creation after peering must enforce the same rule as peering after prefix creation, and concurrent mutations must not each validate stale state.

Current state

• VPC peering in crates/api-core/src/handlers/vpc_peering.rs checks virtualization compatibility and site policy but not IP overlap.
• crates/api-db/src/vpc_peering.rs::get_prefixes_by_vpcs already collects VpcPrefix and NetworkPrefix CIDRs for VPCs.
• FnnRoutingProfileConfig already exposes imports, exports, underlay/default/host leaks, accepted tenant leak communities, allowed anycast prefixes, and related settings.
• Site-wide common/additional route targets and site_global_vpc_vni can reconnect VPCs even when a direct peering edge looks safe.
• The FNN template installs its site-root isolation ACL only when VpcIsolationBehaviorType::MutualIsolation produces UseVpcIsolation = true.
• Core currently sends one shared SitePrefix isolation list to every FNN policy. Until rendering becomes per-profile/per-VPC, adding a tenant root can affect an otherwise unrelated active FNN path at the same site.
• Tenant and global NSG permit policies are ordered before the isolation policy in the current FNN template, so their ability to bypass the final deny must be treated as unsafe until proved otherwise.

General idea

• Add an explicit default-false FNN profile control for overlap eligibility and retain the separate default-false site gate.
• Require FNN, tenant SitePrefix lineage, DatacenterOnly routing, MutualIsolation, and a unique native VNI/route-target domain.
• Compute the resolved effective policy for every active tenant-serving FNN path at an enabled site. Shared/global route targets, profile imports/exports outside the reviewed peer path, underlay/default/host leaks, tenant leak communities, public/anycast behavior, site_global_vpc_vni, traffic-intercept/VMaaS paths, and unsafe effective per-interface overrides make the site ineligible while that path is active.
• Treat permit-bearing tenant NSGs and global policy overrides as ineligible when they could run before and bypass tenant-root isolation. Allow them only after the implementation establishes safe precedence or packet-level qualification proves the effective policy cannot bypass isolation; deny-only or provably non-intersecting rules need not become a permanent blanket ban.
• Build one effective address-space view from VpcPrefixes and direct NetworkPrefixes. Include rows being withdrawn until their DPU removal barrier completes.
• Evaluate each receiving routing domain, not just each edge. The receiver's local prefixes and every imported peer/sibling set must be pairwise disjoint.
• Run the same evaluator for SitePrefix/VpcPrefix/NetworkPrefix and VpcPeering mutations, including profile-relevant changes.
• Serialize those mutations with one site-scoped advisory lock, re-read the effective graph inside the transaction, validate, and write before releasing the lock.
• At Core startup, resolve current static site/profile configuration and revalidate every VPC already using isolated scope before serving managed-host network configuration. Refuse unsafe overlap-enabled startup rather than rendering a newly shared path.
• Treat gate disablement as a freeze rather than a rollback: keep existing roots and routes protected, allow reads and safe contraction/drain, and reject new or overlap-increasing mutations. Reject removal of a live profile opt-in when duplicate space still depends on it.
• Return stable, actionable Core errors that identify the receiver and conflicting resources without exposing another tenant's CIDR through the tenant-facing path.

Acceptance criteria

• Equal address space in separate eligible, unconnected FNN VPCs is accepted by application validation.
• Same-VPC overlap is rejected.
• Direct peering between overlapping VPCs is rejected.
• The A-imports-B-and-C sibling conflict is rejected when B and C overlap.
• Prefix-before-peer and peer-before-prefix requests produce the same answer.
• Concurrent prefix and peering requests cannot both pass by observing stale state.
• An unsafe active tenant-serving FNN path makes the site ineligible while the isolation list is site-wide; an unused profile alone does not.
• Open VPC isolation, shared routing/leak paths, unqualified permit-bearing NSG policies, traffic intercept/VMaaS, and unsafe per-interface overrides are rejected.
• A static routing-profile/site-config change that makes existing overlap unsafe stops readiness/config serving with an actionable error.
• Disabling a gate or removing an opt-in with live duplicate space cannot withdraw protection or reinterpret the rows as globally unique.
• Draining resources remain visible to conflict admission until the DPU withdrawal barrier completes.
• Error mapping is covered without leaking another tenant's private resource details.

Not in this issue

• Database constraint replacement.
• A second routing-profile framework.
• Public/BYOIP routing.
• Mixed safe and unsafe active profiles at one enabled site. Supporting that requires a reviewed per-profile/per-VPC isolation-rendering design.
• Making peering creation asynchronous. A delayed route addition loses connectivity but does not expose another tenant; unsafe relationships are rejected before insertion.

Likely areas

• crates/api-core/src/handlers/vpc_peering.rs
• crates/api-db/src/vpc_peering.rs
• crates/api-core/src/handlers/vpc_prefix.rs
• NetworkSegment/NetworkPrefix mutation paths
• crates/api-core/src/cfg/file.rs and FNN profile resolution
• a focused routing-safety module in crates/api-core
• crates/api-core/src/tests/vpc_peering.rs, VpcPrefix, NetworkSegment, concurrency, and profile tests

Delivery plan

#4940 is closed and retained only as a reference implementation and test corpus. Deliver this tracker through smaller reviewable slices:

Wave 1 — parallel foundations

• Extract VPC peering import direction as a tested renderer contract #5109 extracts and tests the renderer's peering import-direction contract.
• Define overlap eligibility configuration and profile policy #5110 defines the overlap eligibility configuration and profile policy.
• Add a transaction-scoped routing admission lock #5111 adds the transaction-scoped routing admission lock.
• Fence Admin instance force-delete against configuration updates #5112 separately fences Admin instance force-delete against configuration updates before tenant prefix reuse is enabled.

Wave 2 — parallel writer adoption

• Enforce overlap admission for prefix-bearing writes #5113 applies admission to prefix-bearing writes.
• Enforce overlap admission for VPC and peering policy changes #5114 applies admission to VPC and peering policy changes.
• Include retained instance routing paths in overlap admission #5115 applies admission to retained instance routing paths.

Wave 3 — composition and closure

• Audit retained routing safety at startup and close admission coverage #5116 audits retained state at startup and proves every routing-expanding writer participates.

Keep candidate and evaluator code close to the call sites that own it. A later organization-only lift-and-shift is reasonable only after landed code shows stable duplication. Do not recreate a broad routing_safety.rs module or a generic lock_site_mutation sweep.

Planning

• Size: L
• Risk: Critical
• Best fit: Principal networking/control-plane engineer

Dependencies

• Native blocked by: Associate VpcPrefixes with exact SitePrefix parents and parent-scoped allocation #3886 — Associate VpcPrefixes with exact SitePrefix parents and parent-scoped allocation
• Coding can start: Associate VpcPrefixes with exact SitePrefix parents and parent-scoped allocation #3886 for final parent fields; pure evaluator work can begin earlier
• Enablement: Install scoped prefix constraints while global exclusions remain #3891, Retire global prefix exclusions for eligible FNN VPCs #3892, Drain VpcPrefix and NetworkSegment withdrawals through existing DPU acknowledgements #3895, Drain VPC peering withdrawals through existing DPU acknowledgements #3896, and Qualify and gate tenant-managed SitePrefixes across Core, DPU, and REST #3902
• Parallel work: Compare all agent-rendered network inputs before skipping apply #3893 and the REST lane

Part of #3883.