NVIDIA · chet · Aug 13, 2026
@@ -42,6 +42,7 @@ Use `site_explorer.dpu_policy` instead.
 | `enable_route_servers` | `bool` | `false` | `networking` | Enables route server injection into DPU FRR configs for L2VPN. |
 | `deny_prefixes` | `Vec<IpNetwork>` | `[]` | `networking` | IPv4 and IPv6 CIDR prefixes that tenant instances are blocked from reaching. FNN generates family-specific NVUE ACL policies; all non-FNN virtualizers apply the IPv4 prefixes only. |
 | `site_fabric_prefixes` | `Vec<IpNetwork>` | `[]` | `networking` | IP prefixes (v4/v6) assigned for tenant use within this site. |
+| `tenant_prefix_overlap_enabled` | `bool` | `false` | `networking` | Enables application admission for tenant-owned VPC prefixes to reuse CIDRs in separate, mutually isolated FNN routing domains after the legacy database exclusions are removed. Enabling the site gate alone is insufficient: every overlapping VPC must use an `overlap_eligible` safe base profile and satisfy the routing-safety preflight described below. Disabling the gate freezes expansion but still permits deletion and drain. |
 | `max_site_prefixes_per_tenant` | `u32` | `8` | `networking` | Maximum tenant-managed SitePrefixes retained for one tenant at this site. Prefixes awaiting removal still count against this limit and keep their CIDR reserved. |
 | `anycast_site_prefixes` | `Vec<Ipv4Network>` | `[]` | `networking` | Aggregate IPv4 prefixes containing tenant-announced prefixes (e.g., BYOIP). **Deprecated.** Use [`routing_profiles.allowed_anycast_prefixes`](#fnnroutingprofileconfig) instead. |
 | `common_tenant_host_asn` | `Option<u32>` | — | `networking` | ASN that tenants use to peer with the DPU. If unset, any ASN is accepted. |
@@ -635,6 +636,7 @@ client-certificate authentication is not used.
 
 | Field | Type | Default | Description |
 | ------- | ------ | --------- | ------------- |
+| `overlap_eligible` | `bool` | `false` | Operator opt-in allowing VPCs based on this profile to participate in tenant prefix overlap. This base-profile property cannot be overridden on a VPC. |
 | `route_target_imports` | `Option<Vec<RouteTargetConfig>>` | — (effective `[]`) | Route targets imported into DPU VRFs for VPC routes. |
 | `route_targets_on_exports` | `Option<Vec<RouteTargetConfig>>` | — (effective `[]`) | Route targets added to routes exported by the DPU. |
 | `internal` | `Option<bool>` | — (effective `false`) | Whether the profile uses internal VNI allocation. This property cannot be overridden on a VPC. |
@@ -649,6 +651,40 @@ Unset properties retain presence information so a VPC's inline
 `routing_profile_overrides` can inherit them. After the named profile and VPC
 override are combined, properties still unset use the effective defaults above.
 
+Tenant prefix overlap is admitted only when both the top-level
+`tenant_prefix_overlap_enabled` gate and every overlapping VPC's base-profile
+`overlap_eligible` flag are true. An eligible effective profile must also set
+`internal = true` and leave route-target imports/exports, underlay/default-route
+leaks, tenant leak communities, and allowed anycast prefixes disabled or empty.
+The site must use mutual VPC isolation, distinct actual VNIs, deny-only
+non-stateful effective NSGs, and no permit-bearing global
+`network_security_group.policy_overrides`. Active interface routing overrides,
+a site-global VPC VNI, VMaaS, a site-wide anycast prefix, a common internal
+route target, and additional FNN imports are also unsafe. VPC peering is
+rejected if it would make overlapping routes visible in either direction.
+
+These settings and checks are staged safety plumbing. The legacy database
+exclusions continue to block duplicate VPC-prefix persistence until
+[#3891](https://github.com/NVIDIA/infra-controller/issues/3891) and
+[#3892](https://github.com/NVIDIA/infra-controller/issues/3892) replace them;
+the settings alone do not make tenant CIDR reuse available to operators.
+
+NICo renders one SitePrefix isolation list for the whole site. While any
+retained duplicate address space exists, every graph-active tenant-serving FNN
+path must satisfy the isolation rules above, including VPCs whose own prefixes
+do not overlap. Only VPCs participating in an overlap must set their base
+profile's `overlap_eligible` flag. A graph-active FNN VPC without a resolvable
+profile fails closed; a profile or VPC that is not referenced by an address,
+peering, or retained instance is not graph-active and does not block startup.
+
+These settings are read at process startup. Before opening its listeners, NICo
+checks the retained routing graph (including resources draining after soft
+deletion) and refuses to start when live duplicate tenant address space is
+unsafe. This check remains active when `tenant_prefix_overlap_enabled = false`:
+disabling the gate freezes expansion but cannot remove protection from retained
+duplicates. The first unsafe expansion of a latent profile is rejected. Errors
+intentionally do not identify another tenant's CIDR or resource.
+
 ### `VpcDefinition`
 
 | Field | Type | Default | Description |

@@ -223,6 +223,15 @@ pub struct CarbideConfig {
     #[serde(default)]
     pub site_fabric_prefixes: Vec<IpNetwork>,
 
+    /// Enables exact-CIDR reuse across isolated tenant FNN VPCs.
+    ///
+    /// Defaults to false. Disabling the gate blocks expansion but still
+    /// protects retained overlap and permits contraction or drain. Participating
+    /// base profiles must also set `overlap_eligible`; this setting does not
+    /// bypass the legacy database exclusion constraints.
+    #[serde(default)]
+    pub tenant_prefix_overlap_enabled: bool,
+
     /// Maximum number of tenant-managed SitePrefixes retained for one tenant
     /// at this site. Prefixes awaiting removal still count against this limit
     /// and keep their CIDR reserved.
@@ -2432,6 +2441,13 @@ pub struct FnnConfig {
 #[derive(Clone, Debug, Deserialize, Serialize, PartialEq, Default)]
 #[serde(deny_unknown_fields)]
 pub struct FnnRoutingProfileConfig {
+    /// Allows VPCs based on this profile to participate in tenant prefix overlap.
+    ///
+    /// This operator-owned value defaults to false, cannot be overridden by a
+    /// VPC, and still requires the site gate plus every routing-safety check.
+    #[serde(default)]
+    pub overlap_eligible: bool,
+
     /// These are used for import policies to import routes
     /// that match these targets.
     #[serde(default)]
@@ -2513,6 +2529,7 @@ impl FnnConfig {
         };
 
         Ok(Cow::Owned(FnnRoutingProfileConfig {
+            overlap_eligible: base_profile.overlap_eligible,
             route_target_imports: overrides
                 .route_target_imports
                 .clone()
@@ -3618,7 +3635,7 @@ impl MeasuredBootMetricsCollectorConfig {
 }
 
 /// The VPC isolation behavior enforced within a site.
-#[derive(Clone, Copy, Debug, Default, Deserialize, Serialize)]
+#[derive(Clone, Copy, Debug, Default, Deserialize, Eq, PartialEq, Serialize)]
 #[serde(rename_all = "snake_case")]
 pub enum VpcIsolationBehaviorType {
     #[default]
@@ -4328,6 +4345,7 @@ mod tests {
         let profile: FnnRoutingProfileConfig = Figment::new()
             .merge(Toml::string(
                 r#"
+                    overlap_eligible = true
                     route_target_imports = [{ asn = 64512, vni = 10 }]
                     route_targets_on_exports = []
                     internal = true
@@ -4345,6 +4363,7 @@ mod tests {
         assert_eq!(
             profile,
             FnnRoutingProfileConfig {
+                overlap_eligible: true,
                 route_target_imports: Some(vec![RouteTargetConfig {
                     asn: 64512,
                     vni: 10,
@@ -4421,6 +4440,7 @@ mod tests {
             prefix: "192.0.2.0/24".parse().expect("valid test prefix"),
         };
         let base = FnnRoutingProfileConfig {
+            overlap_eligible: true,
             route_target_imports: Some(vec![RouteTargetConfig { asn: 3, vni: 4 }]),
             route_targets_on_exports: Some(vec![inherited_export.clone()]),
             internal: Some(true),
@@ -4453,6 +4473,7 @@ mod tests {
         assert_eq!(
             fnn.resolve_vpc_routing_profile(&vpc).unwrap().as_ref(),
             &FnnRoutingProfileConfig {
+                overlap_eligible: true,
                 route_target_imports: Some(vec![]),
                 route_targets_on_exports: Some(vec![inherited_export]),
                 internal: Some(true),
@@ -5124,6 +5145,7 @@ mod tests {
             }
         );
         assert!(config.dhcp_servers.is_empty());
+        assert!(!config.tenant_prefix_overlap_enabled);
         assert!(!config.allow_insecure_discovery);
         assert!(config.route_servers.is_empty());
         assert!(config.tls.is_none());
@@ -5545,6 +5567,7 @@ mod tests {
             std::time::Duration::from_secs(45 * 60)
         );
         assert_eq!(config.asn, 123);
+        assert!(config.tenant_prefix_overlap_enabled);
         assert_eq!(config.bmc_session_lockout_threshold, 4);
         assert_eq!(
             config.dhcp_servers,

@@ -6,6 +6,7 @@ database_pool_acquire_timeout = "15s"
 database_pool_idle_timeout = "20m"
 database_pool_max_lifetime = "45m"
 asn = 123
+tenant_prefix_overlap_enabled = true
 dhcp_servers = ["1.2.3.4", "5.6.7.8"]
 ntp_servers = ["10.20.30.40", "50.60.70.80"]
 route_servers = ["9.10.11.12"]

@@ -121,6 +121,7 @@ pub(crate) async fn create_initial_networks(
     networks: &HashMap<String, NetworkDefinition>,
 ) -> Result<(), CarbideError> {
     let mut txn = Transaction::begin(db_pool).await?;
+    crate::routing_safety::lock_site_mutation(&mut txn).await?;
     let domains = db::dns::domain::find_by(
         &mut txn,
         ObjectColumnFilter::<db::dns::domain::IdColumn>::All,
@@ -172,6 +173,17 @@ pub(crate) async fn create_initial_networks(
             None
         };
 
+        // Re-read transaction-visible routing state for each configured
+        // segment so earlier inserts in this loop constrain later candidates.
+        // The final live-state check preserves legacy occupancy and cannot
+        // identify which conflict this startup transaction introduced.
+        crate::routing_safety::validate_network_segment_candidate(
+            &api.runtime_config,
+            &mut txn,
+            &ns,
+        )
+        .await?;
+
         // Capture before `save_without_reverse_zones` moves `ns`.
         // `insert_network_def` needs the id because
         // `network_def.segment_id` is FK-bound to it.
@@ -212,6 +224,7 @@ pub(crate) async fn create_initial_networks(
         );
     }
     db::dns::ensure_reverse_zones(&reverse_zone_prefixes, &mut txn).await?;
+    crate::routing_safety::validate_live_state(&api.runtime_config, &mut txn).await?;
 
     txn.commit().await?;
     Ok(())
@@ -243,6 +256,7 @@ pub(crate) async fn create_initial_vpcs(
     validate_initial_vpcs(vpcs).map_err(CarbideError::InvalidConfiguration)?;
 
     let mut txn = Transaction::begin(db_pool).await?;
+    crate::routing_safety::lock_site_mutation(&mut txn).await?;
     for (name, def) in vpcs {
         if db::vpc::find_by_name(&mut txn, name)
             .await
@@ -465,6 +479,7 @@ pub(crate) async fn store_initial_dpu_agent_upgrade_policy(
 
 pub(crate) async fn create_admin_vpc(
     db_pool: &Pool<Postgres>,
+    config: &crate::cfg::file::CarbideConfig,
     vpc_vni: Option<u32>,
 ) -> Result<(), CarbideError> {
     let Some(vpc_vni) = vpc_vni else {
@@ -474,6 +489,7 @@ pub(crate) async fn create_admin_vpc(
     };
 
     let mut txn = Transaction::begin(db_pool).await?;
+    crate::routing_safety::lock_site_mutation(&mut txn).await?;
 
     let configured_vni = vpc_vni as i32;
     let admin_segments = db::network_segment::admin(&mut txn).await?;
@@ -562,7 +578,10 @@ pub(crate) async fn create_admin_vpc(
                 }
                 Some(_) => {}
                 None => {
-                    // Attach any newly-created admin segment to the existing admin VPC.
+                    // Upgrade reconciliation may bind a legacy Admin prefix
+                    // that contains tenant space. It is not tenant reuse and
+                    // remains outside candidate admission; the post-write live
+                    // check still protects exact cross-VPC VPC-prefix reuse.
                     db::network_segment::set_vpc_id_and_can_stretch(
                         &admin_segment,
                         &mut txn,
@@ -573,6 +592,7 @@ pub(crate) async fn create_admin_vpc(
             }
         }
 
+        crate::routing_safety::validate_live_state(config, &mut txn).await?;
         txn.commit().await?;
 
         return Ok(());
@@ -608,9 +628,12 @@ pub(crate) async fn create_admin_vpc(
 
     // Attach it to admin network segments.
     for admin_segment in admin_segments {
+        // See the existing-VPC branch above: startup preserves the legacy
+        // Admin-containment contract and validates tenant reuse after binding.
         db::network_segment::set_vpc_id_and_can_stretch(&admin_segment, &mut txn, vpc.id).await?;
     }
 
+    crate::routing_safety::validate_live_state(config, &mut txn).await?;
     txn.commit().await?;
 
     Ok(())

@@ -23,7 +23,6 @@ use ::rpc::model::RpcTryFrom;
 use carbide_redfish::libredfish::RedfishAuth;
 use carbide_secrets::credentials::{BmcCredentialType, CredentialKey};
 use carbide_uuid::infiniband::IBPartitionId;
-use carbide_uuid::instance::InstanceId;
 use carbide_uuid::machine::MachineId;
 use carbide_uuid::network::NetworkSegmentId;
 use carbide_uuid::vpc::VpcId;
@@ -722,6 +721,7 @@ pub(crate) async fn release(
         .ok_or(RpcDataConversionError::MissingArgument("id"))?;
 
     let mut txn = api.txn_begin().await?;
+    crate::routing_safety::lock_site_mutation(&mut txn).await?;
 
     let instance = db::instance::find_by_id(&mut txn, instance_id)
         .await?
@@ -1236,6 +1236,10 @@ pub(crate) async fn update_instance_config(
     })?;
 
     let mut txn = api.txn_begin().await?;
+    // Current and pending old/new network configs all remain routable during an
+    // update. Take the site lock before snapshot and row locks, then retain it
+    // through the post-write graph check.
+    crate::routing_safety::lock_site_mutation(&mut txn).await?;
 
     let instance = db::instance::find_by_id(&mut txn, instance_id)
         .await?
@@ -1424,6 +1428,7 @@ pub(crate) async fn update_instance_config(
     update_instance_spx_config(&mh_snapshot, &instance, &mut config.spxconfig, &mut txn).await?;
 
     db::instance::update_config(&mut txn, instance.id, expected_version, config, metadata).await?;
+    crate::routing_safety::validate_live_state(&api.runtime_config, &mut txn).await?;
 
     let mh_snapshot = db::managed_host::load_snapshot(
         &mut txn,
@@ -1746,22 +1751,33 @@ fn snapshot_to_instance(
 }
 
 pub(super) async fn force_delete_instance(
-    instance_id: InstanceId,
+    fenced_instance: &InstanceSnapshot,
     api: &Api,
     response: &mut AdminForceDeleteMachineResponse,
 ) -> CarbideResult<()> {
-    let instance = db::instance::find_by_id(&api.database_connection, instance_id)
-        .await?
-        .ok_or_else(|| {
-            CarbideError::internal(format!("could not find an instance for {instance_id}"))
-        })?
-        .to_owned();
-
-    response.ufm_unregistrations += unbind_all_instance_ib_ports(api, &instance).await?;
+    // The caller captured this snapshot in the same site-locked transaction
+    // that marked the instance deleted and published `ForceDeletion`. Earlier
+    // config updates are therefore present, and later API updates reject the
+    // durable `instances.deleted` fence even if an old controller write replaces
+    // the machine state. Use that authoritative snapshot so UFM I/O stays
+    // outside a database transaction without reopening a read race.
+    response.ufm_unregistrations += unbind_all_instance_ib_ports(api, fenced_instance).await?;
 
     // Delete the instance and allocated address
     // TODO: This might need some changes with the new state machine
     let mut txn = api.txn_begin().await?;
+    crate::routing_safety::lock_site_mutation(&mut txn).await?;
+    // UFM work cannot hold a database transaction. Reload after serializing
+    // with routing mutations so cleanup uses the current and pending configs.
+    let Some(instance) = db::instance::find_by_id(&mut txn, fenced_instance.id).await? else {
+        // The machine controller is the only other hard-delete owner. It
+        // removes the instance, generated segments, and loopbacks in one
+        // site-locked transaction, so a missing row means DB cleanup already
+        // completed while this function performed external UFM work.
+        txn.commit().await?;
+        return Ok(());
+    };
+    let instance_id = instance.id;
     db::instance::delete(instance_id, &mut txn).await?;
 
     let mut network_segment_ids_with_vpc = vec![];

@@ -537,6 +537,11 @@ pub(crate) async fn admin_force_delete_machine(
     response.machine_unlocked = false;
 
     let mut txn = api.txn_begin().await?;
+    // Publish the instance and machine deletion fences behind the same lock as
+    // instance configuration updates. Once this transaction commits, updates
+    // that started first are visible to cleanup and updates that start later
+    // observe the fence before changing InfiniBand or routing configuration.
+    crate::routing_safety::lock_site_mutation(&mut txn).await?;
 
     let machine = match db::machine::find_by_query(&mut txn, query).await? {
         Some(machine) => machine,
@@ -603,10 +608,12 @@ pub(crate) async fn admin_force_delete_machine(
         host_machine = Some(machine);
     }
 
-    let mut instance_id = None;
-    if let Some(host_machine) = &host_machine {
-        instance_id = db::instance::find_id_by_machine_id(&mut txn, &host_machine.id).await?;
-    }
+    let instance = if let Some(host_machine) = &host_machine {
+        db::instance::find_by_machine_id(&mut txn, &host_machine.id).await?
+    } else {
+        None
+    };
+    let instance_id = instance.as_ref().map(|instance| instance.id);
 
     if let Some(host_machine) = &host_machine {
         response.managed_host_machine_id = host_machine.id.to_string();
@@ -657,6 +664,15 @@ pub(crate) async fn admin_force_delete_machine(
 
     // So far we only inspected state - now we start the deletion process
     // TODO: In the new model we might just need to move one Machine to this state
+    if let Some(instance) = &instance
+        && instance.deleted.is_none()
+    {
+        // This flag is the durable update fence. A machine-state controller
+        // finishing work from an older snapshot can overwrite `ForceDeletion`,
+        // but it cannot clear `instances.deleted`. Preserve an earlier deletion
+        // timestamp so retrying this idempotent admin operation does not move it.
+        db::instance::mark_as_deleted(instance.id, &mut txn).await?;
+    }
     if let Some(host_machine) = &host_machine {
         db::machine::advance(
             host_machine,
@@ -681,8 +697,8 @@ pub(crate) async fn admin_force_delete_machine(
     txn.commit().await?;
 
     // Note: The following deletion steps are all ordered in an idempotent fashion
-    if let Some(instance_id) = instance_id {
-        crate::handlers::instance::force_delete_instance(instance_id, api, &mut response).await?;
+    if let Some(instance) = &instance {
+        crate::handlers::instance::force_delete_instance(instance, api, &mut response).await?;
     }
 
     if let Some(machine) = &host_machine {