diff --git a/crates/api-core/src/cfg/README.md b/crates/api-core/src/cfg/README.md index b615039083..63bce887dd 100644 --- a/crates/api-core/src/cfg/README.md +++ b/crates/api-core/src/cfg/README.md @@ -42,6 +42,7 @@ Use `site_explorer.dpu_policy` instead. | `enable_route_servers` | `bool` | `false` | `networking` | Enables route server injection into DPU FRR configs for L2VPN. | | `deny_prefixes` | `Vec` | `[]` | `networking` | IPv4 and IPv6 CIDR prefixes that tenant instances are blocked from reaching. FNN generates family-specific NVUE ACL policies; all non-FNN virtualizers apply the IPv4 prefixes only. | | `site_fabric_prefixes` | `Vec` | `[]` | `networking` | IP prefixes (v4/v6) assigned for tenant use within this site. | +| `tenant_prefix_overlap_enabled` | `bool` | `false` | `networking` | Enables application admission for tenant-owned VPC prefixes to reuse CIDRs in separate, mutually isolated FNN routing domains after the legacy database exclusions are removed. Enabling the site gate alone is insufficient: every overlapping VPC must use an `overlap_eligible` safe base profile and satisfy the routing-safety preflight described below. Disabling the gate freezes expansion but still permits deletion and drain. | | `max_site_prefixes_per_tenant` | `u32` | `8` | `networking` | Maximum tenant-managed SitePrefixes retained for one tenant at this site. Prefixes awaiting removal still count against this limit and keep their CIDR reserved. | | `anycast_site_prefixes` | `Vec` | `[]` | `networking` | Aggregate IPv4 prefixes containing tenant-announced prefixes (e.g., BYOIP). **Deprecated.** Use [`routing_profiles.allowed_anycast_prefixes`](#fnnroutingprofileconfig) instead. | | `common_tenant_host_asn` | `Option` | — | `networking` | ASN that tenants use to peer with the DPU. If unset, any ASN is accepted. | @@ -635,6 +636,7 @@ client-certificate authentication is not used. | Field | Type | Default | Description | | ------- | ------ | --------- | ------------- | +| `overlap_eligible` | `bool` | `false` | Operator opt-in allowing VPCs based on this profile to participate in tenant prefix overlap. This base-profile property cannot be overridden on a VPC. | | `route_target_imports` | `Option>` | — (effective `[]`) | Route targets imported into DPU VRFs for VPC routes. | | `route_targets_on_exports` | `Option>` | — (effective `[]`) | Route targets added to routes exported by the DPU. | | `internal` | `Option` | — (effective `false`) | Whether the profile uses internal VNI allocation. This property cannot be overridden on a VPC. | @@ -649,6 +651,40 @@ Unset properties retain presence information so a VPC's inline `routing_profile_overrides` can inherit them. After the named profile and VPC override are combined, properties still unset use the effective defaults above. +Tenant prefix overlap is admitted only when both the top-level +`tenant_prefix_overlap_enabled` gate and every overlapping VPC's base-profile +`overlap_eligible` flag are true. An eligible effective profile must also set +`internal = true` and leave route-target imports/exports, underlay/default-route +leaks, tenant leak communities, and allowed anycast prefixes disabled or empty. +The site must use mutual VPC isolation, distinct actual VNIs, deny-only +non-stateful effective NSGs, and no permit-bearing global +`network_security_group.policy_overrides`. Active interface routing overrides, +a site-global VPC VNI, VMaaS, a site-wide anycast prefix, a common internal +route target, and additional FNN imports are also unsafe. VPC peering is +rejected if it would make overlapping routes visible in either direction. + +These settings and checks are staged safety plumbing. The legacy database +exclusions continue to block duplicate VPC-prefix persistence until +[#3891](https://github.com/NVIDIA/infra-controller/issues/3891) and +[#3892](https://github.com/NVIDIA/infra-controller/issues/3892) replace them; +the settings alone do not make tenant CIDR reuse available to operators. + +NICo renders one SitePrefix isolation list for the whole site. While any +retained duplicate address space exists, every graph-active tenant-serving FNN +path must satisfy the isolation rules above, including VPCs whose own prefixes +do not overlap. Only VPCs participating in an overlap must set their base +profile's `overlap_eligible` flag. A graph-active FNN VPC without a resolvable +profile fails closed; a profile or VPC that is not referenced by an address, +peering, or retained instance is not graph-active and does not block startup. + +These settings are read at process startup. Before opening its listeners, NICo +checks the retained routing graph (including resources draining after soft +deletion) and refuses to start when live duplicate tenant address space is +unsafe. This check remains active when `tenant_prefix_overlap_enabled = false`: +disabling the gate freezes expansion but cannot remove protection from retained +duplicates. The first unsafe expansion of a latent profile is rejected. Errors +intentionally do not identify another tenant's CIDR or resource. + ### `VpcDefinition` | Field | Type | Default | Description | diff --git a/crates/api-core/src/cfg/file.rs b/crates/api-core/src/cfg/file.rs index 43eeb9df60..1d434183a8 100644 --- a/crates/api-core/src/cfg/file.rs +++ b/crates/api-core/src/cfg/file.rs @@ -223,6 +223,15 @@ pub struct CarbideConfig { #[serde(default)] pub site_fabric_prefixes: Vec, + /// Enables exact-CIDR reuse across isolated tenant FNN VPCs. + /// + /// Defaults to false. Disabling the gate blocks expansion but still + /// protects retained overlap and permits contraction or drain. Participating + /// base profiles must also set `overlap_eligible`; this setting does not + /// bypass the legacy database exclusion constraints. + #[serde(default)] + pub tenant_prefix_overlap_enabled: bool, + /// Maximum number of tenant-managed SitePrefixes retained for one tenant /// at this site. Prefixes awaiting removal still count against this limit /// and keep their CIDR reserved. @@ -2432,6 +2441,13 @@ pub struct FnnConfig { #[derive(Clone, Debug, Deserialize, Serialize, PartialEq, Default)] #[serde(deny_unknown_fields)] pub struct FnnRoutingProfileConfig { + /// Allows VPCs based on this profile to participate in tenant prefix overlap. + /// + /// This operator-owned value defaults to false, cannot be overridden by a + /// VPC, and still requires the site gate plus every routing-safety check. + #[serde(default)] + pub overlap_eligible: bool, + /// These are used for import policies to import routes /// that match these targets. #[serde(default)] @@ -2513,6 +2529,7 @@ impl FnnConfig { }; Ok(Cow::Owned(FnnRoutingProfileConfig { + overlap_eligible: base_profile.overlap_eligible, route_target_imports: overrides .route_target_imports .clone() @@ -3618,7 +3635,7 @@ impl MeasuredBootMetricsCollectorConfig { } /// The VPC isolation behavior enforced within a site. -#[derive(Clone, Copy, Debug, Default, Deserialize, Serialize)] +#[derive(Clone, Copy, Debug, Default, Deserialize, Eq, PartialEq, Serialize)] #[serde(rename_all = "snake_case")] pub enum VpcIsolationBehaviorType { #[default] @@ -4328,6 +4345,7 @@ mod tests { let profile: FnnRoutingProfileConfig = Figment::new() .merge(Toml::string( r#" + overlap_eligible = true route_target_imports = [{ asn = 64512, vni = 10 }] route_targets_on_exports = [] internal = true @@ -4345,6 +4363,7 @@ mod tests { assert_eq!( profile, FnnRoutingProfileConfig { + overlap_eligible: true, route_target_imports: Some(vec![RouteTargetConfig { asn: 64512, vni: 10, @@ -4421,6 +4440,7 @@ mod tests { prefix: "192.0.2.0/24".parse().expect("valid test prefix"), }; let base = FnnRoutingProfileConfig { + overlap_eligible: true, route_target_imports: Some(vec![RouteTargetConfig { asn: 3, vni: 4 }]), route_targets_on_exports: Some(vec![inherited_export.clone()]), internal: Some(true), @@ -4453,6 +4473,7 @@ mod tests { assert_eq!( fnn.resolve_vpc_routing_profile(&vpc).unwrap().as_ref(), &FnnRoutingProfileConfig { + overlap_eligible: true, route_target_imports: Some(vec![]), route_targets_on_exports: Some(vec![inherited_export]), internal: Some(true), @@ -5124,6 +5145,7 @@ mod tests { } ); assert!(config.dhcp_servers.is_empty()); + assert!(!config.tenant_prefix_overlap_enabled); assert!(!config.allow_insecure_discovery); assert!(config.route_servers.is_empty()); assert!(config.tls.is_none()); @@ -5545,6 +5567,7 @@ mod tests { std::time::Duration::from_secs(45 * 60) ); assert_eq!(config.asn, 123); + assert!(config.tenant_prefix_overlap_enabled); assert_eq!(config.bmc_session_lockout_threshold, 4); assert_eq!( config.dhcp_servers, diff --git a/crates/api-core/src/cfg/test_data/full_config.toml b/crates/api-core/src/cfg/test_data/full_config.toml index 890d7f6bc4..b8282bb2ce 100644 --- a/crates/api-core/src/cfg/test_data/full_config.toml +++ b/crates/api-core/src/cfg/test_data/full_config.toml @@ -6,6 +6,7 @@ database_pool_acquire_timeout = "15s" database_pool_idle_timeout = "20m" database_pool_max_lifetime = "45m" asn = 123 +tenant_prefix_overlap_enabled = true dhcp_servers = ["1.2.3.4", "5.6.7.8"] ntp_servers = ["10.20.30.40", "50.60.70.80"] route_servers = ["9.10.11.12"] diff --git a/crates/api-core/src/db_init.rs b/crates/api-core/src/db_init.rs index 04aefc6f88..42526fca84 100644 --- a/crates/api-core/src/db_init.rs +++ b/crates/api-core/src/db_init.rs @@ -121,6 +121,7 @@ pub(crate) async fn create_initial_networks( networks: &HashMap, ) -> Result<(), CarbideError> { let mut txn = Transaction::begin(db_pool).await?; + crate::routing_safety::lock_site_mutation(&mut txn).await?; let domains = db::dns::domain::find_by( &mut txn, ObjectColumnFilter::::All, @@ -172,6 +173,17 @@ pub(crate) async fn create_initial_networks( None }; + // Re-read transaction-visible routing state for each configured + // segment so earlier inserts in this loop constrain later candidates. + // The final live-state check preserves legacy occupancy and cannot + // identify which conflict this startup transaction introduced. + crate::routing_safety::validate_network_segment_candidate( + &api.runtime_config, + &mut txn, + &ns, + ) + .await?; + // Capture before `save_without_reverse_zones` moves `ns`. // `insert_network_def` needs the id because // `network_def.segment_id` is FK-bound to it. @@ -212,6 +224,7 @@ pub(crate) async fn create_initial_networks( ); } db::dns::ensure_reverse_zones(&reverse_zone_prefixes, &mut txn).await?; + crate::routing_safety::validate_live_state(&api.runtime_config, &mut txn).await?; txn.commit().await?; Ok(()) @@ -243,6 +256,7 @@ pub(crate) async fn create_initial_vpcs( validate_initial_vpcs(vpcs).map_err(CarbideError::InvalidConfiguration)?; let mut txn = Transaction::begin(db_pool).await?; + crate::routing_safety::lock_site_mutation(&mut txn).await?; for (name, def) in vpcs { if db::vpc::find_by_name(&mut txn, name) .await @@ -465,6 +479,7 @@ pub(crate) async fn store_initial_dpu_agent_upgrade_policy( pub(crate) async fn create_admin_vpc( db_pool: &Pool, + config: &crate::cfg::file::CarbideConfig, vpc_vni: Option, ) -> Result<(), CarbideError> { let Some(vpc_vni) = vpc_vni else { @@ -474,6 +489,7 @@ pub(crate) async fn create_admin_vpc( }; let mut txn = Transaction::begin(db_pool).await?; + crate::routing_safety::lock_site_mutation(&mut txn).await?; let configured_vni = vpc_vni as i32; let admin_segments = db::network_segment::admin(&mut txn).await?; @@ -562,7 +578,10 @@ pub(crate) async fn create_admin_vpc( } Some(_) => {} None => { - // Attach any newly-created admin segment to the existing admin VPC. + // Upgrade reconciliation may bind a legacy Admin prefix + // that contains tenant space. It is not tenant reuse and + // remains outside candidate admission; the post-write live + // check still protects exact cross-VPC VPC-prefix reuse. db::network_segment::set_vpc_id_and_can_stretch( &admin_segment, &mut txn, @@ -573,6 +592,7 @@ pub(crate) async fn create_admin_vpc( } } + crate::routing_safety::validate_live_state(config, &mut txn).await?; txn.commit().await?; return Ok(()); @@ -608,9 +628,12 @@ pub(crate) async fn create_admin_vpc( // Attach it to admin network segments. for admin_segment in admin_segments { + // See the existing-VPC branch above: startup preserves the legacy + // Admin-containment contract and validates tenant reuse after binding. db::network_segment::set_vpc_id_and_can_stretch(&admin_segment, &mut txn, vpc.id).await?; } + crate::routing_safety::validate_live_state(config, &mut txn).await?; txn.commit().await?; Ok(()) diff --git a/crates/api-core/src/handlers/instance.rs b/crates/api-core/src/handlers/instance.rs index de1cec8b97..e0c47e38c2 100644 --- a/crates/api-core/src/handlers/instance.rs +++ b/crates/api-core/src/handlers/instance.rs @@ -23,7 +23,6 @@ use ::rpc::model::RpcTryFrom; use carbide_redfish::libredfish::RedfishAuth; use carbide_secrets::credentials::{BmcCredentialType, CredentialKey}; use carbide_uuid::infiniband::IBPartitionId; -use carbide_uuid::instance::InstanceId; use carbide_uuid::machine::MachineId; use carbide_uuid::network::NetworkSegmentId; use carbide_uuid::vpc::VpcId; @@ -722,6 +721,7 @@ pub(crate) async fn release( .ok_or(RpcDataConversionError::MissingArgument("id"))?; let mut txn = api.txn_begin().await?; + crate::routing_safety::lock_site_mutation(&mut txn).await?; let instance = db::instance::find_by_id(&mut txn, instance_id) .await? @@ -1236,6 +1236,10 @@ pub(crate) async fn update_instance_config( })?; let mut txn = api.txn_begin().await?; + // Current and pending old/new network configs all remain routable during an + // update. Take the site lock before snapshot and row locks, then retain it + // through the post-write graph check. + crate::routing_safety::lock_site_mutation(&mut txn).await?; let instance = db::instance::find_by_id(&mut txn, instance_id) .await? @@ -1424,6 +1428,7 @@ pub(crate) async fn update_instance_config( update_instance_spx_config(&mh_snapshot, &instance, &mut config.spxconfig, &mut txn).await?; db::instance::update_config(&mut txn, instance.id, expected_version, config, metadata).await?; + crate::routing_safety::validate_live_state(&api.runtime_config, &mut txn).await?; let mh_snapshot = db::managed_host::load_snapshot( &mut txn, @@ -1746,22 +1751,33 @@ fn snapshot_to_instance( } pub(super) async fn force_delete_instance( - instance_id: InstanceId, + fenced_instance: &InstanceSnapshot, api: &Api, response: &mut AdminForceDeleteMachineResponse, ) -> CarbideResult<()> { - let instance = db::instance::find_by_id(&api.database_connection, instance_id) - .await? - .ok_or_else(|| { - CarbideError::internal(format!("could not find an instance for {instance_id}")) - })? - .to_owned(); - - response.ufm_unregistrations += unbind_all_instance_ib_ports(api, &instance).await?; + // The caller captured this snapshot in the same site-locked transaction + // that marked the instance deleted and published `ForceDeletion`. Earlier + // config updates are therefore present, and later API updates reject the + // durable `instances.deleted` fence even if an old controller write replaces + // the machine state. Use that authoritative snapshot so UFM I/O stays + // outside a database transaction without reopening a read race. + response.ufm_unregistrations += unbind_all_instance_ib_ports(api, fenced_instance).await?; // Delete the instance and allocated address // TODO: This might need some changes with the new state machine let mut txn = api.txn_begin().await?; + crate::routing_safety::lock_site_mutation(&mut txn).await?; + // UFM work cannot hold a database transaction. Reload after serializing + // with routing mutations so cleanup uses the current and pending configs. + let Some(instance) = db::instance::find_by_id(&mut txn, fenced_instance.id).await? else { + // The machine controller is the only other hard-delete owner. It + // removes the instance, generated segments, and loopbacks in one + // site-locked transaction, so a missing row means DB cleanup already + // completed while this function performed external UFM work. + txn.commit().await?; + return Ok(()); + }; + let instance_id = instance.id; db::instance::delete(instance_id, &mut txn).await?; let mut network_segment_ids_with_vpc = vec![]; diff --git a/crates/api-core/src/handlers/machine.rs b/crates/api-core/src/handlers/machine.rs index ffa8de4a20..ee543504c4 100644 --- a/crates/api-core/src/handlers/machine.rs +++ b/crates/api-core/src/handlers/machine.rs @@ -537,6 +537,11 @@ pub(crate) async fn admin_force_delete_machine( response.machine_unlocked = false; let mut txn = api.txn_begin().await?; + // Publish the instance and machine deletion fences behind the same lock as + // instance configuration updates. Once this transaction commits, updates + // that started first are visible to cleanup and updates that start later + // observe the fence before changing InfiniBand or routing configuration. + crate::routing_safety::lock_site_mutation(&mut txn).await?; let machine = match db::machine::find_by_query(&mut txn, query).await? { Some(machine) => machine, @@ -603,10 +608,12 @@ pub(crate) async fn admin_force_delete_machine( host_machine = Some(machine); } - let mut instance_id = None; - if let Some(host_machine) = &host_machine { - instance_id = db::instance::find_id_by_machine_id(&mut txn, &host_machine.id).await?; - } + let instance = if let Some(host_machine) = &host_machine { + db::instance::find_by_machine_id(&mut txn, &host_machine.id).await? + } else { + None + }; + let instance_id = instance.as_ref().map(|instance| instance.id); if let Some(host_machine) = &host_machine { response.managed_host_machine_id = host_machine.id.to_string(); @@ -657,6 +664,15 @@ pub(crate) async fn admin_force_delete_machine( // So far we only inspected state - now we start the deletion process // TODO: In the new model we might just need to move one Machine to this state + if let Some(instance) = &instance + && instance.deleted.is_none() + { + // This flag is the durable update fence. A machine-state controller + // finishing work from an older snapshot can overwrite `ForceDeletion`, + // but it cannot clear `instances.deleted`. Preserve an earlier deletion + // timestamp so retrying this idempotent admin operation does not move it. + db::instance::mark_as_deleted(instance.id, &mut txn).await?; + } if let Some(host_machine) = &host_machine { db::machine::advance( host_machine, @@ -681,8 +697,8 @@ pub(crate) async fn admin_force_delete_machine( txn.commit().await?; // Note: The following deletion steps are all ordered in an idempotent fashion - if let Some(instance_id) = instance_id { - crate::handlers::instance::force_delete_instance(instance_id, api, &mut response).await?; + if let Some(instance) = &instance { + crate::handlers::instance::force_delete_instance(instance, api, &mut response).await?; } if let Some(machine) = &host_machine { diff --git a/crates/api-core/src/handlers/network_security_group.rs b/crates/api-core/src/handlers/network_security_group.rs index c2f39be830..487874444b 100644 --- a/crates/api-core/src/handlers/network_security_group.rs +++ b/crates/api-core/src/handlers/network_security_group.rs @@ -97,6 +97,7 @@ pub(crate) async fn create( // Start a new transaction for a db write. let mut txn = api.txn_begin().await?; + crate::routing_safety::lock_site_mutation(&mut txn).await?; // Write a new NetworkSecurityGroup to the DB and get back // our new NetworkSecurityGroup. @@ -308,6 +309,7 @@ pub(crate) async fn update( // Start a new transaction for a db write. let mut txn = api.txn_begin().await?; + crate::routing_safety::lock_site_mutation(&mut txn).await?; // Look up the NetworkSecurityGroup. We'll need to check the current // version. We could probably do everything with a single query @@ -380,6 +382,7 @@ pub(crate) async fn update( None, ) .await?; + crate::routing_safety::validate_live_state(&api.runtime_config, &mut txn).await?; // Prepare the response to send back let rpc_out = rpc::UpdateNetworkSecurityGroupResponse { @@ -420,6 +423,7 @@ pub(crate) async fn delete( // Prepare our txn to delete from the DB let mut txn = api.txn_begin().await?; + crate::routing_safety::lock_site_mutation(&mut txn).await?; // Make our DB query for the NetworkSecurityGroup. // This is mainly to get a row-level lock if the record exists diff --git a/crates/api-core/src/handlers/network_segment.rs b/crates/api-core/src/handlers/network_segment.rs index bb6b0138b0..09f0d7881c 100644 --- a/crates/api-core/src/handlers/network_segment.rs +++ b/crates/api-core/src/handlers/network_segment.rs @@ -131,6 +131,7 @@ pub(crate) async fn create( } let mut txn = api.txn_begin().await?; + crate::routing_safety::lock_site_mutation(&mut txn).await?; let allocate_svi_ip = if let Some(vpc_id) = new_network_segment.vpc_id { let vpcs = db::vpc::find_by( @@ -156,7 +157,15 @@ pub(crate) async fn create( false }; + crate::routing_safety::validate_network_segment_candidate( + &api.runtime_config, + &mut txn, + &new_network_segment, + ) + .await?; + let network_segment = save(api, &mut txn, new_network_segment, false, allocate_svi_ip).await?; + crate::routing_safety::validate_live_state(&api.runtime_config, &mut txn).await?; txn.commit().await?; @@ -181,6 +190,7 @@ pub(crate) async fn attach_to_vpc( let vpc_id = vpc_id.ok_or(CarbideError::MissingArgument("vpc_id"))?; let mut txn = api.txn_begin().await?; + crate::routing_safety::lock_site_mutation(&mut txn).await?; let vpcs = db::vpc::find_by_with_lock( txn.as_mut(), @@ -228,9 +238,20 @@ pub(crate) async fn attach_to_vpc( )) .into()); } - _ => db::network_segment::attach_to_vpc(&segment, txn.as_mut(), vpc_id).await?, + _ => { + crate::routing_safety::validate_network_segment_attachment( + &api.runtime_config, + &mut txn, + &segment, + vpc_id, + ) + .await?; + db::network_segment::attach_to_vpc(&segment, txn.as_mut(), vpc_id).await? + } }; + crate::routing_safety::validate_live_state(&api.runtime_config, &mut txn).await?; + txn.commit().await?; Ok(Response::new(network_segment.into())) } @@ -242,6 +263,7 @@ pub(crate) async fn delete( crate::api::log_request_data(&request); let mut txn = api.txn_begin().await?; + crate::routing_safety::lock_site_mutation(&mut txn).await?; let rpc::NetworkSegmentDeletionRequest { id, .. } = request.into_inner(); diff --git a/crates/api-core/src/handlers/site_prefix.rs b/crates/api-core/src/handlers/site_prefix.rs index 2784b0124c..dd3df69cc7 100644 --- a/crates/api-core/src/handlers/site_prefix.rs +++ b/crates/api-core/src/handlers/site_prefix.rs @@ -223,6 +223,16 @@ pub(crate) async fn create( return Err(CarbideError::from(error).into()); } }; + if let Err(error) = crate::routing_safety::lock_site_mutation(&mut txn).await { + emit_admission( + SitePrefixAdmissionResult::Failed, + &site_prefix_id, + &tenant_organization_id, + &prefix, + &error, + ); + return Err(error.into()); + } let result = match db::site_prefix::create_tenant_managed( new_site_prefix, quota_limit, @@ -326,6 +336,7 @@ pub(crate) async fn delete( let retire = RetireTenantManagedSitePrefix::try_from(request.into_inner())?; let mut txn = api.txn_begin().await?; + crate::routing_safety::lock_site_mutation(&mut txn).await?; let current = db::site_prefix::find_by_id_for_update(&mut txn, retire.id) .await? .ok_or_else(|| CarbideError::NotFoundError { diff --git a/crates/api-core/src/handlers/vpc.rs b/crates/api-core/src/handlers/vpc.rs index b13ee050d0..843cd064fb 100644 --- a/crates/api-core/src/handlers/vpc.rs +++ b/crates/api-core/src/handlers/vpc.rs @@ -44,6 +44,7 @@ pub(crate) async fn create( let vpc_creation_request = request.get_ref(); let mut txn = api.txn_begin().await?; + crate::routing_safety::lock_site_mutation(&mut txn).await?; // Grab the tenant details and a row-lock if found so we can coordinate around the tenant record. // If we're still allowing VPC creation for tenant org IDs that don't actually exist @@ -171,6 +172,7 @@ pub(crate) async fn update( })?; let mut txn = api.txn_begin().await?; + crate::routing_safety::lock_site_mutation(&mut txn).await?; // Security-group and routing-profile changes both require validation // against the VPC's persisted tenant and virtualization type. @@ -245,6 +247,7 @@ pub(crate) async fn update( // It's better to keep the property immutable. let vpc = db::vpc::update(&vpc_update, &mut txn).await?; + crate::routing_safety::validate_live_state(&api.runtime_config, &mut txn).await?; txn.commit().await?; @@ -260,6 +263,7 @@ pub(crate) async fn update_virtualization( log_request_data(&request); let mut txn = api.txn_begin().await?; + crate::routing_safety::lock_site_mutation(&mut txn).await?; let updater = UpdateVpcVirtualization::try_from(request.into_inner())?; @@ -305,6 +309,7 @@ pub(crate) async fn update_virtualization( .into()); } db::vpc::update_virtualization(&updater, &mut txn).await?; + crate::routing_safety::validate_live_state(&api.runtime_config, &mut txn).await?; txn.commit().await?; @@ -318,6 +323,7 @@ pub(crate) async fn delete( log_request_data(&request); let mut txn = api.txn_begin().await?; + crate::routing_safety::lock_site_mutation(&mut txn).await?; // TODO: This needs to validate that nothing references the VPC anymore // (like NetworkSegments) diff --git a/crates/api-core/src/handlers/vpc_peering.rs b/crates/api-core/src/handlers/vpc_peering.rs index cffb7b541e..5b0235fb3e 100644 --- a/crates/api-core/src/handlers/vpc_peering.rs +++ b/crates/api-core/src/handlers/vpc_peering.rs @@ -49,6 +49,7 @@ pub(crate) async fn create( peer_vpc_id.ok_or_else(|| CarbideError::MissingArgument("peer_vpc_id cannot be null"))?; let mut txn = api.txn_begin().await?; + crate::routing_safety::lock_site_mutation(&mut txn).await?; // Check this VPC peering is permitted under current site vpc_peering_policy match api.runtime_config.vpc_peering_policy { @@ -86,6 +87,7 @@ pub(crate) async fn create( } let vpc_peering = db::create(&mut txn, vpc_id, peer_vpc_id, id).await?; + crate::routing_safety::validate_live_state(&api.runtime_config, &mut txn).await?; txn.commit().await?; @@ -141,6 +143,7 @@ pub(crate) async fn delete( let id = id.ok_or_else(|| CarbideError::MissingArgument("id cannot be null"))?; let mut txn = api.txn_begin().await?; + crate::routing_safety::lock_site_mutation(&mut txn).await?; let _ = db::delete(&mut txn, id).await?; diff --git a/crates/api-core/src/handlers/vpc_prefix.rs b/crates/api-core/src/handlers/vpc_prefix.rs index 658a013542..746ec3e3e4 100644 --- a/crates/api-core/src/handlers/vpc_prefix.rs +++ b/crates/api-core/src/handlers/vpc_prefix.rs @@ -19,6 +19,7 @@ use ::db::{ObjectColumnFilter, vpc_prefix as db}; use ::rpc::forge as rpc; use ::rpc::forge::PrefixMatchType; use carbide_network::virtualization::VpcVirtualizationType; +use carbide_uuid::vpc::VpcId; use ipnetwork::IpNetwork; use model::network_prefix::NetworkPrefix; use model::site_prefix::{ @@ -39,6 +40,25 @@ fn contains_prefix(parent: IpNetwork, child: IpNetwork) -> bool { } } +/// `adoptable_segment_prefixes` keeps overlapping segment prefixes owned by +/// the candidate VPC. +/// +/// Foreign prefixes remain in the common routing snapshot so one candidate +/// evaluator owns their rejection. Generated children remain in this list so +/// the caller's existing-parent guard rejects them; only direct prefixes are +/// ultimately adopted. +fn adoptable_segment_prefixes( + segment_prefixes: Vec<(VpcId, NetworkPrefix)>, + vpc_id: VpcId, +) -> Vec { + segment_prefixes + .into_iter() + .filter_map(|(segment_vpc_id, segment_prefix)| { + (segment_vpc_id == vpc_id).then_some(segment_prefix) + }) + .collect() +} + fn validate_site_prefix_attachment( site_prefix: &SitePrefix, vpc: &Vpc, @@ -113,6 +133,7 @@ pub(crate) async fn create( } let mut txn = api.txn_begin().await?; + crate::routing_safety::lock_site_mutation(&mut txn).await?; // Resolve and lock the exact SitePrefix before locking the VPC. The shared // row lock permits concurrent child creation but conflicts with retirement @@ -220,48 +241,9 @@ pub(crate) async fn create( } let expected_vpc_version = vpc.version; - let conflicting_vpc_prefixes = db::probe(new_prefix.config.prefix, &mut txn).await?; - if !conflicting_vpc_prefixes.is_empty() { - let conflicting_vpc_prefixes = conflicting_vpc_prefixes - .into_iter() - .map(|p| p.config.prefix); - let conflicting_vpc_prefixes = itertools::join(conflicting_vpc_prefixes, ", "); - let msg = format!( - "The requested VPC prefix ({vpc_prefix}) overlaps at least one \ - existing VPC prefix ({conflicting_vpc_prefixes})", - vpc_prefix = new_prefix.config.prefix, - ); - return Err(CarbideError::InvalidArgument(msg).into()); - } - let segment_prefixes = db::probe_segment_prefixes(new_prefix.config.prefix, &mut txn).await?; - // Check that all the prefixes we found are on segments that belong to our - // own VPC. - let segment_prefixes: Vec = { - let (own_segment_prefixes, foreign_segment_prefixes) = segment_prefixes - .into_iter() - .partition::, _>(|(segment_vpc_id, _)| segment_vpc_id == &new_prefix.vpc_id); - - if !foreign_segment_prefixes.is_empty() { - let foreign_segment_prefixes = foreign_segment_prefixes - .into_iter() - .map(|(_, np)| np.prefix); - let foreign_segment_prefixes = itertools::join(foreign_segment_prefixes, ", "); - let msg = format!( - "The requested VPC prefix of {vpc_prefix} conflicts with at \ - least one network segment prefix ({foreign_segment_prefixes}) \ - owned by another VPC", - vpc_prefix = new_prefix.config.prefix, - ); - return Err(CarbideError::InvalidArgument(msg).into()); - } - // We don't need the associated VpcIds anymore, get rid of them. - own_segment_prefixes - .into_iter() - .map(|(_, segment_prefix)| segment_prefix) - .collect() - }; + let segment_prefixes = adoptable_segment_prefixes(segment_prefixes, new_prefix.vpc_id); // Check that the network segment prefixes we found can actually fit into // this new VPC prefix container. @@ -303,6 +285,18 @@ pub(crate) async fn create( .validate(true) .map_err(CarbideError::from)?; + let adopted_network_prefix_ids = segment_prefixes + .iter() + .map(|prefix| prefix.id) + .collect::>(); + crate::routing_safety::validate_vpc_prefix_candidate( + &api.runtime_config, + &mut txn, + &new_prefix, + &adopted_network_prefix_ids, + ) + .await?; + let vpc_prefix = db::persist(new_prefix, expected_vpc_version, &mut txn).await?; let vpc_prefix_id = vpc_prefix.id; let vpc_prefix_network = vpc_prefix.config.prefix; @@ -317,6 +311,7 @@ pub(crate) async fn create( ) .await?; } + crate::routing_safety::validate_live_state(&api.runtime_config, &mut txn).await?; // Reload through the normal read path so create responses include computed utilization stats. let vpc_prefix = db::get_by_id( @@ -513,6 +508,7 @@ pub(crate) async fn delete( let delete_prefix = vpc_prefix::DeleteVpcPrefix::try_from(request.into_inner())?; let mut txn = api.txn_begin().await?; + crate::routing_safety::lock_site_mutation(&mut txn).await?; // Load the active prefix so repeat deletes preserve current NotFound // behavior unless the DB layer deliberately makes soft-delete idempotent. @@ -548,3 +544,54 @@ pub(crate) async fn delete( Ok(tonic::Response::new(rpc::VpcPrefixDeletionResult {})) } + +#[cfg(test)] +mod routing_safety_tests { + use carbide_uuid::network::{NetworkPrefixId, NetworkSegmentId}; + use carbide_uuid::vpc::VpcPrefixId; + + use super::*; + + fn prefix(vpc_prefix_id: Option) -> NetworkPrefix { + NetworkPrefix { + id: NetworkPrefixId::new(), + segment_id: NetworkSegmentId::new(), + prefix: "192.0.2.0/31".parse().unwrap(), + gateway: None, + dhcpv6_link_address: None, + num_reserved: 0, + vpc_prefix_id, + vpc_prefix: vpc_prefix_id.map(|_| "192.0.2.0/24".parse().unwrap()), + svi_ip: None, + num_free_ips: None, + } + } + + #[test] + fn adoptable_segment_prefixes_keep_only_candidate_vpc_overlaps() { + let candidate_vpc_id = VpcId::new(); + let foreign_vpc_id = VpcId::new(); + + let cases = [ + ("candidate direct prefix", true, false, 1), + ("candidate generated prefix", true, true, 1), + ("foreign direct prefix", false, false, 0), + ("foreign generated prefix", false, true, 0), + ]; + for (scenario, owned_by_candidate, generated, expected_count) in cases { + let owner = if owned_by_candidate { + candidate_vpc_id + } else { + foreign_vpc_id + }; + let vpc_prefix_id = generated.then(VpcPrefixId::new); + let result = + adoptable_segment_prefixes(vec![(owner, prefix(vpc_prefix_id))], candidate_vpc_id); + + assert_eq!(result.len(), expected_count, "{scenario}"); + if let Some(result) = result.first() { + assert_eq!(result.vpc_prefix_id.is_some(), generated, "{scenario}"); + } + } + } +} diff --git a/crates/api-core/src/instance/mod.rs b/crates/api-core/src/instance/mod.rs index 7ece8c7d80..9f180ca8b5 100644 --- a/crates/api-core/src/instance/mod.rs +++ b/crates/api-core/src/instance/mod.rs @@ -1458,6 +1458,10 @@ pub(crate) async fn batch_allocate_instances( // Start a single transaction for all allocations let mut txn = api.txn_begin().await?; + // Network allocation can create generated VPC-prefix segments. Hold the + // site lock from the first database read through final graph validation so + // the complete batch is admitted atomically. + crate::routing_safety::lock_site_mutation(&mut txn).await?; // ==== Phase 2: Check against allocations for tenants in requests ==== @@ -2112,6 +2116,8 @@ pub(crate) async fn batch_allocate_instances( .collect(); db::instance::batch_update_spx_config(&mut txn, &spx_refs, false).await?; + crate::routing_safety::validate_live_state(&api.runtime_config, &mut txn).await?; + // ==== Phase 9: Load final instances ==== let machine_id_refs: Vec<&MachineId> = processed_requests .iter() diff --git a/crates/api-core/src/lib.rs b/crates/api-core/src/lib.rs index 37ccf9191c..caaea1e242 100644 --- a/crates/api-core/src/lib.rs +++ b/crates/api-core/src/lib.rs @@ -71,6 +71,7 @@ mod measured_boot; mod mqtt_state_change_hook; mod network_segment; mod node_auth; +mod routing_safety; mod scout_stream; pub mod secrets; mod setup; diff --git a/crates/api-core/src/routing_safety.rs b/crates/api-core/src/routing_safety.rs new file mode 100644 index 0000000000..c0b512ccae --- /dev/null +++ b/crates/api-core/src/routing_safety.rs @@ -0,0 +1,3291 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +//! Admission for tenant address-space reuse across the site routing graph. +//! +//! Every routing-graph writer first takes one transaction-scoped site lock, +//! then compares its candidate state with the rows visible in that transaction. +//! Existing overlap is preserved during contraction, while newly introduced +//! overlap must be exact cross-VPC tenant reuse and pass the full isolation +//! policy. Startup runs the same live-state check after any database seeding and +//! before routing controllers or listeners start. +//! +//! The evaluator loads addresses first and avoids the more expensive peering, +//! security-group, and retained-instance projection when no tenant reuse is +//! present. Detailed resource identifiers stay in operator logs; client errors +//! deliberately expose only the stable violation category. + +use std::collections::{HashMap, HashSet}; +use std::fmt; + +use carbide_network::virtualization::VpcVirtualizationType; +use carbide_uuid::instance::InstanceId; +use carbide_uuid::network::{NetworkPrefixId, NetworkSegmentId}; +use carbide_uuid::network_security_group::NetworkSecurityGroupId; +use carbide_uuid::site_prefix::SitePrefixId; +use carbide_uuid::vpc::{VpcId, VpcPrefixId}; +use ipnetwork::IpNetwork; +use model::instance::config::network::{ + InstanceInterfaceRoutingProfile, InstanceNetworkConfig, NetworkDetails, +}; +use model::network_security_group::{NetworkSecurityGroup, NetworkSecurityGroupRuleAction}; +use model::network_segment::{NetworkSegment, NetworkSegmentType, NewNetworkSegment}; +use model::site_prefix::{ + SitePrefix, SitePrefixAuthority, SitePrefixLifecycleState, SitePrefixRoutingScope, +}; +use model::vpc::{Vpc, VpcVirtualizationTypeCapabilities}; +use model::vpc_prefix::NewVpcPrefix; +use sqlx::PgConnection; + +use crate::cfg::file::{ + CarbideConfig, FnnRoutingProfileConfig, VpcIsolationBehaviorType, VpcPeeringPolicy, +}; +use crate::{CarbideError, CarbideResult}; + +const OVERLAPPING_ADDRESS_SPACE: &str = + "requested address space overlaps existing routed address space"; +const INELIGIBLE_OVERLAP: &str = + "requested address space is not eligible for tenant prefix overlap"; +const REACHABLE_OVERLAP: &str = "requested routing would expose overlapping tenant address space"; +const UNSAFE_POLICY: &str = + "active tenant routing policy is not safe for overlapping address space"; + +/// Identifies the resource category that contributes a routed prefix. +/// +/// `Vpc` retains the optional SitePrefix link used for eligibility. `Network` +/// has no ID for a candidate segment prefix that has not been persisted. +#[derive(Clone, Debug, Eq, PartialEq)] +enum AddressSource { + /// A VPC prefix whose SitePrefix association determines reuse eligibility. + Vpc { + site_prefix_id: Option, + }, + /// A direct NetworkSegment prefix, which is never eligible for reuse. + Network { id: Option }, +} + +/// Stable typed identity for one persisted or candidate routed address. +/// +/// Variant order matches the rendered namespace order. Derived ordering gives +/// overlap pairs a canonical identity and gives callers a stable order when +/// they explicitly sort an address inventory. +#[derive(Clone, Copy, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)] +enum AddressKey { + CandidateNetwork(usize), + CandidateVpc(VpcPrefixId), + Network(NetworkPrefixId), + Vpc(VpcPrefixId), +} + +impl fmt::Display for AddressKey { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::CandidateNetwork(index) => { + write!(formatter, "candidate-network-prefix:{index}") + } + Self::CandidateVpc(id) => write!(formatter, "candidate-vpc-prefix:{id}"), + Self::Network(id) => write!(formatter, "network-prefix:{id}"), + Self::Vpc(id) => write!(formatter, "vpc-prefix:{id}"), + } + } +} + +/// Normalized address inventory used by both persisted and candidate checks. +#[derive(Clone, Debug)] +struct RoutedAddress { + /// Stable identity used to distinguish pre-existing overlap from a new pair. + key: AddressKey, + vpc_id: VpcId, + prefix: IpNetwork, + source: AddressSource, +} + +/// Routing references retained by one instance across current and pending config. +/// +/// Pending old and new configurations both remain relevant until the transition +/// completes. An unresolved retained reference therefore fails closed whenever +/// tenant address reuse is active. +#[derive(Debug)] +struct InstancePath { + instance_id: InstanceId, + vpc_ids: HashSet, + network_security_group_id: Option, + has_routing_override: bool, + has_unresolved_reference: bool, +} + +/// Transaction-consistent routing state evaluated for one admission decision. +/// +/// `from_addresses` builds the inexpensive address phase. `apply_policy_paths` +/// enriches it only when an exact tenant-reuse pair requires policy analysis. +#[derive(Debug)] +struct RoutingState<'a> { + vpcs: HashMap, + site_prefixes: HashMap, + addresses: Vec, + admin_only_vpcs: HashSet, + peerings: Vec<(VpcId, VpcId)>, + network_security_groups: HashMap, + instance_paths: Vec, +} + +/// Cached result of the single pairwise address scan for a `RoutingState`. +/// +/// `occupancy_pairs` detects candidate expansion, `all_indices` covers every +/// containment conflict, and `tenant_reuse_indices` selects exact cross-VPC +/// VPC-prefix pairs that require the full routing-policy check. +#[derive(Debug)] +struct OverlapAnalysis { + occupancy_pairs: HashSet<(AddressKey, AddressKey)>, + all_indices: Vec<(usize, usize)>, + tenant_reuse_indices: Vec<(usize, usize)>, +} + +impl<'a> RoutingState<'a> { + /// Builds the address-only state while retaining soft-deleted routing rows. + /// + /// Generated NetworkPrefix children are omitted because their VpcPrefix + /// parent is the routed claim; direct segment prefixes remain independent. + fn from_addresses(snapshot: &'a db::routing_safety::RoutingAddressSnapshot) -> Self { + let vpcs = snapshot.vpcs.iter().map(|vpc| (vpc.id, vpc)).collect(); + let site_prefixes = snapshot + .site_prefixes + .iter() + .map(|prefix| (prefix.id, prefix)) + .collect(); + let mut addresses = snapshot + .vpc_prefixes + .iter() + .map(|prefix| RoutedAddress { + key: AddressKey::Vpc(prefix.id), + vpc_id: prefix.vpc_id, + prefix: prefix.config.prefix, + source: AddressSource::Vpc { + site_prefix_id: prefix.site_prefix_id, + }, + }) + .collect::>(); + + let mut vpcs_with_admin_segments = HashSet::new(); + let mut vpcs_with_non_admin_segments = HashSet::new(); + for segment in &snapshot.network_segments { + let Some(vpc_id) = segment.config.vpc_id else { + continue; + }; + if segment.config.segment_type == NetworkSegmentType::Admin { + vpcs_with_admin_segments.insert(vpc_id); + } else { + vpcs_with_non_admin_segments.insert(vpc_id); + } + addresses.extend( + segment + .prefixes + .iter() + .filter(|prefix| prefix.vpc_prefix_id.is_none()) + .map(|prefix| RoutedAddress { + key: AddressKey::Network(prefix.id), + vpc_id, + prefix: prefix.prefix, + source: AddressSource::Network { + id: Some(prefix.id), + }, + }), + ); + } + addresses.sort_by_key(|address| address.key); + let admin_only_vpcs = vpcs_with_admin_segments + .difference(&vpcs_with_non_admin_segments) + .copied() + .collect(); + + Self { + vpcs, + site_prefixes, + addresses, + admin_only_vpcs, + peerings: Vec::new(), + network_security_groups: HashMap::new(), + instance_paths: Vec::new(), + } + } + + /// Adds peer visibility and every retained instance path to the address state. + /// + /// Callers invoke this only after exact tenant reuse is found. Current and + /// pending old/new instance configs are unioned, and a missing retained row + /// remains unresolved so policy validation fails closed. + fn apply_policy_paths( + &mut self, + snapshot: &db::routing_safety::RoutingPolicySnapshot, + addresses: &db::routing_safety::RoutingAddressSnapshot, + ) { + let segment_vpcs = addresses + .network_segments + .iter() + .map(|segment| (segment.id, segment.config.vpc_id)) + .collect::>(); + let vpc_prefix_vpcs = addresses + .vpc_prefixes + .iter() + .map(|prefix| (prefix.id, prefix.vpc_id)) + .collect::>(); + self.instance_paths = snapshot + .instances + .iter() + .map(|instance| { + let mut configs = vec![&instance.network_config.0]; + if let Some(update) = &instance.update_network_config_request { + configs.push(&update.0.old_config); + configs.push(&update.0.new_config); + } + let mut vpc_ids = HashSet::new(); + let mut has_routing_override = false; + let mut has_unresolved_reference = false; + for config in configs { + collect_instance_path( + config, + &segment_vpcs, + &vpc_prefix_vpcs, + &mut vpc_ids, + &mut has_routing_override, + &mut has_unresolved_reference, + ); + } + InstancePath { + instance_id: instance.id, + vpc_ids, + network_security_group_id: instance.network_security_group_id.clone(), + has_routing_override, + has_unresolved_reference, + } + }) + .collect(); + self.instance_paths.sort_by_key(|path| path.instance_id); + self.peerings = snapshot + .peerings + .iter() + .map(|peering| (peering.vpc_id, peering.peer_vpc_id)) + .collect(); + self.peerings.sort(); + self.network_security_groups = snapshot + .network_security_groups + .iter() + .map(|group| (group.id.clone(), group.clone())) + .collect(); + } + + /// Returns whether this is the configured, unconsumed startup admin VPC. + /// + /// Segment type alone is insufficient: callers can create `Admin` segments. + /// The startup VPC is identified by its internal tenant and configured VNI. + /// Its exemption is revoked as soon as a retained instance or peering + /// references it; no consumed path is assumed to remain control-only. + fn is_unconsumed_admin_vpc(&self, config: &CarbideConfig, vpc: &Vpc) -> bool { + let Some(configured_vni) = config + .fnn + .as_ref() + .and_then(|fnn| fnn.admin_vpc.as_ref()) + .filter(|admin| admin.enabled) + .and_then(|admin| admin.vpc_vni) + .and_then(|vni| i32::try_from(vni).ok()) + else { + return false; + }; + + self.admin_only_vpcs.contains(&vpc.id) + && vpc.config.tenant_organization_id == "carbide_internal" + && vpc.config.network_virtualization_type == VpcVirtualizationType::Fnn + && vpc.config.vni == Some(configured_vni) + && vpc.status.vni == Some(configured_vni) + && !self + .peerings + .iter() + .any(|(left, right)| *left == vpc.id || *right == vpc.id) + && !self + .instance_paths + .iter() + .any(|path| path.vpc_ids.contains(&vpc.id)) + } + + /// Scans the address inventory once and classifies every overlapping pair. + /// + /// Admission and policy checks reuse the returned indices; they remain + /// valid because neither phase reorders `addresses`. The complete + /// site-wide comparison is deliberate until + /// [#3891](https://github.com/NVIDIA/infra-controller/issues/3891) and + /// [#3892](https://github.com/NVIDIA/infra-controller/issues/3892) replace + /// the legacy global database exclusions: candidate admission must still + /// see tolerated direct and same-VPC occupancy. Policy rows are loaded + /// lazily only after this address phase finds an exact tenant-reuse pair. + fn analyze_overlaps(&self) -> OverlapAnalysis { + let mut occupancy_pairs = HashSet::new(); + let mut all_indices = Vec::new(); + let mut tenant_reuse_indices = Vec::new(); + for (left_index, left) in self.addresses.iter().enumerate() { + for (right_offset, right) in self.addresses[left_index + 1..].iter().enumerate() { + if !prefixes_overlap(left.prefix, right.prefix) { + continue; + } + let indices = (left_index, left_index + right_offset + 1); + all_indices.push(indices); + if is_tenant_reuse_pair(left, right) { + tenant_reuse_indices.push(indices); + } + let pair = if left.key < right.key { + (left.key, right.key) + } else { + (right.key, left.key) + }; + occupancy_pairs.insert(pair); + } + } + OverlapAnalysis { + occupancy_pairs, + all_indices, + tenant_reuse_indices, + } + } + + /// Validates the SitePrefix association, profile, tenant, CIDR, and VNI of + /// every reuse pair. + fn validate_tenant_reuse( + &self, + config: &CarbideConfig, + analysis: &OverlapAnalysis, + ) -> Result<(), RoutingSafetyFailure> { + for (left_index, right_index) in &analysis.tenant_reuse_indices { + let left = &self.addresses[*left_index]; + let right = &self.addresses[*right_index]; + self.validate_overlap_eligibility(config, left, right)?; + } + + Ok(()) + } + + /// Validates site-wide policy and receiver visibility after reuse is present. + fn validate_policy_paths( + &self, + config: &CarbideConfig, + analysis: &OverlapAnalysis, + ) -> Result<(), RoutingSafetyFailure> { + self.validate_site_policy(config) + .map_err(|failure| self.with_overlap_context(failure, analysis))?; + self.validate_active_paths(config)?; + self.validate_reachability(config, analysis)?; + Ok(()) + } + + /// Runs both evaluator phases for pure, table-driven policy tests. + #[cfg(test)] + fn validate(&self, config: &CarbideConfig) -> Result<(), RoutingSafetyFailure> { + let analysis = self.analyze_overlaps(); + self.validate_tenant_reuse(config, &analysis)?; + if analysis.tenant_reuse_indices.is_empty() { + return Ok(()); + } + self.validate_policy_paths(config, &analysis) + } + + /// Attaches one affected pair to a site-wide failure for operator diagnosis. + fn with_overlap_context( + &self, + mut failure: RoutingSafetyFailure, + analysis: &OverlapAnalysis, + ) -> RoutingSafetyFailure { + if let Some((left_index, right_index)) = analysis.tenant_reuse_indices.first() { + let left = &self.addresses[*left_index]; + let right = &self.addresses[*right_index]; + failure.vpc_ids.extend([left.vpc_id, right.vpc_id]); + failure + .resource_ids + .extend([left.key.to_string(), right.key.to_string()]); + } + failure + } + + /// Checks the complete per-pair contract before policy-path analysis. + /// + /// Reuse requires an equal CIDR in separate FNN VPCs and tenants, valid + /// tenant-managed SitePrefix ownership and containment, overlap-safe base + /// profiles, and distinct effective VNIs. + fn validate_overlap_eligibility( + &self, + config: &CarbideConfig, + left: &RoutedAddress, + right: &RoutedAddress, + ) -> Result<(), RoutingSafetyFailure> { + if left.prefix != right.prefix + || matches!(left.source, AddressSource::Network { .. }) + || matches!(right.source, AddressSource::Network { .. }) + { + return Err(RoutingSafetyFailure::for_pair( + RoutingSafetyViolation::IneligibleOverlap, + "prefixes_not_equal_or_direct", + left, + right, + )); + } + let left_vpc = self.vpcs.get(&left.vpc_id).ok_or_else(|| { + RoutingSafetyFailure::for_pair( + RoutingSafetyViolation::IneligibleOverlap, + "missing_vpc", + left, + right, + ) + })?; + let right_vpc = self.vpcs.get(&right.vpc_id).ok_or_else(|| { + RoutingSafetyFailure::for_pair( + RoutingSafetyViolation::IneligibleOverlap, + "missing_vpc", + left, + right, + ) + })?; + if left_vpc.config.network_virtualization_type != VpcVirtualizationType::Fnn + || right_vpc.config.network_virtualization_type != VpcVirtualizationType::Fnn + || left_vpc.config.tenant_organization_id == right_vpc.config.tenant_organization_id + { + return Err(RoutingSafetyFailure::for_pair( + RoutingSafetyViolation::IneligibleOverlap, + "vpc_type_or_tenant_ineligible", + left, + right, + )); + } + self.validate_vpc_prefix_site_association(left_vpc, left)?; + self.validate_vpc_prefix_site_association(right_vpc, right)?; + self.validate_profile(config, left_vpc, profile_allows_overlap)?; + self.validate_profile(config, right_vpc, profile_allows_overlap)?; + + let left_vni = left_vpc.status.vni.ok_or_else(|| { + RoutingSafetyFailure::for_vpc( + RoutingSafetyViolation::UnsafePolicy, + "missing_vpc_vni", + left_vpc.id, + ) + })?; + let right_vni = right_vpc.status.vni.ok_or_else(|| { + RoutingSafetyFailure::for_vpc( + RoutingSafetyViolation::UnsafePolicy, + "missing_vpc_vni", + right_vpc.id, + ) + })?; + if left_vni == right_vni { + return Err(RoutingSafetyFailure::for_pair( + RoutingSafetyViolation::UnsafePolicy, + "duplicate_vpc_vni", + left, + right, + )); + } + Ok(()) + } + + /// Proves that a VPC prefix belongs to a usable tenant-managed SitePrefix. + fn validate_vpc_prefix_site_association( + &self, + vpc: &Vpc, + address: &RoutedAddress, + ) -> Result<(), RoutingSafetyFailure> { + let AddressSource::Vpc { site_prefix_id } = address.source else { + return Err(RoutingSafetyFailure::for_address( + RoutingSafetyViolation::IneligibleOverlap, + "direct_prefix_has_no_site_prefix", + vpc.id, + address, + )); + }; + let root = site_prefix_id + .and_then(|id| self.site_prefixes.get(&id).copied()) + .ok_or_else(|| { + RoutingSafetyFailure::for_address( + RoutingSafetyViolation::IneligibleOverlap, + "missing_site_prefix", + vpc.id, + address, + ) + })?; + if root.status.authority != SitePrefixAuthority::TenantManaged + || root.config.routing_scope != SitePrefixRoutingScope::DatacenterOnly + || !matches!( + root.status.lifecycle_state, + SitePrefixLifecycleState::Ready | SitePrefixLifecycleState::Deleting + ) + || root + .config + .tenant_organization_id + .as_ref() + .map(|id| id.as_str()) + != Some(vpc.config.tenant_organization_id.as_str()) + || !prefix_contains(root.config.prefix, address.prefix) + { + return Err(RoutingSafetyFailure::for_address( + RoutingSafetyViolation::IneligibleOverlap, + "invalid_site_prefix_association", + vpc.id, + address, + )); + } + Ok(()) + } + + /// Resolves the effective VPC profile and applies the requested safety rule. + fn validate_profile( + &self, + config: &CarbideConfig, + vpc: &Vpc, + is_safe: fn(&FnnRoutingProfileConfig) -> bool, + ) -> Result<(), RoutingSafetyFailure> { + let profile = config + .fnn + .as_ref() + .ok_or_else(|| { + RoutingSafetyFailure::for_vpc( + RoutingSafetyViolation::UnsafePolicy, + "missing_fnn_config", + vpc.id, + ) + })? + .resolve_vpc_routing_profile(&vpc.config) + .map_err(|_| { + RoutingSafetyFailure::for_vpc( + RoutingSafetyViolation::UnsafePolicy, + "unresolved_vpc_routing_profile", + vpc.id, + ) + })?; + if is_safe(&profile) { + Ok(()) + } else { + Err(RoutingSafetyFailure::for_vpc( + RoutingSafetyViolation::UnsafePolicy, + "unsafe_vpc_routing_profile", + vpc.id, + )) + } + } + + /// Rejects site-wide features that can bridge otherwise isolated VPCs. + fn validate_site_policy(&self, config: &CarbideConfig) -> Result<(), RoutingSafetyFailure> { + let fnn = config.fnn.as_ref().ok_or_else(|| { + RoutingSafetyFailure::new(RoutingSafetyViolation::UnsafePolicy, "missing_fnn_config") + })?; + let reason = if config.vpc_isolation_behavior != VpcIsolationBehaviorType::MutualIsolation { + Some("site_isolation_not_mutual") + } else if config.site_global_vpc_vni.is_some() { + Some("site_global_vpc_vni_enabled") + } else if !config.anycast_site_prefixes.is_empty() { + Some("site_anycast_prefixes_enabled") + } else if config.vmaas_config.is_some() { + Some("vmaas_enabled") + } else if fnn.common_internal_route_target.is_some() { + Some("common_internal_route_target_enabled") + } else if !fnn.additional_route_target_imports.is_empty() { + Some("additional_route_target_imports_enabled") + } else if config + .network_security_group + .policy_overrides + .iter() + .any(|rule| rule.action == NetworkSecurityGroupRuleAction::Permit) + { + Some("site_nsg_permit_override_enabled") + } else { + None + }; + if let Some(reason) = reason { + return Err(RoutingSafetyFailure::new( + RoutingSafetyViolation::UnsafePolicy, + reason, + )); + } + Ok(()) + } + + /// Checks every graph-active tenant path, not only the VPCs sharing a CIDR. + /// + /// The renderer builds one site-wide isolation list, so an unrelated active + /// path with a permit or leak can invalidate the site's reuse guarantee. + /// Every active FNN VPC also needs a present, site-unique rendered VNI. + fn validate_active_paths(&self, config: &CarbideConfig) -> Result<(), RoutingSafetyFailure> { + let mut active_vpcs = self + .addresses + .iter() + .map(|address| address.vpc_id) + .collect::>(); + if let Some(policy) = config + .vpc_peering_policy_on_existing + .or(config.vpc_peering_policy) + { + for (vpc_id, peer_vpc_id) in &self.peerings { + let vpc = self.vpcs.get(vpc_id).ok_or_else(|| { + RoutingSafetyFailure::for_vpc( + RoutingSafetyViolation::UnsafePolicy, + "missing_peered_vpc", + *vpc_id, + ) + })?; + let peer_vpc = self.vpcs.get(peer_vpc_id).ok_or_else(|| { + RoutingSafetyFailure::for_vpc( + RoutingSafetyViolation::UnsafePolicy, + "missing_peered_vpc", + *peer_vpc_id, + ) + .with_vpc(*vpc_id) + })?; + let vpc_type = vpc.config.network_virtualization_type; + let peer_vpc_type = peer_vpc.config.network_virtualization_type; + if peering_direction_is_active(policy, vpc_type, peer_vpc_type) + || peering_direction_is_active(policy, peer_vpc_type, vpc_type) + { + active_vpcs.insert(*vpc_id); + active_vpcs.insert(*peer_vpc_id); + } + } + } + let mut referenced_groups = HashSet::new(); + for path in &self.instance_paths { + if path.has_unresolved_reference { + return Err(RoutingSafetyFailure::new( + RoutingSafetyViolation::UnsafePolicy, + "unresolved_instance_network_reference", + ) + .with_resource(format!("instance:{}", path.instance_id))); + } + if !path.vpc_ids.is_empty() && path.has_routing_override { + let mut failure = RoutingSafetyFailure::new( + RoutingSafetyViolation::UnsafePolicy, + "instance_routing_override_enabled", + ) + .with_resource(format!("instance:{}", path.instance_id)); + failure.vpc_ids.extend(path.vpc_ids.iter().copied()); + return Err(failure); + } + active_vpcs.extend(path.vpc_ids.iter().copied()); + if !path.vpc_ids.is_empty() + && let Some(group_id) = &path.network_security_group_id + { + referenced_groups.insert(group_id.clone()); + } + } + let mut active_vpcs = active_vpcs.into_iter().collect::>(); + active_vpcs.sort(); + let mut active_fnn_vnis = HashMap::new(); + for vpc_id in active_vpcs { + let vpc = self.vpcs.get(&vpc_id).ok_or_else(|| { + RoutingSafetyFailure::for_vpc( + RoutingSafetyViolation::UnsafePolicy, + "missing_active_vpc", + vpc_id, + ) + })?; + if vpc.config.network_virtualization_type == VpcVirtualizationType::Fnn { + let vni = vpc.status.vni.ok_or_else(|| { + RoutingSafetyFailure::for_vpc( + RoutingSafetyViolation::UnsafePolicy, + "missing_active_vpc_vni", + vpc_id, + ) + })?; + if let Some(other_vpc_id) = active_fnn_vnis.insert(vni, vpc_id) { + return Err(RoutingSafetyFailure::for_vpc( + RoutingSafetyViolation::UnsafePolicy, + "duplicate_active_vpc_vni", + vpc_id, + ) + .with_vpc(other_vpc_id)); + } + } + + let is_admin_control_path = self.is_unconsumed_admin_vpc(config, vpc); + if !is_admin_control_path + && vpc.config.network_virtualization_type == VpcVirtualizationType::Fnn + { + self.validate_profile(config, vpc, profile_preserves_isolation)?; + } + if !is_admin_control_path && let Some(group_id) = &vpc.config.network_security_group_id + { + referenced_groups.insert(group_id.clone()); + } + } + let mut referenced_groups = referenced_groups.into_iter().collect::>(); + referenced_groups.sort_by_cached_key(ToString::to_string); + for group_id in referenced_groups { + let group = self.network_security_groups.get(&group_id).ok_or_else(|| { + RoutingSafetyFailure::new( + RoutingSafetyViolation::UnsafePolicy, + "missing_active_network_security_group", + ) + .with_resource(format!("network-security-group:{group_id}")) + })?; + if !network_security_group_is_safe(group) { + return Err(RoutingSafetyFailure::new( + RoutingSafetyViolation::UnsafePolicy, + "unsafe_active_network_security_group", + ) + .with_resource(format!("network-security-group:{group_id}"))); + } + } + Ok(()) + } + + /// Returns every VPC visible to one receiver through direct attachments and peers. + fn visible_vpcs( + &self, + policy: Option, + directly_visible: &HashSet, + ) -> Result, RoutingSafetyFailure> { + let mut visible_vpcs = directly_visible.clone(); + let Some(policy) = policy else { + return Ok(visible_vpcs); + }; + + for receiver_id in directly_visible { + let receiver = self.vpcs.get(receiver_id).ok_or_else(|| { + RoutingSafetyFailure::for_vpc( + RoutingSafetyViolation::UnsafePolicy, + "missing_active_vpc", + *receiver_id, + ) + })?; + for (left, right) in &self.peerings { + let peer_id = if *receiver_id == *left { + Some(*right) + } else if *receiver_id == *right { + Some(*left) + } else { + None + }; + let Some(peer_id) = peer_id else { + continue; + }; + let peer = self.vpcs.get(&peer_id).ok_or_else(|| { + RoutingSafetyFailure::for_vpc( + RoutingSafetyViolation::UnsafePolicy, + "missing_peered_vpc", + peer_id, + ) + .with_vpc(*receiver_id) + })?; + if peering_direction_is_active( + policy, + receiver.config.network_virtualization_type, + peer.config.network_virtualization_type, + ) { + visible_vpcs.insert(peer_id); + } + } + } + Ok(visible_vpcs) + } + + /// Builds the standard failure when one receiver sees both sides of a reuse pair. + fn visible_overlap_failure( + &self, + analysis: &OverlapAnalysis, + visible_vpcs: &HashSet, + ) -> Option { + analysis + .tenant_reuse_indices + .iter() + .find_map(|(left_index, right_index)| { + let left = &self.addresses[*left_index]; + let right = &self.addresses[*right_index]; + (visible_vpcs.contains(&left.vpc_id) && visible_vpcs.contains(&right.vpc_id)).then( + || { + RoutingSafetyFailure::for_pair( + RoutingSafetyViolation::ReachableOverlap, + "receiver_sees_overlapping_address_space", + left, + right, + ) + }, + ) + }) + } + + /// Rejects a VPC or retained instance receiver that can see both reused prefixes. + /// + /// VPC receivers cover direct and sibling peer imports. Instance receivers + /// additionally union every attached VPC and each attachment's direct peers. + fn validate_reachability( + &self, + config: &CarbideConfig, + analysis: &OverlapAnalysis, + ) -> Result<(), RoutingSafetyFailure> { + let policy = config + .vpc_peering_policy_on_existing + .or(config.vpc_peering_policy); + if policy.is_some() { + let mut receivers = self.vpcs.values().copied().collect::>(); + receivers.sort_by_key(|vpc| vpc.id); + for receiver in receivers { + let directly_visible = HashSet::from([receiver.id]); + let visible_vpcs = self.visible_vpcs(policy, &directly_visible)?; + if let Some(failure) = self.visible_overlap_failure(analysis, &visible_vpcs) { + return Err(failure.with_vpc(receiver.id)); + } + } + } + + // An instance receives the union of every attached VPC interface. Each + // interface can also import its VPC's direct peers, so no individual + // VPC receiver need see both reused prefixes for the host to see both. + for path in &self.instance_paths { + let visible_vpcs = self.visible_vpcs(policy, &path.vpc_ids)?; + if let Some(failure) = self.visible_overlap_failure(analysis, &visible_vpcs) { + return Err(failure.with_resource(format!("instance:{}", path.instance_id))); + } + } + Ok(()) + } +} + +/// Merges one retained network config into an instance's routing dependencies. +/// +/// An existing unbound segment is acceptable when the interface supplies its +/// logical VPC explicitly. Missing rows remain unresolved so deletion and +/// update races cannot silently remove a policy constraint. +fn collect_instance_path( + config: &InstanceNetworkConfig, + segment_vpcs: &HashMap>, + vpc_prefix_vpcs: &HashMap, + vpc_ids: &mut HashSet, + has_routing_override: &mut bool, + has_unresolved_reference: &mut bool, +) { + if let Some(auto) = config.auto_config { + vpc_ids.insert(auto.vpc_id); + } + for interface in &config.interfaces { + let logical_vpc_id = interface + .vpc_id + .or_else(|| interface.vpc_selection.map(|selection| selection.vpc_id)) + .or(config.auto_config.map(|auto| auto.vpc_id)); + if let Some(vpc_id) = logical_vpc_id { + vpc_ids.insert(vpc_id); + } + if let Some(segment_id) = interface.network_segment_id { + collect_segment_vpc( + segment_id, + logical_vpc_id, + segment_vpcs, + vpc_ids, + has_unresolved_reference, + ); + } + match interface.network_details { + Some(NetworkDetails::NetworkSegment(segment_id)) => { + collect_segment_vpc( + segment_id, + logical_vpc_id, + segment_vpcs, + vpc_ids, + has_unresolved_reference, + ); + } + Some(NetworkDetails::VpcPrefixId(vpc_prefix_id)) => { + match vpc_prefix_vpcs.get(&vpc_prefix_id) { + Some(vpc_id) => { + vpc_ids.insert(*vpc_id); + } + None => *has_unresolved_reference = true, + } + } + None => {} + } + if let Some(ipv6) = &interface.ipv6_interface_config { + match vpc_prefix_vpcs.get(&ipv6.vpc_prefix_id) { + Some(vpc_id) => { + vpc_ids.insert(*vpc_id); + } + None => *has_unresolved_reference = true, + } + } + if let Some(InstanceInterfaceRoutingProfile { + allowed_anycast_prefixes, + }) = interface.routing_profile.as_ref() + { + *has_routing_override |= !allowed_anycast_prefixes.is_empty(); + } + } +} + +/// Resolves one retained segment reference without treating a valid unbound +/// HostInband segment as missing when the interface records its logical VPC. +fn collect_segment_vpc( + segment_id: NetworkSegmentId, + logical_vpc_id: Option, + segment_vpcs: &HashMap>, + vpc_ids: &mut HashSet, + has_unresolved_reference: &mut bool, +) { + match segment_vpcs.get(&segment_id) { + Some(Some(vpc_id)) => { + vpc_ids.insert(*vpc_id); + } + Some(None) if logical_vpc_id.is_some() => {} + Some(None) | None => *has_unresolved_reference = true, + } +} + +/// Returns whether the renderer imports anything from `peer_type` into `receiver_type`. +/// +/// A configured `None` policy disables prefix imports but retains capability-driven +/// VNI imports. The caller handles the distinct case where both policy options are unset. +fn peering_direction_is_active( + policy: VpcPeeringPolicy, + receiver_type: VpcVirtualizationType, + peer_type: VpcVirtualizationType, +) -> bool { + let imports_peer_prefixes = match policy { + VpcPeeringPolicy::Exclusive => receiver_type.capabilities().peers_with.contains(&peer_type), + VpcPeeringPolicy::Mixed => true, + VpcPeeringPolicy::None => false, + }; + let imports_peer_vni = + receiver_type.imports_peer_vnis_into_overlay() && peer_type.vni_advertised_to_peers(); + imports_peer_prefixes || imports_peer_vni +} + +/// Returns whether an effective FNN profile may participate in tenant overlap. +fn profile_allows_overlap(profile: &FnnRoutingProfileConfig) -> bool { + profile.overlap_eligible && profile_preserves_isolation(profile) +} + +/// Returns whether an effective FNN profile keeps its VPC routing domain private. +/// +/// This is an explicit allowlist. Every new profile field that can introduce +/// reachability must be considered here before it is safe for reuse. +fn profile_preserves_isolation(profile: &FnnRoutingProfileConfig) -> bool { + let FnnRoutingProfileConfig { + // This flag controls overlap admission and does not alter rendered routes. + overlap_eligible: _, + route_target_imports, + route_targets_on_exports, + internal, + leak_default_route_from_underlay, + leak_tenant_host_routes_to_underlay, + tenant_leak_communities_accepted, + accepted_leaks_from_underlay, + allowed_anycast_prefixes, + // Access tier controls who can select a profile, not the routes it renders. + access_tier: _, + } = profile; + + *internal == Some(true) + && route_target_imports.as_ref().is_none_or(Vec::is_empty) + && route_targets_on_exports.as_ref().is_none_or(Vec::is_empty) + && !leak_default_route_from_underlay.unwrap_or_default() + && !leak_tenant_host_routes_to_underlay.unwrap_or_default() + && !tenant_leak_communities_accepted.unwrap_or_default() + && accepted_leaks_from_underlay + .as_ref() + .is_none_or(Vec::is_empty) + && allowed_anycast_prefixes.as_ref().is_none_or(Vec::is_empty) +} + +/// Returns whether an NSG can only remove, rather than introduce, reachability. +fn network_security_group_is_safe(group: &NetworkSecurityGroup) -> bool { + !group.stateful_egress + && group + .rules + .iter() + .all(|rule| rule.action == NetworkSecurityGroupRuleAction::Deny) +} + +fn prefixes_overlap(left: IpNetwork, right: IpNetwork) -> bool { + left.is_ipv4() == right.is_ipv4() + && (left.contains(right.network()) || right.contains(left.network())) +} + +/// Classifies the only overlap case that may proceed to eligibility checks. +/// +/// This proves neither tenant ownership nor isolation; those checks deliberately +/// remain in `validate_overlap_eligibility` and the policy phase. +fn is_tenant_reuse_pair(left: &RoutedAddress, right: &RoutedAddress) -> bool { + left.vpc_id != right.vpc_id + && left.prefix == right.prefix + && matches!(left.source, AddressSource::Vpc { .. }) + && matches!(right.source, AddressSource::Vpc { .. }) +} + +fn prefix_contains(parent: IpNetwork, child: IpNetwork) -> bool { + parent.is_ipv4() == child.is_ipv4() + && parent.prefix() <= child.prefix() + && parent.contains(child.network()) +} + +/// Stable violation categories used to classify privacy-safe client errors. +#[derive(Clone, Copy, Debug, Eq, PartialEq, thiserror::Error)] +enum RoutingSafetyViolation { + /// A candidate introduces overlap while the site opt-in is disabled. + #[error("{OVERLAPPING_ADDRESS_SPACE}")] + OverlapDisabled, + /// A candidate introduces address occupancy that is not tenant reuse. + #[error("{OVERLAPPING_ADDRESS_SPACE}")] + AddressConflict, + /// An exact reuse pair fails its SitePrefix association or isolation prerequisites. + #[error("{INELIGIBLE_OVERLAP}")] + IneligibleOverlap, + /// A route receiver would import both copies of reused address space. + #[error("{REACHABLE_OVERLAP}")] + ReachableOverlap, + /// A VNI, site, profile, NSG, or retained-instance condition cannot prove isolation. + #[error("{UNSAFE_POLICY}")] + UnsafePolicy, +} + +impl RoutingSafetyViolation { + fn code(self) -> &'static str { + match self { + Self::OverlapDisabled => "overlap_disabled", + Self::AddressConflict => "address_conflict", + Self::IneligibleOverlap => "ineligible_overlap", + Self::ReachableOverlap => "reachable_overlap", + Self::UnsafePolicy => "unsafe_policy", + } + } +} + +/// Internal diagnostic for one rejected admission decision. +/// +/// `reason` and resource identifiers are logged for operators but are never +/// copied into the client response, where foreign tenant details would leak. +#[derive(Clone, Debug, Eq, PartialEq)] +struct RoutingSafetyFailure { + violation: RoutingSafetyViolation, + reason: &'static str, + vpc_ids: Vec, + resource_ids: Vec, +} + +impl RoutingSafetyFailure { + fn new(violation: RoutingSafetyViolation, reason: &'static str) -> Self { + Self { + violation, + reason, + vpc_ids: Vec::new(), + resource_ids: Vec::new(), + } + } + + fn for_vpc(violation: RoutingSafetyViolation, reason: &'static str, vpc_id: VpcId) -> Self { + Self { + vpc_ids: vec![vpc_id], + ..Self::new(violation, reason) + } + } + + fn for_address( + violation: RoutingSafetyViolation, + reason: &'static str, + vpc_id: VpcId, + address: &RoutedAddress, + ) -> Self { + Self { + vpc_ids: vec![vpc_id], + resource_ids: vec![address.key.to_string()], + ..Self::new(violation, reason) + } + } + + fn for_pair( + violation: RoutingSafetyViolation, + reason: &'static str, + left: &RoutedAddress, + right: &RoutedAddress, + ) -> Self { + Self { + vpc_ids: vec![left.vpc_id, right.vpc_id], + resource_ids: vec![left.key.to_string(), right.key.to_string()], + ..Self::new(violation, reason) + } + } + + fn with_vpc(mut self, vpc_id: VpcId) -> Self { + self.vpc_ids.push(vpc_id); + self + } + + fn with_resource(mut self, resource_id: String) -> Self { + self.resource_ids.push(resource_id); + self + } + + /// Logs the private diagnostic context at the boundary that rejects a write. + fn log(&self, operation: &'static str) { + let mut vpc_ids = self.vpc_ids.clone(); + vpc_ids.sort(); + vpc_ids.dedup(); + let mut resource_ids = self.resource_ids.clone(); + resource_ids.sort(); + resource_ids.dedup(); + tracing::warn!( + operation, + violation = self.violation.code(), + reason = self.reason, + vpc_ids = ?vpc_ids, + resource_ids = ?resource_ids, + "routing safety validation rejected state" + ); + } +} + +// Keep foreign resource IDs and internal reasons in `RoutingSafetyFailure`. +// Tenant-facing conversion intentionally collapses them into stable messages. +impl From for CarbideError { + fn from(error: RoutingSafetyViolation) -> Self { + match error { + RoutingSafetyViolation::OverlapDisabled | RoutingSafetyViolation::AddressConflict => { + CarbideError::InvalidArgument(OVERLAPPING_ADDRESS_SPACE.to_string()) + } + RoutingSafetyViolation::IneligibleOverlap + | RoutingSafetyViolation::ReachableOverlap + | RoutingSafetyViolation::UnsafePolicy => { + CarbideError::FailedPrecondition(error.to_string()) + } + } + } +} + +impl From for CarbideError { + fn from(error: RoutingSafetyFailure) -> Self { + error.violation.into() + } +} + +/// `lock_site_mutation` acquires the routing-safety lock before any +/// resource-specific row lock. +/// +/// The lock intentionally waits instead of treating ordinary writer +/// contention as an admission failure. The wait follows the caller's request +/// or controller-task lifecycle, and PostgreSQL releases the lock when the +/// owning transaction commits or rolls back. +pub(crate) async fn lock_site_mutation(txn: &mut PgConnection) -> CarbideResult<()> { + db::routing_safety::lock_site_mutation(txn) + .await + .map_err(Into::into) +} + +/// `validate_live_state` validates all transaction-visible persisted routing state. +/// +/// Callers use it after a transactional policy or graph mutation and before +/// committing it, while they still hold the site mutation lock. +pub(crate) async fn validate_live_state( + config: &CarbideConfig, + txn: &mut PgConnection, +) -> CarbideResult<()> { + validate_persisted_state(config, txn, "routing_mutation").await +} + +/// Runs shared startup and post-mutation validation over transaction-visible rows. +/// +/// The address phase runs first. Policy rows are fetched only when an exact +/// VPC-prefix reuse pair exists. +async fn validate_persisted_state( + config: &CarbideConfig, + txn: &mut PgConnection, + operation: &'static str, +) -> CarbideResult<()> { + let addresses = db::routing_safety::load_addresses(txn).await?; + let mut state = RoutingState::from_addresses(&addresses); + let analysis = state.analyze_overlaps(); + state + .validate_tenant_reuse(config, &analysis) + .map_err(|failure| { + failure.log(operation); + CarbideError::from(failure) + })?; + if analysis.tenant_reuse_indices.is_empty() { + return Ok(()); + } + + let policy = db::routing_safety::load_policy_paths(txn).await?; + state.apply_policy_paths(&policy, &addresses); + state + .validate_policy_paths(config, &analysis) + .map_err(|failure| { + failure.log(operation); + failure.into() + }) +} + +/// `validate_startup` fails before controllers or listeners can serve a graph +/// that already violates overlap isolation. A latent unsafe profile is +/// harmless until an overlapping route is present, so the evaluator +/// deliberately short-circuits when the live graph has no exact cross-VPC +/// VPC-prefix pair. +pub(crate) async fn validate_startup( + config: &CarbideConfig, + pool: &sqlx::PgPool, +) -> CarbideResult<()> { + let mut txn = db::Transaction::begin(pool).await?; + lock_site_mutation(&mut txn).await?; + validate_persisted_state(config, &mut txn, "startup_preflight").await?; + txn.commit().await?; + Ok(()) +} + +/// `validate_vpc_prefix_candidate` checks an insertion before the legacy +/// physical-overlap constraint runs. Adopted network prefixes are removed from +/// the candidate view because the same transaction will attach them to the new +/// parent. +pub(crate) async fn validate_vpc_prefix_candidate( + config: &CarbideConfig, + txn: &mut PgConnection, + candidate: &NewVpcPrefix, + adopted_network_prefix_ids: &[NetworkPrefixId], +) -> CarbideResult<()> { + let addresses = db::routing_safety::load_addresses(txn).await?; + let current = RoutingState::from_addresses(&addresses); + let current_overlap_pairs = current.analyze_overlaps().occupancy_pairs; + let mut proposed = RoutingState::from_addresses(&addresses); + proposed.addresses.retain(|address| { + !matches!( + address.source, + AddressSource::Network { id: Some(id) } + if adopted_network_prefix_ids.contains(&id) + ) + }); + proposed.addresses.push(RoutedAddress { + key: AddressKey::CandidateVpc(candidate.id), + vpc_id: candidate.vpc_id, + prefix: candidate.config.prefix, + source: AddressSource::Vpc { + site_prefix_id: candidate.site_prefix_id, + }, + }); + validate_candidate( + config, + txn, + &addresses, + current_overlap_pairs, + &mut proposed, + "vpc_prefix_candidate", + ) + .await +} + +/// `validate_network_segment_candidate` checks a segment before its legacy +/// physical-overlap constraint runs. +pub(crate) async fn validate_network_segment_candidate( + config: &CarbideConfig, + txn: &mut PgConnection, + candidate: &NewNetworkSegment, +) -> CarbideResult<()> { + let Some(vpc_id) = candidate.vpc_id else { + return Ok(()); + }; + let prefixes = candidate + .prefixes + .iter() + .map(|prefix| prefix.prefix) + .collect::>(); + validate_network_prefix_candidates(config, txn, vpc_id, &prefixes).await +} + +/// Checks the address-space effect of binding an existing segment to a VPC. +/// +/// An unbound segment is absent from the routed snapshot, while replacement +/// changes the routing classification of every existing pair that contains its +/// prefixes. The simulated snapshot therefore exposes the target VPC to both +/// address and retained-instance resolution before classifying the candidate +/// relationships. +pub(crate) async fn validate_network_segment_attachment( + config: &CarbideConfig, + txn: &mut PgConnection, + candidate: &NetworkSegment, + vpc_id: VpcId, +) -> CarbideResult<()> { + let direct_prefixes = candidate + .prefixes + .iter() + .filter(|prefix| prefix.vpc_prefix_id.is_none()) + .collect::>(); + let moved_keys = direct_prefixes + .iter() + .map(|prefix| AddressKey::Network(prefix.id)) + .collect::>(); + let mut addresses = db::routing_safety::load_addresses(txn).await?; + let current = RoutingState::from_addresses(&addresses); + let mut current_overlap_pairs = current.analyze_overlaps().occupancy_pairs; + current_overlap_pairs + .retain(|(left, right)| !moved_keys.contains(left) && !moved_keys.contains(right)); + + let simulated_segment = addresses + .network_segments + .iter_mut() + .find(|segment| segment.id == candidate.id) + .ok_or_else(|| { + CarbideError::internal(format!( + "network segment {} is missing from the routing snapshot", + candidate.id + )) + })?; + simulated_segment.config.vpc_id = Some(vpc_id); + let mut proposed = RoutingState::from_addresses(&addresses); + validate_candidate( + config, + txn, + &addresses, + current_overlap_pairs, + &mut proposed, + "network_segment_attachment", + ) + .await +} + +/// Adds direct prefixes to a candidate view without mutating persisted rows. +async fn validate_network_prefix_candidates( + config: &CarbideConfig, + txn: &mut PgConnection, + vpc_id: VpcId, + prefixes: &[IpNetwork], +) -> CarbideResult<()> { + let addresses = db::routing_safety::load_addresses(txn).await?; + let current = RoutingState::from_addresses(&addresses); + let current_overlap_pairs = current.analyze_overlaps().occupancy_pairs; + let mut proposed = RoutingState::from_addresses(&addresses); + proposed.addresses.extend( + prefixes + .iter() + .enumerate() + .map(|(index, prefix)| RoutedAddress { + key: AddressKey::CandidateNetwork(index), + vpc_id, + prefix: *prefix, + source: AddressSource::Network { id: None }, + }), + ); + validate_candidate( + config, + txn, + &addresses, + current_overlap_pairs, + &mut proposed, + "network_prefix_candidate", + ) + .await +} + +/// Rejects only newly introduced occupancy, then loads policy rows if needed. +async fn validate_candidate( + config: &CarbideConfig, + txn: &mut PgConnection, + addresses: &db::routing_safety::RoutingAddressSnapshot, + current_overlap_pairs: HashSet<(AddressKey, AddressKey)>, + proposed: &mut RoutingState<'_>, + operation: &'static str, +) -> CarbideResult<()> { + let analysis = proposed.analyze_overlaps(); + validate_candidate_addresses(config, ¤t_overlap_pairs, proposed, &analysis).map_err( + |failure| { + failure.log(operation); + CarbideError::from(failure) + }, + )?; + if analysis.tenant_reuse_indices.is_empty() { + return Ok(()); + } + let policy = db::routing_safety::load_policy_paths(txn).await?; + proposed.apply_policy_paths(&policy, addresses); + proposed + .validate_policy_paths(config, &analysis) + .map_err(|failure| { + failure.log(operation); + failure.into() + }) +} + +/// Compares current and proposed pair identities to distinguish drain from expansion. +/// +/// Existing occupancy may contract, but every new pair is classified. With +/// the site gate disabled this freezes expansion without invalidating retained, +/// safely isolated tenant reuse. +fn validate_candidate_addresses( + config: &CarbideConfig, + current_overlap_pairs: &HashSet<(AddressKey, AddressKey)>, + proposed: &RoutingState<'_>, + analysis: &OverlapAnalysis, +) -> Result<(), RoutingSafetyFailure> { + if !config.tenant_prefix_overlap_enabled + && !analysis.occupancy_pairs.is_subset(current_overlap_pairs) + { + let mut failure = RoutingSafetyFailure::new( + RoutingSafetyViolation::OverlapDisabled, + "site_overlap_gate_disabled", + ); + if let Some((left, right)) = analysis + .occupancy_pairs + .difference(current_overlap_pairs) + .min() + { + failure + .resource_ids + .extend([left.to_string(), right.to_string()]); + } + return Err(failure); + } + + for (left_index, right_index) in &analysis.all_indices { + let left = &proposed.addresses[*left_index]; + let right = &proposed.addresses[*right_index]; + let pair = if left.key < right.key { + (left.key, right.key) + } else { + (right.key, left.key) + }; + if current_overlap_pairs.contains(&pair) { + continue; + } + // Tenant-reuse pairs are deferred to the full eligibility and policy + // validation below rather than exempted from candidate admission. + if is_tenant_reuse_pair(left, right) { + continue; + } + let reason = if left.vpc_id == right.vpc_id { + "same_vpc_address_conflict" + } else { + "address_conflict_not_tenant_reuse" + }; + return Err(RoutingSafetyFailure::for_pair( + RoutingSafetyViolation::AddressConflict, + reason, + left, + right, + )); + } + + proposed.validate_tenant_reuse(config, analysis) +} + +#[cfg(test)] +mod tests { + use std::collections::HashMap; + use std::str::FromStr; + use std::time::Duration; + + use carbide_uuid::instance::InstanceId; + use carbide_uuid::machine::{MachineId, MachineIdSource, MachineType}; + use carbide_uuid::network::NetworkSegmentId; + use carbide_uuid::network_security_group::NetworkSecurityGroupId; + use chrono::Utc; + use config_version::{ConfigVersion, Versioned}; + use model::instance::config::network::{ + InstanceNetworkAutoConfig, InstanceNetworkConfig, InstanceNetworkConfigUpdate, + }; + use model::metadata::Metadata; + use model::network_prefix::NewNetworkPrefix; + use model::network_security_group::{ + NetworkSecurityGroupRule, NetworkSecurityGroupRuleDirection, NetworkSecurityGroupRuleNet, + NetworkSecurityGroupRuleProtocol, + }; + use model::network_segment::{ + AllocationStrategy, NetworkSegment, NetworkSegmentConfig, NetworkSegmentControllerState, + NetworkSegmentStatus, + }; + use model::site_prefix::{SitePrefixConfig, SitePrefixStatus}; + use model::tenant::TenantOrganizationId; + use model::vpc::{ + ALL_VPC_VIRTUALIZATION_TYPES, PrefixFilterPolicyEntry, RouteTargetConfig, VpcConfig, + VpcStatus, + }; + use model::vpc_prefix::{ + VpcPrefix, VpcPrefixConfig, VpcPrefixControllerState, VpcPrefixStatus, + }; + use sqlx::PgPool; + + use super::*; + + const ISOLATED_PROFILE: &str = "ISOLATED"; + const PROFILE: &str = "OVERLAP"; + const UNSAFE_PROFILE: &str = "UNSAFE"; + + fn config() -> CarbideConfig { + let mut config = crate::test_support::default_config::get(); + config.tenant_prefix_overlap_enabled = true; + config.vmaas_config = None; + config.fnn = Some(crate::cfg::file::FnnConfig { + admin_vpc: None, + common_internal_route_target: None, + additional_route_target_imports: vec![], + routing_profiles: HashMap::from([ + ( + ISOLATED_PROFILE.to_string(), + FnnRoutingProfileConfig { + overlap_eligible: false, + internal: Some(true), + ..FnnRoutingProfileConfig::default() + }, + ), + ( + PROFILE.to_string(), + FnnRoutingProfileConfig { + overlap_eligible: true, + internal: Some(true), + ..FnnRoutingProfileConfig::default() + }, + ), + ( + UNSAFE_PROFILE.to_string(), + FnnRoutingProfileConfig { + overlap_eligible: true, + internal: Some(false), + ..FnnRoutingProfileConfig::default() + }, + ), + ]), + use_vpc_vrf_loopback: false, + }); + config + } + + fn vpc(tenant: &str, profile: &str, vni: i32) -> Vpc { + let now = Utc::now(); + Vpc { + id: VpcId::new(), + version: ConfigVersion::initial(), + config: VpcConfig { + tenant_organization_id: tenant.to_string(), + tenant_keyset_id: None, + network_virtualization_type: VpcVirtualizationType::Fnn, + network_security_group_id: None, + default_nvlink_logical_partition_id: None, + vni: Some(vni), + routing_profile_type: Some(profile.to_string()), + routing_profile_overrides: None, + power_resource_group: None, + }, + status: VpcStatus { vni: Some(vni) }, + metadata: Metadata::default(), + created: now, + updated: now, + deleted: None, + } + } + + fn site_prefix(tenant: &str, prefix: &str) -> SitePrefix { + let now = Utc::now(); + SitePrefix { + id: SitePrefixId::new(), + config: SitePrefixConfig { + prefix: prefix.parse().unwrap(), + tenant_organization_id: Some(TenantOrganizationId::from_str(tenant).unwrap()), + routing_scope: SitePrefixRoutingScope::DatacenterOnly, + }, + metadata: Metadata::default(), + status: SitePrefixStatus { + authority: SitePrefixAuthority::TenantManaged, + lifecycle_state: SitePrefixLifecycleState::Ready, + }, + version: ConfigVersion::initial(), + created_at: now, + updated_at: now, + } + } + + fn address(key: usize, vpc: &Vpc, root: &SitePrefix, prefix: &str) -> RoutedAddress { + RoutedAddress { + key: AddressKey::Vpc(VpcPrefixId::from(uuid::Uuid::from_u128(key as u128))), + vpc_id: vpc.id, + prefix: prefix.parse().unwrap(), + source: AddressSource::Vpc { + site_prefix_id: Some(root.id), + }, + } + } + + #[test] + fn address_keys_preserve_namespace_order_and_rendering() { + let network_prefix_id = NetworkPrefixId::new(); + let vpc_prefix_id = VpcPrefixId::new(); + let keys = [ + AddressKey::CandidateNetwork(7), + AddressKey::CandidateVpc(vpc_prefix_id), + AddressKey::Network(network_prefix_id), + AddressKey::Vpc(vpc_prefix_id), + ]; + + assert!(keys.is_sorted()); + assert_eq!( + keys.map(|key| key.to_string()), + [ + "candidate-network-prefix:7".to_string(), + format!("candidate-vpc-prefix:{vpc_prefix_id}"), + format!("network-prefix:{network_prefix_id}"), + format!("vpc-prefix:{vpc_prefix_id}"), + ] + ); + } + + fn vpc_prefix(vpc: &Vpc, root: &SitePrefix, prefix: &str) -> VpcPrefix { + VpcPrefix { + id: VpcPrefixId::new(), + site_prefix_id: Some(root.id), + vpc_id: vpc.id, + config: VpcPrefixConfig { + prefix: prefix.parse().unwrap(), + }, + metadata: Metadata::default(), + status: VpcPrefixStatus { + controller_state: Versioned::new( + VpcPrefixControllerState::Ready, + ConfigVersion::initial(), + ), + controller_state_outcome: None, + last_used_prefix: None, + total_31_segments: 0, + available_31_segments: 0, + total_linknet_segments: 0, + available_linknet_segments: 0, + }, + deleted: None, + } + } + + fn unbound_host_inband_segment(id: NetworkSegmentId) -> NetworkSegment { + let now = Utc::now(); + NetworkSegment { + id, + version: ConfigVersion::initial(), + config: NetworkSegmentConfig { + name: "unbound-host-inband".to_string(), + subdomain_id: None, + mtu: 1500, + segment_type: NetworkSegmentType::HostInband, + allocation_strategy: AllocationStrategy::Dynamic, + infer_slaac_eui64_addresses: false, + vpc_id: None, + }, + status: NetworkSegmentStatus { + controller_state: Versioned::new( + NetworkSegmentControllerState::Ready, + ConfigVersion::initial(), + ), + controller_state_outcome: None, + history: Vec::new(), + vlan_id: None, + vni: None, + can_stretch: None, + }, + prefixes: Vec::new(), + created: now, + updated: now, + deleted: None, + } + } + + fn bound_segment(vpc_id: VpcId, segment_type: NetworkSegmentType) -> NetworkSegment { + let mut segment = unbound_host_inband_segment(NetworkSegmentId::new()); + segment.config.name = "bound-routing-safety-test".to_string(); + segment.config.segment_type = segment_type; + segment.config.vpc_id = Some(vpc_id); + segment + } + + fn state<'a>( + vpcs: &'a [Vpc], + roots: &'a [SitePrefix], + addresses: Vec, + ) -> RoutingState<'a> { + RoutingState { + vpcs: vpcs.iter().map(|vpc| (vpc.id, vpc)).collect(), + site_prefixes: roots.iter().map(|root| (root.id, root)).collect(), + addresses, + admin_only_vpcs: HashSet::new(), + peerings: vec![], + network_security_groups: HashMap::new(), + instance_paths: vec![], + } + } + + fn overlapping_state<'a>(vpcs: &'a [Vpc], roots: &'a [SitePrefix]) -> RoutingState<'a> { + state( + vpcs, + roots, + vec![ + address(0, &vpcs[0], &roots[0], "10.0.0.0/24"), + address(1, &vpcs[1], &roots[1], "10.0.0.0/24"), + ], + ) + } + + fn instance_path(vpc_ids: impl IntoIterator) -> InstancePath { + InstancePath { + instance_id: InstanceId::new(), + vpc_ids: vpc_ids.into_iter().collect(), + network_security_group_id: None, + has_routing_override: false, + has_unresolved_reference: false, + } + } + + fn security_group_rule(action: NetworkSecurityGroupRuleAction) -> NetworkSecurityGroupRule { + NetworkSecurityGroupRule { + id: None, + src_net: NetworkSecurityGroupRuleNet::Prefix("0.0.0.0/0".parse().unwrap()), + dst_net: NetworkSecurityGroupRuleNet::Prefix("0.0.0.0/0".parse().unwrap()), + direction: NetworkSecurityGroupRuleDirection::Ingress, + ipv6: false, + src_port_start: None, + src_port_end: None, + dst_port_start: None, + dst_port_end: None, + protocol: NetworkSecurityGroupRuleProtocol::Any, + action, + priority: 1, + } + } + + fn network_security_group( + id: NetworkSecurityGroupId, + tenant: &str, + stateful_egress: bool, + rules: Vec, + ) -> NetworkSecurityGroup { + NetworkSecurityGroup { + id, + tenant_organization_id: TenantOrganizationId::from_str(tenant).unwrap(), + stateful_egress, + rules, + version: ConfigVersion::initial(), + created: Utc::now(), + deleted: None, + metadata: Metadata::default(), + created_by: None, + updated_by: None, + } + } + + fn violation(result: Result<(), RoutingSafetyFailure>) -> Result<(), RoutingSafetyViolation> { + result.map_err(|failure| failure.violation) + } + + fn new_vpc_prefix( + id: VpcPrefixId, + vpc_id: VpcId, + site_prefix_id: SitePrefixId, + prefix: IpNetwork, + ) -> NewVpcPrefix { + NewVpcPrefix { + id, + site_prefix_id: Some(site_prefix_id), + vpc_id, + config: VpcPrefixConfig { prefix }, + metadata: Metadata::default(), + } + } + + #[crate::sqlx_test] + async fn no_overlap_validation_skips_policy_projection( + pool: PgPool, + ) -> Result<(), Box> { + let machine_id = MachineId::new( + MachineIdSource::ProductBoardChassisSerial, + [0x3a; 32], + MachineType::Host, + ); + let mut setup = pool.begin().await?; + sqlx::query( + "INSERT INTO tenants (organization_id, organization_name, version) \ + VALUES ('tenant-a', 'Tenant A', $1)", + ) + .bind(ConfigVersion::initial()) + .execute(setup.as_mut()) + .await?; + sqlx::query("INSERT INTO machines (id, dpf) VALUES ($1, '{}'::jsonb)") + .bind(machine_id) + .execute(setup.as_mut()) + .await?; + sqlx::query( + "INSERT INTO instances \ + (machine_id, tenant_org, os_ipxe_script, network_config, nvlink_config) \ + VALUES ($1, 'tenant-a', '#!ipxe invalid-routing-projection', \ + '{\"invalid\": true}'::jsonb, '{\"gpu_configs\": []}'::jsonb)", + ) + .bind(machine_id) + .execute(setup.as_mut()) + .await?; + setup.commit().await?; + + let mut validation = pool.begin().await?; + lock_site_mutation(validation.as_mut()).await?; + validate_live_state(&config(), validation.as_mut()).await?; + validation.commit().await?; + + let mut proof = pool.begin().await?; + assert!( + db::routing_safety::load_policy_paths(proof.as_mut()) + .await + .is_err(), + "the invalid instance row proves the no-overlap path did not hydrate policy rows" + ); + proof.rollback().await?; + Ok(()) + } + + #[crate::sqlx_test] + async fn startup_accepts_legacy_direct_prefix_containment( + pool: PgPool, + ) -> Result<(), Box> { + let same_vpc_id = VpcId::new(); + let tenant_vpc_id = VpcId::new(); + let admin_vpc_id = VpcId::new(); + let version = ConfigVersion::initial(); + let mut setup = pool.begin().await?; + + for (vpc_id, name, organization_id) in [ + (same_vpc_id, "same-vpc-legacy", "tenant-a"), + (tenant_vpc_id, "tenant-prefix-legacy", "tenant-b"), + (admin_vpc_id, "admin-segment-legacy", "carbide_internal"), + ] { + sqlx::query( + "INSERT INTO vpcs (id, name, organization_id, version) VALUES ($1, $2, $3, $4)", + ) + .bind(vpc_id) + .bind(name) + .bind(organization_id) + .bind(version) + .execute(setup.as_mut()) + .await?; + } + + for (id, vpc_id, name, prefix) in [ + ( + VpcPrefixId::new(), + same_vpc_id, + "same-vpc-parent", + "10.70.0.0/24".parse::()?, + ), + ( + VpcPrefixId::new(), + tenant_vpc_id, + "admin-contained-prefix", + "192.0.2.128/25".parse::()?, + ), + ] { + sqlx::query( + "INSERT INTO network_vpc_prefixes (id, prefix, name, vpc_id) \ + VALUES ($1, $2, $3, $4)", + ) + .bind(id) + .bind(prefix) + .bind(name) + .bind(vpc_id) + .execute(setup.as_mut()) + .await?; + } + + for (name, vpc_id, segment_type, prefix) in [ + ( + "same-vpc-direct-child", + same_vpc_id, + NetworkSegmentType::Tenant, + "10.70.0.0/25".parse::()?, + ), + ( + "bound-admin-parent", + admin_vpc_id, + NetworkSegmentType::Admin, + "192.0.2.0/24".parse::()?, + ), + ] { + db::network_segment::persist( + NewNetworkSegment { + id: NetworkSegmentId::new(), + name: name.to_string(), + subdomain_id: None, + vpc_id: Some(vpc_id), + mtu: 1500, + prefixes: vec![NewNetworkPrefix { + prefix, + gateway: None, + dhcpv6_link_address: None, + num_reserved: 0, + }], + vlan_id: None, + vni: None, + segment_type, + can_stretch: None, + allocation_strategy: AllocationStrategy::Dynamic, + infer_slaac_eui64_addresses: false, + }, + setup.as_mut(), + NetworkSegmentControllerState::Ready, + ) + .await?; + } + setup.commit().await?; + + validate_startup(&config(), &pool).await?; + Ok(()) + } + + #[crate::sqlx_test] + async fn concurrent_candidates_reread_after_the_site_lock( + pool: PgPool, + ) -> Result<(), Box> { + let vpc_id = VpcId::new(); + let site_prefix_id = SitePrefixId::new(); + let prefix: IpNetwork = "10.0.0.0/24".parse()?; + let version = ConfigVersion::initial(); + let mut setup = pool.begin().await?; + sqlx::query( + "INSERT INTO tenants (organization_id, organization_name, version) \ + VALUES ('tenant-a', 'Tenant A', $1)", + ) + .bind(version) + .execute(setup.as_mut()) + .await?; + sqlx::query( + "INSERT INTO vpcs \ + (id, name, organization_id, version, network_virtualization_type, vni, \ + routing_profile_type, status) \ + VALUES ($1, 'routing-safety-vpc', 'tenant-a', $2, 'fnn', 1001, $3, \ + '{\"vni\": 1001}'::jsonb)", + ) + .bind(vpc_id) + .bind(version) + .bind(PROFILE) + .execute(setup.as_mut()) + .await?; + sqlx::query( + "INSERT INTO site_prefixes \ + (id, prefix, authority, tenant_organization_id, routing_scope, \ + lifecycle_state, name, version) \ + VALUES ($1, $2, 'tenant_managed', 'tenant-a', 'datacenter_only', \ + 'ready', 'routing-safety-root', $3)", + ) + .bind(site_prefix_id) + .bind(prefix) + .bind(version) + .execute(setup.as_mut()) + .await?; + setup.commit().await?; + + let config = config(); + let first_candidate = new_vpc_prefix(VpcPrefixId::new(), vpc_id, site_prefix_id, prefix); + let mut first = pool.begin().await?; + let first_pid: i32 = sqlx::query_scalar("SELECT pg_backend_pid()") + .fetch_one(first.as_mut()) + .await?; + lock_site_mutation(first.as_mut()).await?; + validate_vpc_prefix_candidate(&config, first.as_mut(), &first_candidate, &[]).await?; + db::vpc_prefix::persist(first_candidate, version, first.as_mut()).await?; + + let second_pool = pool.clone(); + let second_config = config.clone(); + let second_candidate = new_vpc_prefix(VpcPrefixId::new(), vpc_id, site_prefix_id, prefix); + let (pid_sender, pid_receiver) = tokio::sync::oneshot::channel(); + let second = tokio::spawn(async move { + let mut txn = second_pool + .begin() + .await + .map_err(|error| error.to_string())?; + let pid: i32 = sqlx::query_scalar("SELECT pg_backend_pid()") + .fetch_one(txn.as_mut()) + .await + .map_err(|error| error.to_string())?; + pid_sender + .send(pid) + .map_err(|_| "could not report second transaction pid".to_string())?; + lock_site_mutation(txn.as_mut()) + .await + .map_err(|error| error.to_string())?; + let result = + validate_vpc_prefix_candidate(&second_config, txn.as_mut(), &second_candidate, &[]) + .await; + txn.rollback().await.map_err(|error| error.to_string())?; + result.map_err(|error| error.to_string()) + }); + + let second_pid = pid_receiver.await?; + tokio::time::timeout(Duration::from_secs(5), async { + loop { + let blocked_by_first: bool = + sqlx::query_scalar("SELECT $1 = ANY(pg_blocking_pids($2))") + .bind(first_pid) + .bind(second_pid) + .fetch_one(&pool) + .await?; + if blocked_by_first { + return Ok::<(), sqlx::Error>(()); + } + tokio::task::yield_now().await; + } + }) + .await + .map_err(|_| std::io::Error::other("second candidate did not wait on the site lock"))??; + + first.commit().await?; + let second_error = second + .await + .map_err(|error| std::io::Error::other(error.to_string()))? + .expect_err("the waiter must reject after re-reading the committed candidate"); + assert!(second_error.contains(OVERLAPPING_ADDRESS_SPACE)); + let persisted: i64 = + sqlx::query_scalar("SELECT COUNT(*) FROM network_vpc_prefixes WHERE prefix = $1") + .bind(prefix) + .fetch_one(&pool) + .await?; + assert_eq!(persisted, 1, "exactly one competing candidate may win"); + Ok(()) + } + + #[test] + fn overlap_is_family_aware_and_detects_nesting() { + let cases = [ + ("10.0.0.0/24", "10.0.0.0/24", true), + ("10.0.0.0/24", "10.0.0.128/25", true), + ("10.0.0.0/25", "10.0.0.128/25", false), + ("2001:db8::/64", "2001:db8::/80", true), + ("10.0.0.0/24", "2001:db8::/64", false), + ]; + for (left, right, expected) in cases { + assert_eq!( + prefixes_overlap(left.parse().unwrap(), right.parse().unwrap()), + expected, + "{left} and {right}" + ); + } + } + + #[test] + fn moderate_disjoint_inventory_reuses_empty_overlap_analysis() { + let vpcs = [vpc("tenant-a", PROFILE, 1001)]; + let roots = [site_prefix("tenant-a", "10.0.0.0/16")]; + let addresses = (0..1024) + .map(|index| { + address( + index, + &vpcs[0], + &roots[0], + &format!("10.0.{}.{}/32", index / 256, index % 256), + ) + }) + .collect(); + let state = state(&vpcs, &roots, addresses); + let analysis = state.analyze_overlaps(); + + assert!(analysis.all_indices.is_empty()); + assert!(analysis.tenant_reuse_indices.is_empty()); + assert!(analysis.occupancy_pairs.is_empty()); + assert_eq!(state.validate_tenant_reuse(&config(), &analysis), Ok(())); + } + + #[test] + fn only_equal_prefixes_are_overlap_eligible() { + let vpcs = [ + vpc("tenant-a", PROFILE, 1001), + vpc("tenant-b", PROFILE, 1002), + ]; + let cases = [ + ("10.0.0.0/24", "10.0.0.0/25", "10.0.0.0/24"), + ("10.0.0.0/25", "10.0.0.0/24", "10.0.0.0/24"), + ("2001:db8::/64", "2001:db8::/80", "2001:db8::/64"), + ("2001:db8::/80", "2001:db8::/64", "2001:db8::/64"), + ]; + + for (left, right, root) in cases { + let roots = [site_prefix("tenant-a", root), site_prefix("tenant-b", root)]; + let state = state( + &vpcs, + &roots, + vec![ + address(0, &vpcs[0], &roots[0], left), + address(1, &vpcs[1], &roots[1], right), + ], + ); + let analysis = state.analyze_overlaps(); + let failure = + validate_candidate_addresses(&config(), &HashSet::new(), &state, &analysis) + .expect_err("nested overlap must not be eligible"); + assert_eq!(failure.violation, RoutingSafetyViolation::AddressConflict); + assert!( + matches!( + CarbideError::from(failure), + CarbideError::InvalidArgument(ref message) if message == OVERLAPPING_ADDRESS_SPACE + ), + "nested overlap {left} and {right} must retain the generic client error" + ); + } + } + + #[test] + fn routing_profile_contract_separates_opt_in_from_isolation() { + let safe = FnnRoutingProfileConfig { + overlap_eligible: true, + internal: Some(true), + ..FnnRoutingProfileConfig::default() + }; + assert!(profile_preserves_isolation(&safe)); + assert!(profile_allows_overlap(&safe)); + + let route_target = RouteTargetConfig { + asn: 64512, + vni: 1001, + }; + let prefix_filter = PrefixFilterPolicyEntry { + prefix: "192.0.2.0/24".parse().expect("valid test prefix"), + }; + let cases = [ + ( + "profile opt-in disabled", + FnnRoutingProfileConfig { + overlap_eligible: false, + ..safe.clone() + }, + true, + ), + ( + "external profile", + FnnRoutingProfileConfig { + internal: Some(false), + ..safe.clone() + }, + false, + ), + ( + "missing internal classification", + FnnRoutingProfileConfig { + internal: None, + ..safe.clone() + }, + false, + ), + ( + "route-target import", + FnnRoutingProfileConfig { + route_target_imports: Some(vec![route_target.clone()]), + ..safe.clone() + }, + false, + ), + ( + "route-target export", + FnnRoutingProfileConfig { + route_targets_on_exports: Some(vec![route_target]), + ..safe.clone() + }, + false, + ), + ( + "default-route leak", + FnnRoutingProfileConfig { + leak_default_route_from_underlay: Some(true), + ..safe.clone() + }, + false, + ), + ( + "tenant host-route leak", + FnnRoutingProfileConfig { + leak_tenant_host_routes_to_underlay: Some(true), + ..safe.clone() + }, + false, + ), + ( + "tenant leak community", + FnnRoutingProfileConfig { + tenant_leak_communities_accepted: Some(true), + ..safe.clone() + }, + false, + ), + ( + "accepted underlay leak", + FnnRoutingProfileConfig { + accepted_leaks_from_underlay: Some(vec![prefix_filter.clone()]), + ..safe.clone() + }, + false, + ), + ( + "allowed anycast prefix", + FnnRoutingProfileConfig { + allowed_anycast_prefixes: Some(vec![prefix_filter]), + ..safe + }, + false, + ), + ]; + for (name, profile, preserves_isolation) in cases { + assert_eq!( + profile_preserves_isolation(&profile), + preserves_isolation, + "{name} isolation" + ); + assert!(!profile_allows_overlap(&profile), "{name} overlap"); + } + } + + #[test] + fn site_policy_rejects_every_feature_that_can_bridge_routing_domains() { + struct SitePolicyCase { + scenario: &'static str, + mutate: fn(&mut CarbideConfig), + expect: Option<(RoutingSafetyViolation, &'static str)>, + } + + let cases = [ + SitePolicyCase { + scenario: "safe defaults", + mutate: |_| {}, + expect: None, + }, + SitePolicyCase { + scenario: "missing FNN configuration", + mutate: |config| config.fnn = None, + expect: Some((RoutingSafetyViolation::UnsafePolicy, "missing_fnn_config")), + }, + SitePolicyCase { + scenario: "open VPC isolation", + mutate: |config| config.vpc_isolation_behavior = VpcIsolationBehaviorType::Open, + expect: Some(( + RoutingSafetyViolation::UnsafePolicy, + "site_isolation_not_mutual", + )), + }, + SitePolicyCase { + scenario: "site-global VPC VNI", + mutate: |config| config.site_global_vpc_vni = Some(1001), + expect: Some(( + RoutingSafetyViolation::UnsafePolicy, + "site_global_vpc_vni_enabled", + )), + }, + SitePolicyCase { + scenario: "site anycast prefix", + mutate: |config| { + config.anycast_site_prefixes = vec!["192.0.2.0/24".parse().unwrap()]; + }, + expect: Some(( + RoutingSafetyViolation::UnsafePolicy, + "site_anycast_prefixes_enabled", + )), + }, + SitePolicyCase { + scenario: "VMaaS configuration", + mutate: |config| { + config.vmaas_config = Some(crate::cfg::file::VmaasConfig { + allow_instance_vf: true, + hbn_reps: None, + bridging: None, + }); + }, + expect: Some((RoutingSafetyViolation::UnsafePolicy, "vmaas_enabled")), + }, + SitePolicyCase { + scenario: "common internal route target", + mutate: |config| { + config.fnn.as_mut().unwrap().common_internal_route_target = + Some(RouteTargetConfig { asn: 64512, vni: 1 }); + }, + expect: Some(( + RoutingSafetyViolation::UnsafePolicy, + "common_internal_route_target_enabled", + )), + }, + SitePolicyCase { + scenario: "additional route-target import", + mutate: |config| { + config.fnn.as_mut().unwrap().additional_route_target_imports = + vec![RouteTargetConfig { asn: 64512, vni: 1 }]; + }, + expect: Some(( + RoutingSafetyViolation::UnsafePolicy, + "additional_route_target_imports_enabled", + )), + }, + SitePolicyCase { + scenario: "deny-only site NSG override", + mutate: |config| { + config.network_security_group.policy_overrides = + vec![security_group_rule(NetworkSecurityGroupRuleAction::Deny)]; + }, + expect: None, + }, + SitePolicyCase { + scenario: "site NSG permit override", + mutate: |config| { + config.network_security_group.policy_overrides = + vec![security_group_rule(NetworkSecurityGroupRuleAction::Permit)]; + }, + expect: Some(( + RoutingSafetyViolation::UnsafePolicy, + "site_nsg_permit_override_enabled", + )), + }, + ]; + let vpcs = []; + let roots = []; + let state = state(&vpcs, &roots, vec![]); + + for SitePolicyCase { + scenario, + mutate, + expect, + } in cases + { + let mut config = config(); + mutate(&mut config); + let got = state + .validate_site_policy(&config) + .map_err(|failure| (failure.violation, failure.reason)) + .err(); + assert_eq!(got, expect, "{scenario}"); + } + } + + #[test] + fn security_group_safety_requires_stateless_deny_only_rules() { + use carbide_test_support::{Check, check_values}; + + check_values( + [ + Check { + scenario: "stateless empty group", + input: (false, vec![]), + expect: true, + }, + Check { + scenario: "stateless deny-only group", + input: (false, vec![NetworkSecurityGroupRuleAction::Deny]), + expect: true, + }, + Check { + scenario: "stateless group containing a permit", + input: ( + false, + vec![ + NetworkSecurityGroupRuleAction::Deny, + NetworkSecurityGroupRuleAction::Permit, + ], + ), + expect: false, + }, + Check { + scenario: "stateful empty group", + input: (true, vec![]), + expect: false, + }, + Check { + scenario: "stateful deny-only group", + input: (true, vec![NetworkSecurityGroupRuleAction::Deny]), + expect: false, + }, + Check { + scenario: "stateful group containing a permit", + input: (true, vec![NetworkSecurityGroupRuleAction::Permit]), + expect: false, + }, + ], + |(stateful_egress, actions)| { + network_security_group_is_safe(&network_security_group( + NetworkSecurityGroupId::from_str("safety-table").unwrap(), + "tenant-a", + stateful_egress, + actions.into_iter().map(security_group_rule).collect(), + )) + }, + ); + } + + #[test] + fn isolated_eligible_overlap_is_accepted_but_site_gate_freezes_expansion() { + let vpcs = [ + vpc("tenant-a", PROFILE, 1001), + vpc("tenant-b", PROFILE, 1002), + ]; + let roots = [ + site_prefix("tenant-a", "10.0.0.0/24"), + site_prefix("tenant-b", "10.0.0.0/24"), + ]; + let state = overlapping_state(&vpcs, &roots); + let mut config = config(); + + assert_eq!(state.validate(&config), Ok(())); + config.tenant_prefix_overlap_enabled = false; + let analysis = state.analyze_overlaps(); + assert!(matches!( + validate_candidate_addresses(&config, &HashSet::new(), &state, &analysis), + Err(failure) if failure.violation == RoutingSafetyViolation::OverlapDisabled + )); + // Startup ignores the admission gate: an operator can turn future + // expansion off without taking a safe existing deployment down. + assert_eq!(state.validate(&config), Ok(())); + } + + #[test] + fn overlap_requires_tenant_site_prefix_and_distinct_native_vnis() { + let vpcs = [ + vpc("tenant-a", PROFILE, 1001), + vpc("tenant-b", PROFILE, 1002), + ]; + let mut roots = [ + site_prefix("tenant-a", "10.0.0.0/24"), + site_prefix("tenant-b", "10.0.0.0/24"), + ]; + roots[1].status.authority = SitePrefixAuthority::OperatorManaged; + assert_eq!( + violation(overlapping_state(&vpcs, &roots).validate(&config())), + Err(RoutingSafetyViolation::IneligibleOverlap) + ); + + roots[1].status.authority = SitePrefixAuthority::TenantManaged; + let mut colliding_vpcs = vpcs; + colliding_vpcs[1].status.vni = Some(1001); + assert_eq!( + violation(overlapping_state(&colliding_vpcs, &roots).validate(&config())), + Err(RoutingSafetyViolation::UnsafePolicy) + ); + } + + #[test] + fn candidate_rejects_direct_and_same_vpc_overlap() { + let vpcs = [ + vpc("tenant-a", PROFILE, 1001), + vpc("tenant-b", PROFILE, 1002), + ]; + let roots = [ + site_prefix("tenant-a", "10.0.0.0/24"), + site_prefix("tenant-b", "10.0.0.0/24"), + ]; + let mut direct = overlapping_state(&vpcs, &roots); + direct.addresses[1].source = AddressSource::Network { id: None }; + assert_eq!(direct.validate(&config()), Ok(())); + let analysis = direct.analyze_overlaps(); + for (gate_enabled, expected_violation) in [ + (false, RoutingSafetyViolation::OverlapDisabled), + (true, RoutingSafetyViolation::AddressConflict), + ] { + let mut candidate_config = config(); + candidate_config.tenant_prefix_overlap_enabled = gate_enabled; + let failure = validate_candidate_addresses( + &candidate_config, + &HashSet::new(), + &direct, + &analysis, + ) + .expect_err("direct-prefix candidate overlap must be rejected"); + assert_eq!(failure.violation, expected_violation); + assert!(matches!( + CarbideError::from(failure), + CarbideError::InvalidArgument(ref message) if message == OVERLAPPING_ADDRESS_SPACE + )); + } + + let mut same_vpc = overlapping_state(&vpcs, &roots); + same_vpc.addresses[1].vpc_id = vpcs[0].id; + assert_eq!(same_vpc.validate(&config()), Ok(())); + let analysis = same_vpc.analyze_overlaps(); + let failure = + validate_candidate_addresses(&config(), &HashSet::new(), &same_vpc, &analysis) + .expect_err("same-VPC candidate overlap must be rejected"); + assert_eq!(failure.violation, RoutingSafetyViolation::AddressConflict); + assert!(matches!( + CarbideError::from(failure), + CarbideError::InvalidArgument(ref message) if message == OVERLAPPING_ADDRESS_SPACE + )); + + let same_tenant_vpcs = [ + vpc("tenant-a", PROFILE, 1001), + vpc("tenant-a", PROFILE, 1002), + ]; + let same_tenant_roots = [ + site_prefix("tenant-a", "10.0.0.0/24"), + site_prefix("tenant-a", "10.0.0.0/24"), + ]; + assert_eq!( + violation(overlapping_state(&same_tenant_vpcs, &same_tenant_roots).validate(&config()),), + Err(RoutingSafetyViolation::IneligibleOverlap) + ); + + let same_root_addresses = vec![ + address(0, &vpcs[0], &roots[0], "10.0.0.0/24"), + address(1, &vpcs[1], &roots[0], "10.0.0.0/24"), + ]; + assert_eq!( + violation(state(&vpcs, &roots, same_root_addresses).validate(&config())), + Err(RoutingSafetyViolation::IneligibleOverlap) + ); + } + + #[test] + fn retained_instance_union_resolves_existing_unbound_segments() { + let vpcs = [ + vpc("tenant-a", PROFILE, 1001), + vpc("tenant-b", PROFILE, 1002), + vpc("tenant-c", PROFILE, 1003), + ]; + let roots = [ + site_prefix("tenant-a", "10.0.0.0/24"), + site_prefix("tenant-b", "10.0.0.0/24"), + ]; + let segment_id = NetworkSegmentId::new(); + let mut current = InstanceNetworkConfig::for_segment_ids(&[segment_id], &[], &[vpcs[0].id]); + current.auto_config = Some(InstanceNetworkAutoConfig { vpc_id: vpcs[0].id }); + let old_config = InstanceNetworkConfig { + auto_config: Some(InstanceNetworkAutoConfig { vpc_id: vpcs[0].id }), + ..Default::default() + }; + let new_config = InstanceNetworkConfig { + auto_config: Some(InstanceNetworkAutoConfig { vpc_id: vpcs[2].id }), + ..Default::default() + }; + let mut addresses = db::routing_safety::RoutingAddressSnapshot { + vpcs: vpcs.to_vec(), + site_prefixes: roots.to_vec(), + vpc_prefixes: vec![ + vpc_prefix(&vpcs[0], &roots[0], "10.0.0.0/24"), + vpc_prefix(&vpcs[1], &roots[1], "10.0.0.0/24"), + ], + network_segments: vec![unbound_host_inband_segment(segment_id)], + }; + let policy = db::routing_safety::RoutingPolicySnapshot { + peerings: Vec::new(), + network_security_groups: Vec::new(), + instances: vec![db::routing_safety::RoutingInstance { + id: InstanceId::new(), + network_config: sqlx::types::Json(current), + update_network_config_request: Some(sqlx::types::Json( + InstanceNetworkConfigUpdate { + old_config, + new_config, + }, + )), + network_security_group_id: None, + }], + }; + + { + let mut state = RoutingState::from_addresses(&addresses); + state.apply_policy_paths(&policy, &addresses); + assert_eq!(state.instance_paths.len(), 1); + assert_eq!( + state.instance_paths[0].vpc_ids, + [vpcs[0].id, vpcs[2].id].into_iter().collect() + ); + assert!(!state.instance_paths[0].has_unresolved_reference); + assert_eq!(state.validate(&config()), Ok(())); + } + + addresses.vpcs[2].config.routing_profile_type = Some(UNSAFE_PROFILE.to_string()); + { + let mut state = RoutingState::from_addresses(&addresses); + state.apply_policy_paths(&policy, &addresses); + assert_eq!( + violation(state.validate(&config())), + Err(RoutingSafetyViolation::UnsafePolicy), + "the pending new-side VPC must remain graph-active" + ); + } + + addresses.vpcs[2].config.routing_profile_type = Some(PROFILE.to_string()); + addresses.network_segments.clear(); + let mut missing = RoutingState::from_addresses(&addresses); + missing.apply_policy_paths(&policy, &addresses); + assert!(missing.instance_paths[0].has_unresolved_reference); + assert_eq!( + violation(missing.validate(&config())), + Err(RoutingSafetyViolation::UnsafePolicy), + "a missing retained segment must fail closed" + ); + } + + #[test] + fn peering_overlap_is_rejected_in_both_endpoint_orders() { + let vpcs = [ + vpc("tenant-a", PROFILE, 1001), + vpc("tenant-b", PROFILE, 1002), + ]; + let roots = [ + site_prefix("tenant-a", "10.0.0.0/24"), + site_prefix("tenant-b", "10.0.0.0/24"), + ]; + let mut config = config(); + config.vpc_peering_policy_on_existing = Some(VpcPeeringPolicy::Mixed); + + for peering in [(vpcs[0].id, vpcs[1].id), (vpcs[1].id, vpcs[0].id)] { + let mut state = overlapping_state(&vpcs, &roots); + state.peerings.push(peering); + assert_eq!( + violation(state.validate(&config)), + Err(RoutingSafetyViolation::ReachableOverlap) + ); + } + + config.vpc_peering_policy_on_existing = None; + config.vpc_peering_policy = Some(VpcPeeringPolicy::Exclusive); + let mut state = overlapping_state(&vpcs, &roots); + state.peerings.push((vpcs[0].id, vpcs[1].id)); + assert_eq!( + violation(state.validate(&config)), + Err(RoutingSafetyViolation::ReachableOverlap), + "existing-peer policy must fall back to the rendered peering policy" + ); + + config.vpc_peering_policy_on_existing = Some(VpcPeeringPolicy::None); + assert_eq!( + violation(state.validate(&config)), + Err(RoutingSafetyViolation::ReachableOverlap), + "FNN peer-VNI imports remain active for an existing peering even when prefix imports are disabled" + ); + + config.vpc_peering_policy_on_existing = None; + config.vpc_peering_policy = None; + assert_eq!( + state.validate(&config), + Ok(()), + "the renderer imports neither peer prefixes nor peer VNIs when both policy options are unset" + ); + } + + #[test] + fn peering_activity_matches_renderer_directionality() { + let virtualization_types = [ + VpcVirtualizationType::EthernetVirtualizer, + VpcVirtualizationType::EthernetVirtualizerWithNvue, + VpcVirtualizationType::Fnn, + VpcVirtualizationType::Flat, + ]; + assert_eq!( + ALL_VPC_VIRTUALIZATION_TYPES, + virtualization_types.as_slice(), + "the directionality matrix must enumerate every virtualization type" + ); + let cases = [ + ( + VpcPeeringPolicy::Exclusive, + [ + [true, true, false, true], + [true, true, false, true], + [false, false, true, true], + [true, true, true, true], + ], + ), + (VpcPeeringPolicy::Mixed, [[true; 4]; 4]), + ( + VpcPeeringPolicy::None, + [ + [false, false, false, false], + [false, false, false, false], + [false, false, true, true], + [false, false, false, false], + ], + ), + ]; + + for (policy, expected) in cases { + for (receiver_index, receiver_type) in virtualization_types.iter().copied().enumerate() + { + for (peer_index, peer_type) in virtualization_types.iter().copied().enumerate() { + assert_eq!( + peering_direction_is_active(policy, receiver_type, peer_type), + expected[receiver_index][peer_index], + "{policy:?}: {receiver_type:?} receiver, {peer_type:?} peer" + ); + } + } + } + } + + #[test] + fn configured_peering_with_missing_endpoint_fails_closed() { + let vpcs = [ + vpc("tenant-a", PROFILE, 1001), + vpc("tenant-b", PROFILE, 1002), + ]; + let roots = [ + site_prefix("tenant-a", "10.0.0.0/24"), + site_prefix("tenant-b", "10.0.0.0/24"), + ]; + let mut state = overlapping_state(&vpcs, &roots); + state.peerings.push((vpcs[0].id, VpcId::new())); + + let mut config = config(); + config.vpc_peering_policy_on_existing = Some(VpcPeeringPolicy::Mixed); + let failure = state + .validate(&config) + .expect_err("missing peer must fail closed"); + assert_eq!(failure.violation, RoutingSafetyViolation::UnsafePolicy); + assert_eq!(failure.reason, "missing_peered_vpc"); + } + + #[test] + fn dormant_peering_does_not_activate_latent_unsafe_profile() { + let vpcs = [ + vpc("tenant-a", PROFILE, 1001), + vpc("tenant-b", PROFILE, 1002), + vpc("tenant-c", UNSAFE_PROFILE, 1003), + ]; + let roots = [ + site_prefix("tenant-a", "10.0.0.0/24"), + site_prefix("tenant-b", "10.0.0.0/24"), + ]; + let mut state = overlapping_state(&vpcs, &roots); + state.peerings.push((vpcs[0].id, vpcs[2].id)); + + let mut config = config(); + config.vpc_peering_policy = None; + config.vpc_peering_policy_on_existing = None; + assert_eq!( + state.validate(&config), + Ok(()), + "an unrendered peering must not make its otherwise unused endpoint active" + ); + + config.vpc_peering_policy_on_existing = Some(VpcPeeringPolicy::None); + assert_eq!( + violation(state.validate(&config)), + Err(RoutingSafetyViolation::UnsafePolicy), + "a configured policy keeps capability-driven FNN peer-VNI imports active" + ); + } + + #[test] + fn receiver_rejects_overlapping_sibling_peers() { + let vpcs = [ + vpc("tenant-a", PROFILE, 1001), + vpc("tenant-b", PROFILE, 1002), + vpc("tenant-c", PROFILE, 1003), + ]; + let roots = [ + site_prefix("tenant-a", "192.0.2.0/24"), + site_prefix("tenant-b", "10.0.0.0/24"), + site_prefix("tenant-c", "10.0.0.0/24"), + ]; + let mut state = state( + &vpcs, + &roots, + vec![ + address(0, &vpcs[1], &roots[1], "10.0.0.0/24"), + address(1, &vpcs[2], &roots[2], "10.0.0.0/24"), + ], + ); + state.peerings = vec![(vpcs[0].id, vpcs[1].id), (vpcs[0].id, vpcs[2].id)]; + let mut config = config(); + config.vpc_peering_policy_on_existing = Some(VpcPeeringPolicy::Mixed); + + assert_eq!( + violation(state.validate(&config)), + Err(RoutingSafetyViolation::ReachableOverlap) + ); + } + + #[test] + fn instance_receiver_rejects_direct_and_peer_visible_overlap() { + #[derive(Clone, Copy)] + struct Case { + name: &'static str, + attached_vpc_indices: &'static [usize], + peering_indices: Option<(usize, usize)>, + } + + let cases = [ + Case { + name: "direct multi-home", + attached_vpc_indices: &[0, 1], + peering_indices: None, + }, + Case { + name: "multi-home through peer", + attached_vpc_indices: &[0, 2], + peering_indices: Some((2, 1)), + }, + Case { + name: "multi-home through reversed peer", + attached_vpc_indices: &[0, 2], + peering_indices: Some((1, 2)), + }, + ]; + + for case in cases { + let vpcs = [ + vpc("tenant-a", PROFILE, 1001), + vpc("tenant-b", PROFILE, 1002), + vpc("tenant-a", PROFILE, 1003), + ]; + let roots = [ + site_prefix("tenant-a", "10.0.0.0/24"), + site_prefix("tenant-b", "10.0.0.0/24"), + ]; + let mut state = overlapping_state(&vpcs, &roots); + let path = instance_path( + case.attached_vpc_indices + .iter() + .map(|index| vpcs[*index].id), + ); + let instance_id = path.instance_id; + state.instance_paths.push(path); + if let Some((left, right)) = case.peering_indices { + state.peerings.push((vpcs[left].id, vpcs[right].id)); + } + + let mut config = config(); + config.vpc_peering_policy = None; + config.vpc_peering_policy_on_existing = + case.peering_indices.map(|_| VpcPeeringPolicy::Mixed); + let failure = state + .validate(&config) + .expect_err("instance receiver must not see both reused prefixes"); + assert_eq!( + failure.violation, + RoutingSafetyViolation::ReachableOverlap, + "{}", + case.name + ); + assert!( + failure + .resource_ids + .contains(&format!("instance:{instance_id}")), + "{}: instance context", + case.name + ); + } + } + + #[test] + fn peering_and_prefix_order_reject_the_same_reachable_overlap() { + let vpcs = [ + vpc("tenant-a", PROFILE, 1001), + vpc("tenant-b", PROFILE, 1002), + ]; + let roots = [ + site_prefix("tenant-a", "10.0.0.0/24"), + site_prefix("tenant-b", "10.0.0.0/24"), + ]; + let mut config = config(); + config.vpc_peering_policy_on_existing = Some(VpcPeeringPolicy::Mixed); + + let mut prefix_first = overlapping_state(&vpcs, &roots); + assert_eq!(prefix_first.validate(&config), Ok(())); + prefix_first.peerings.push((vpcs[0].id, vpcs[1].id)); + assert_eq!( + violation(prefix_first.validate(&config)), + Err(RoutingSafetyViolation::ReachableOverlap), + "adding the peering after the duplicate prefix must fail" + ); + + let mut peer_first = state( + &vpcs, + &roots, + vec![address(0, &vpcs[0], &roots[0], "10.0.0.0/24")], + ); + peer_first.peerings.push((vpcs[0].id, vpcs[1].id)); + assert_eq!(peer_first.validate(&config), Ok(())); + let current_pairs = peer_first.analyze_overlaps().occupancy_pairs; + peer_first + .addresses + .push(address(1, &vpcs[1], &roots[1], "10.0.0.0/24")); + let proposed_analysis = peer_first.analyze_overlaps(); + assert_eq!( + validate_candidate_addresses(&config, ¤t_pairs, &peer_first, &proposed_analysis,), + Ok(()) + ); + assert!(matches!( + peer_first.validate_policy_paths(&config, &proposed_analysis), + Err(failure) if failure.violation == RoutingSafetyViolation::ReachableOverlap + )); + } + + #[test] + fn latent_unsafe_profile_does_not_block_until_its_vpc_becomes_active() { + let vpcs = [ + vpc("tenant-a", PROFILE, 1001), + vpc("tenant-b", PROFILE, 1002), + vpc("tenant-c", UNSAFE_PROFILE, 1003), + ]; + let roots = [ + site_prefix("tenant-a", "10.0.0.0/24"), + site_prefix("tenant-b", "10.0.0.0/24"), + site_prefix("tenant-c", "192.0.2.0/24"), + ]; + let mut state = overlapping_state(&vpcs, &roots); + assert_eq!(state.validate(&config()), Ok(())); + + state + .addresses + .push(address(2, &vpcs[2], &roots[2], "192.0.2.0/24")); + assert_eq!( + violation(state.validate(&config())), + Err(RoutingSafetyViolation::UnsafePolicy) + ); + } + + #[test] + fn routing_profile_contract_allows_unrelated_active_vpc_without_opt_in() { + let vpcs = [ + vpc("tenant-a", PROFILE, 1001), + vpc("tenant-b", PROFILE, 1002), + vpc("tenant-c", ISOLATED_PROFILE, 1003), + ]; + let roots = [ + site_prefix("tenant-a", "10.0.0.0/24"), + site_prefix("tenant-b", "10.0.0.0/24"), + site_prefix("tenant-c", "192.0.2.0/24"), + ]; + let mut state = overlapping_state(&vpcs, &roots); + state + .addresses + .push(address(2, &vpcs[2], &roots[2], "192.0.2.0/24")); + + assert_eq!(state.validate(&config()), Ok(())); + } + + #[test] + fn routing_profile_contract_requires_opt_in_for_overlap_participants() { + let vpcs = [ + vpc("tenant-a", PROFILE, 1001), + vpc("tenant-b", ISOLATED_PROFILE, 1002), + ]; + let roots = [ + site_prefix("tenant-a", "10.0.0.0/24"), + site_prefix("tenant-b", "10.0.0.0/24"), + ]; + + let failure = overlapping_state(&vpcs, &roots) + .validate(&config()) + .expect_err("each overlapping participant must opt in"); + assert_eq!(failure.violation, RoutingSafetyViolation::UnsafePolicy); + assert_eq!(failure.reason, "unsafe_vpc_routing_profile"); + } + + #[test] + fn routing_profile_contract_requires_resolvable_profile_when_active() { + let mut profileless = vpc("tenant-c", PROFILE, 1003); + profileless.config.routing_profile_type = None; + let vpcs = [ + vpc("tenant-a", PROFILE, 1001), + vpc("tenant-b", PROFILE, 1002), + profileless, + ]; + let roots = [ + site_prefix("tenant-a", "10.0.0.0/24"), + site_prefix("tenant-b", "10.0.0.0/24"), + site_prefix("tenant-c", "192.0.2.0/24"), + ]; + let mut state = overlapping_state(&vpcs, &roots); + assert_eq!(state.validate(&config()), Ok(())); + + state + .addresses + .push(address(2, &vpcs[2], &roots[2], "192.0.2.0/24")); + let failure = state + .validate(&config()) + .expect_err("a graph-active FNN VPC must have a resolvable profile"); + assert_eq!(failure.violation, RoutingSafetyViolation::UnsafePolicy); + assert_eq!(failure.reason, "unresolved_vpc_routing_profile"); + } + + #[test] + fn active_fnn_vnis_must_be_present_and_unique() { + let cases = [ + ("unique VNI", Some(1003), None), + ("missing VNI", None, Some("missing_active_vpc_vni")), + ( + "duplicate VNI", + Some(1001), + Some("duplicate_active_vpc_vni"), + ), + ]; + + for (scenario, third_vni, expected_reason) in cases { + let mut third = vpc("tenant-c", PROFILE, 1003); + third.status.vni = third_vni; + let vpcs = [ + vpc("tenant-a", PROFILE, 1001), + vpc("tenant-b", PROFILE, 1002), + third, + ]; + let roots = [ + site_prefix("tenant-a", "10.0.0.0/24"), + site_prefix("tenant-b", "10.0.0.0/24"), + site_prefix("tenant-c", "192.0.2.0/24"), + ]; + let mut state = overlapping_state(&vpcs, &roots); + state + .addresses + .push(address(2, &vpcs[2], &roots[2], "192.0.2.0/24")); + + assert_eq!( + state + .validate(&config()) + .map_err(|failure| failure.reason) + .err(), + expected_reason, + "{scenario}" + ); + } + } + + #[test] + fn control_path_exemption_requires_authoritative_unconsumed_admin_vpc() { + #[derive(Clone, Copy)] + enum Consumer { + None, + Instance, + Peering, + } + + const CONTROL_ONLY: &[NetworkSegmentType] = &[NetworkSegmentType::Admin]; + const HYBRID: &[NetworkSegmentType] = + &[NetworkSegmentType::Admin, NetworkSegmentType::Tenant]; + #[derive(Clone, Copy)] + struct Case { + name: &'static str, + tenant: &'static str, + segment_types: &'static [NetworkSegmentType], + profile: &'static str, + admin_enabled: bool, + configured_admin_vni: u32, + persisted_config_vni: i32, + persisted_status_vni: i32, + consumer: Consumer, + expect_exempt: bool, + expect_reason: Option<&'static str>, + } + let authoritative = Case { + name: "authoritative unconsumed admin VPC", + tenant: "carbide_internal", + segment_types: CONTROL_ONLY, + profile: UNSAFE_PROFILE, + admin_enabled: true, + configured_admin_vni: 1003, + persisted_config_vni: 1003, + persisted_status_vni: 1003, + consumer: Consumer::None, + expect_exempt: true, + expect_reason: None, + }; + let cases = [ + authoritative, + Case { + name: "tenant-owned Admin-only VPC", + tenant: "tenant-c", + expect_exempt: false, + expect_reason: Some("unsafe_vpc_routing_profile"), + ..authoritative + }, + Case { + name: "hybrid VPC with unsafe profile", + segment_types: HYBRID, + expect_exempt: false, + expect_reason: Some("unsafe_vpc_routing_profile"), + ..authoritative + }, + Case { + name: "hybrid VPC with permit NSG", + segment_types: HYBRID, + profile: PROFILE, + expect_exempt: false, + expect_reason: Some("unsafe_active_network_security_group"), + ..authoritative + }, + Case { + name: "instance-consumed authoritative admin VPC", + consumer: Consumer::Instance, + expect_exempt: false, + expect_reason: Some("unsafe_vpc_routing_profile"), + ..authoritative + }, + Case { + name: "peering-consumed authoritative admin VPC", + consumer: Consumer::Peering, + expect_exempt: false, + expect_reason: Some("unsafe_vpc_routing_profile"), + ..authoritative + }, + Case { + name: "disabled admin config", + admin_enabled: false, + expect_exempt: false, + expect_reason: Some("unsafe_vpc_routing_profile"), + ..authoritative + }, + Case { + name: "configured VNI mismatch", + configured_admin_vni: 2003, + expect_exempt: false, + expect_reason: Some("unsafe_vpc_routing_profile"), + ..authoritative + }, + Case { + name: "persisted config VNI mismatch", + persisted_config_vni: 2003, + expect_exempt: false, + expect_reason: Some("unsafe_vpc_routing_profile"), + ..authoritative + }, + Case { + name: "persisted status VNI mismatch", + persisted_status_vni: 2003, + expect_exempt: false, + expect_reason: Some("unsafe_vpc_routing_profile"), + ..authoritative + }, + ]; + + for case in cases { + let group_id = NetworkSecurityGroupId::from_str("hybrid-vpc-policy").unwrap(); + let mut vpcs = [ + vpc("tenant-a", PROFILE, 1001), + vpc("tenant-b", PROFILE, 1002), + vpc(case.tenant, case.profile, case.persisted_status_vni), + ]; + vpcs[2].config.vni = Some(case.persisted_config_vni); + vpcs[2].config.network_security_group_id = Some(group_id.clone()); + let roots = [ + site_prefix("tenant-a", "10.0.0.0/24"), + site_prefix("tenant-b", "10.0.0.0/24"), + site_prefix(case.tenant, "192.0.2.0/24"), + ]; + let addresses = db::routing_safety::RoutingAddressSnapshot { + vpcs: vpcs.to_vec(), + site_prefixes: roots.to_vec(), + vpc_prefixes: vec![ + vpc_prefix(&vpcs[0], &roots[0], "10.0.0.0/24"), + vpc_prefix(&vpcs[1], &roots[1], "10.0.0.0/24"), + vpc_prefix(&vpcs[2], &roots[2], "192.0.2.0/24"), + ], + network_segments: case + .segment_types + .iter() + .copied() + .map(|segment_type| bound_segment(vpcs[2].id, segment_type)) + .collect(), + }; + let mut state = RoutingState::from_addresses(&addresses); + state.network_security_groups.insert( + group_id.clone(), + network_security_group( + group_id, + case.tenant, + false, + vec![security_group_rule(NetworkSecurityGroupRuleAction::Permit)], + ), + ); + match case.consumer { + Consumer::None => {} + Consumer::Instance => state.instance_paths.push(instance_path([vpcs[2].id])), + Consumer::Peering => state.peerings.push((vpcs[2].id, vpcs[0].id)), + } + let mut config = config(); + config.fnn.as_mut().unwrap().admin_vpc = Some(crate::cfg::file::AdminFnnConfig { + enabled: case.admin_enabled, + vpc_vni: Some(case.configured_admin_vni), + routing_profile: FnnRoutingProfileConfig::default(), + }); + + assert_eq!( + state.is_unconsumed_admin_vpc(&config, &vpcs[2]), + case.expect_exempt, + "{}: control classification", + case.name + ); + assert_eq!( + state + .validate(&config) + .map_err(|failure| failure.reason) + .err(), + case.expect_reason, + "{}: active-path validation", + case.name + ); + } + } + + #[test] + fn active_interface_overrides_and_stateful_security_groups_are_unsafe() { + let mut vpcs = [ + vpc("tenant-a", PROFILE, 1001), + vpc("tenant-b", PROFILE, 1002), + ]; + let roots = [ + site_prefix("tenant-a", "10.0.0.0/24"), + site_prefix("tenant-b", "10.0.0.0/24"), + ]; + let group_id = NetworkSecurityGroupId::from_str("unsafe-group").unwrap(); + vpcs[0].config.network_security_group_id = Some(group_id.clone()); + vpcs[0] + .metadata + .labels + .insert("kind".to_string(), "admin".to_string()); + let mut state = overlapping_state(&vpcs, &roots); + let group = network_security_group(group_id.clone(), "tenant-a", true, vec![]); + state.network_security_groups.insert(group_id, group); + assert_eq!( + violation(state.validate(&config())), + Err(RoutingSafetyViolation::UnsafePolicy) + ); + + let mut vpcs = [ + vpc("tenant-a", PROFILE, 1001), + vpc("tenant-b", PROFILE, 1002), + ]; + vpcs[0].config.network_security_group_id = None; + let mut state = overlapping_state(&vpcs, &roots); + state.instance_paths.push(InstancePath { + instance_id: InstanceId::new(), + vpc_ids: HashSet::from([vpcs[0].id]), + network_security_group_id: None, + has_routing_override: true, + has_unresolved_reference: false, + }); + assert_eq!( + violation(state.validate(&config())), + Err(RoutingSafetyViolation::UnsafePolicy) + ); + + let mut state = overlapping_state(&vpcs, &roots); + state.instance_paths.push(InstancePath { + instance_id: InstanceId::new(), + vpc_ids: HashSet::new(), + network_security_group_id: None, + has_routing_override: true, + has_unresolved_reference: true, + }); + assert_eq!( + violation(state.validate(&config())), + Err(RoutingSafetyViolation::UnsafePolicy) + ); + } + + #[test] + fn public_overlap_errors_do_not_disclose_foreign_resources() { + let disabled_error = CarbideError::from(RoutingSafetyViolation::OverlapDisabled); + assert!(matches!( + disabled_error, + CarbideError::InvalidArgument(ref message) if message == OVERLAPPING_ADDRESS_SPACE + )); + + for violation in [ + RoutingSafetyViolation::OverlapDisabled, + RoutingSafetyViolation::AddressConflict, + RoutingSafetyViolation::IneligibleOverlap, + RoutingSafetyViolation::ReachableOverlap, + RoutingSafetyViolation::UnsafePolicy, + ] { + let error = CarbideError::from(violation).to_string(); + assert!(!error.contains("10.0.0.0")); + assert!(!error.contains("tenant-a")); + assert!(!error.contains("00000000-0000")); + } + } +} diff --git a/crates/api-core/src/setup.rs b/crates/api-core/src/setup.rs index 2cc2a97ffb..f380cbd4e0 100644 --- a/crates/api-core/src/setup.rs +++ b/crates/api-core/src/setup.rs @@ -269,6 +269,9 @@ pub(crate) async fn start_runtime( if let Some(seed_data) = seed_data.as_ref() { // Determine the authoritative list of resource_pools to seed into the database let mut txn = Transaction::begin(&db_pool).await?; + // This must be the transaction's first database lock. Configured + // SitePrefix reconciliation below changes the retained routing graph. + crate::routing_safety::lock_site_mutation(&mut txn).await?; db::resource_pool::reconcile_pool_defs(&mut txn, &seed_data.initial_pools).await?; // We'll always update whatever route servers are in the config @@ -528,7 +531,7 @@ pub(crate) async fn start_runtime( nmxc_client_pool: shared_nmxc_pool.clone(), work_lock_manager_handle, dpf_sdk: dpf_sdk.clone(), - machine_state_handler_enqueuer: Enqueuer::new(db_pool), + machine_state_handler_enqueuer: Enqueuer::new(db_pool.clone()), metric_emitter: ApiMetricsEmitter::new(&meter), component_manager, bms_client: std::sync::OnceLock::new(), @@ -537,6 +540,7 @@ pub(crate) async fn start_runtime( if carbide_config.listen_only { tracing::info!("Not starting background services, as listen_only=true"); + crate::routing_safety::validate_startup(&carbide_config, &db_pool).await?; } else { initialize_and_start_controllers( join_set, @@ -1193,7 +1197,7 @@ async fn initialize_and_start_controllers<'a>( && let Some(admin) = fnn_config.admin_vpc.as_ref() && admin.enabled { - db_init::create_admin_vpc(db_pool, admin.vpc_vni).await?; + db_init::create_admin_vpc(db_pool, carbide_config, admin.vpc_vni).await?; } // Update SVI IP to segments which have VPC attached and type is FNN. db_init::update_network_segments_svi_ip(db_pool).await?; @@ -1218,6 +1222,10 @@ async fn initialize_and_start_controllers<'a>( ); } + // Startup routing writes are complete. Validate them before a state + // controller can render or mutate the retained routing graph. + crate::routing_safety::validate_startup(carbide_config, db_pool).await?; + let downloader = FirmwareDownloader::new(); let upload_limiter = Arc::new(Semaphore::new(carbide_config.firmware_global.max_uploads)); diff --git a/crates/api-core/src/test_support/default_config.rs b/crates/api-core/src/test_support/default_config.rs index 5ef76b8836..4abc2ae8bc 100644 --- a/crates/api-core/src/test_support/default_config.rs +++ b/crates/api-core/src/test_support/default_config.rs @@ -180,6 +180,7 @@ pub fn get() -> CarbideConfig { enable_route_servers: false, deny_prefixes: vec![], site_fabric_prefixes: vec![], + tenant_prefix_overlap_enabled: false, max_site_prefixes_per_tenant: default_max_site_prefixes_per_tenant(), anycast_site_prefixes: vec![], common_tenant_host_asn: None, diff --git a/crates/api-core/src/tests/machine_admin_force_delete.rs b/crates/api-core/src/tests/machine_admin_force_delete.rs index 1cbd15417c..e8f49e7096 100644 --- a/crates/api-core/src/tests/machine_admin_force_delete.rs +++ b/crates/api-core/src/tests/machine_admin_force_delete.rs @@ -26,10 +26,11 @@ use ::rpc::forge::{ }; use carbide_dpf::DpuDeploymentType; use carbide_ib_fabric::config::IBFabricConfig; -use carbide_ib_fabric::ib::{self, IBFabricManager}; +use carbide_ib_fabric::ib::{self, GetPartitionOptions, IBFabricManager}; use carbide_machine_controller::dpf::{DpfOperations, MockDpfOperations}; use carbide_uuid::infiniband::IBPartitionId; use carbide_uuid::machine::{MachineId, MachineType}; +use carbide_uuid::vpc::VpcId; use common::api_fixtures::dpu::create_dpu_machine; use common::api_fixtures::host::host_discover_dhcp; use common::api_fixtures::ib_partition::{DEFAULT_TENANT, create_ib_partition}; @@ -42,6 +43,7 @@ use common::api_fixtures::{ }; use model::hardware_info::TpmEkCertificate; use model::ib::DEFAULT_IB_FABRIC_NAME; +use model::instance::config::network::InstanceNetworkConfig; use model::machine::machine_search_config::MachineSearchConfig; use model::machine::{InstanceState, ManagedHostState}; use model::resource_pool::{ResourcePoolDef, ResourcePoolType}; @@ -421,6 +423,123 @@ async fn test_admin_force_delete_orders_locks_against_exploration(pool: sqlx::Pg validate_machine_deletion(&env, &host.dpu_ids[0], None).await; } +/// Force-delete must serialize behind an in-flight network update and clean +/// every generated segment from the update that committed before its fence. +#[crate::sqlx_test] +async fn test_admin_force_delete_serializes_with_pending_network_update(pool: sqlx::PgPool) { + let env = create_test_env(pool).await; + let segment_id = env.create_vpc_and_tenant_segment().await; + let vpc_id: VpcId = sqlx::query_scalar("SELECT vpc_id FROM network_segments WHERE id = $1") + .bind(segment_id) + .fetch_one(&env.pool) + .await + .unwrap(); + let vpc_prefix_id = env + .api + .create_vpc_prefix(Request::new(rpc::forge::VpcPrefixCreationRequest { + id: None, + prefix: String::new(), + vpc_id: Some(vpc_id), + site_prefix_id: None, + config: Some(rpc::forge::VpcPrefixConfig { + prefix: "192.0.5.0/25".to_string(), + }), + metadata: Some(rpc::forge::Metadata { + name: "force-delete pending prefix".to_string(), + description: "concurrent cleanup regression".to_string(), + labels: vec![], + }), + })) + .await + .unwrap() + .into_inner() + .id + .unwrap(); + let managed_host = create_managed_host(&env).await; + let instance = managed_host + .instance_builer(&env) + .single_interface_network_config(segment_id) + .tenant_org(crate::test_support::network_segment::FIXTURE_TENANT_ORG_ID) + .build() + .await; + + let mut update_txn = env.pool.begin().await.unwrap(); + db::routing_safety::lock_site_mutation(update_txn.as_mut()) + .await + .unwrap(); + + let api = managed_host.api.clone(); + let host_id = managed_host.id; + let force_delete_task = tokio::spawn(async move { + api.admin_force_delete_machine(tonic::Request::new(AdminForceDeleteMachineRequest { + host_query: host_id.to_string(), + delete_interfaces: false, + delete_bmc_interfaces: false, + delete_bmc_credentials: false, + allow_delete_with_orphaned_dpf_crds: false, + })) + .await + }); + wait_until_blocked_on(&env.pool, "pg_advisory_xact_lock").await; + + let current = db::instance::find_by_id(update_txn.as_mut(), instance.id) + .await + .unwrap() + .unwrap(); + let mut requested = InstanceNetworkConfig::for_vpc_prefix_id(vpc_prefix_id, Some(vpc_id)); + crate::instance::allocate_network( + &mut requested, + ¤t.config.tenant.tenant_organization_id, + update_txn.as_mut(), + ) + .await + .unwrap(); + let generated_segment_id = requested.interfaces[0] + .generated_network_segment_id() + .expect("prefix allocation creates a distinct generated segment"); + assert_ne!(generated_segment_id, segment_id); + db::instance::trigger_update_network_config_request( + &instance.id, + ¤t.config.network, + &requested, + &mut update_txn, + ) + .await + .unwrap(); + update_txn.commit().await.unwrap(); + + let response = tokio::time::timeout(std::time::Duration::from_secs(60), force_delete_task) + .await + .expect( + "force delete did not complete after the network update commit released the site lock", + ) + .unwrap() + .expect("force delete completes after the network update commits") + .into_inner(); + assert!(response.all_done); + + let generated_deleted: Option> = + sqlx::query_scalar("SELECT deleted FROM network_segments WHERE id = $1") + .bind(generated_segment_id) + .fetch_one(&env.pool) + .await + .unwrap(); + assert!( + generated_deleted.is_some(), + "force delete must clean the generated segment from the committed update" + ); + let original_deleted: Option> = + sqlx::query_scalar("SELECT deleted FROM network_segments WHERE id = $1") + .bind(segment_id) + .fetch_one(&env.pool) + .await + .unwrap(); + assert!( + original_deleted.is_none(), + "force delete must not mistake the original shared segment for a generated one" + ); +} + /// Multi-endpoint exploration persistence and force-delete must acquire /// `explored_endpoints` rows in the same ascending address order. The fixture /// allocates DPU BMC addresses before the host BMC address, so the old @@ -721,10 +840,8 @@ async fn validate_machine_deletion( txn.rollback().await.unwrap(); } -// TODO: Test deletion for machines with active instances on them - #[crate::sqlx_test] -async fn test_admin_force_delete_host_with_ib_instance(pool: sqlx::PgPool) { +async fn test_admin_force_delete_host_with_concurrent_ib_update(pool: sqlx::PgPool) { let mut config = common::api_fixtures::get_config(); config.ib_config = Some(IBFabricConfig { enabled: true, @@ -868,10 +985,193 @@ async fn test_admin_force_delete_host_with_ib_instance(pool: sqlx::PgPool) { }; assert_eq!(ib_fabric.find_ib_port(Some(filter)).await.unwrap().len(), 1); - let response = force_delete(&env, &mh.id).await; + // Move the fabric to the state produced by the pending config update. The + // old implementation could unregister the stale GUID before the update + // committed, then delete the row without ever unregistering this GUID. + let sorted_ib_interfaces = crate::instance::sort_ib_by_slot( + &machine + .status + .hardware_info + .as_ref() + .unwrap() + .infiniband_interfaces, + ); + let updated_guid = sorted_ib_interfaces["MT2910 Family [ConnectX-7]"][0] + .guid + .clone(); + assert!(!guids.contains(&updated_guid)); + let ib_network = ib_fabric + .get_ib_network( + pkey, + GetPartitionOptions { + include_guids_data: false, + include_qos_conf: true, + }, + ) + .await + .unwrap(); + ib_fabric + .bind_ib_ports(ib_network, vec![updated_guid.clone()]) + .await + .unwrap(); + ib_fabric + .unbind_ib_ports(pkey, guids.iter().cloned().collect()) + .await + .unwrap(); + + let late_config = check_instance.config().inner().clone(); + let mut updated_config = late_config.clone(); + updated_config.infiniband = Some(rpc::forge::InstanceInfinibandConfig { + ib_interfaces: vec![rpc::forge::InstanceIbInterfaceConfig { + function_type: rpc::forge::InterfaceFunctionType::Physical as i32, + virtual_function_id: None, + ib_partition_id: Some(ib_partition_id), + device: "MT2910 Family [ConnectX-7]".to_string(), + vendor: None, + device_instance: 0, + }], + }); + let metadata = check_instance.metadata().clone(); + let instance_id = check_instance.id(); + let late_metadata = metadata.clone(); + + // Pause the real update after it has acquired the site lock and loaded the + // `Ready` machine snapshot. Force-delete must wait behind that update before + // it publishes `ForceDeletion` and takes its UFM cleanup snapshot. + let mut instance_row_lock = env.pool.begin().await.unwrap(); + sqlx::query("UPDATE instances SET id = id WHERE id = $1") + .bind(instance_id) + .execute(instance_row_lock.as_mut()) + .await + .unwrap(); + + let update_api = env.api.clone(); + let update_task = tokio::spawn(async move { + update_api + .update_instance_config(Request::new(rpc::forge::InstanceConfigUpdateRequest { + instance_id: Some(instance_id), + if_version_match: None, + config: Some(updated_config), + metadata: Some(metadata), + })) + .await + }); + wait_until_blocked_on(&env.pool, "UPDATE instances SET").await; + + // Queue an exclusive IB-partition table lock behind the update's ownership + // validation. Once the update commits, this pauses force-delete after its + // deletion fence commits but before its unlocked UFM lookup completes. + let partition_lock_pool = env.pool.clone(); + let partition_lock_task = tokio::spawn(async move { + let mut partition_lock = partition_lock_pool.begin().await.unwrap(); + sqlx::query("LOCK TABLE ib_partitions IN ACCESS EXCLUSIVE MODE") + .execute(partition_lock.as_mut()) + .await + .unwrap(); + partition_lock + }); + wait_until_blocked_on(&env.pool, "LOCK TABLE ib_partitions").await; + + let delete_api = env.api.clone(); + let machine_id = mh.id; + let force_delete_task = tokio::spawn(async move { + delete_api + .admin_force_delete_machine(Request::new(AdminForceDeleteMachineRequest { + host_query: machine_id.to_string(), + delete_interfaces: false, + delete_bmc_interfaces: false, + delete_bmc_credentials: false, + allow_delete_with_orphaned_dpf_crds: false, + })) + .await + }); + wait_until_blocked_on(&env.pool, "pg_advisory_xact_lock").await; + + instance_row_lock.commit().await.unwrap(); + update_task + .await + .unwrap() + .expect("the config update that owns the site lock must commit first"); + + let partition_lock = partition_lock_task.await.unwrap(); + wait_until_blocked_on(&env.pool, "FROM ib_partitions").await; + + let deleted: Option> = + sqlx::query_scalar("SELECT deleted FROM instances WHERE id = $1") + .bind(instance_id) + .fetch_one(&env.pool) + .await + .unwrap(); + assert!( + deleted.is_some(), + "force delete must publish the durable instance fence before UFM cleanup" + ); + + // Model a machine controller finishing work from its pre-delete snapshot. + // The current controller persistence path can replace `ForceDeletion`, so + // the instance flag must continue to fence config writes on its own. + let mut stale_controller_write = env.pool.begin().await.unwrap(); + db::machine::update_state( + stale_controller_write.as_mut(), + &mh.id, + &ManagedHostState::Assigned { + instance_state: InstanceState::Ready, + }, + ) + .await + .unwrap(); + stale_controller_write.commit().await.unwrap(); + + // This is the original device selection, so accepting it would perform a + // real InfiniBand mutation rather than an idempotent metadata-only write. + let late_update_error = env + .api + .update_instance_config(Request::new(rpc::forge::InstanceConfigUpdateRequest { + instance_id: Some(instance_id), + if_version_match: None, + config: Some(late_config), + metadata: Some(late_metadata), + })) + .await + .expect_err("an update starting after the deletion fence must be rejected"); + assert_eq!(late_update_error.code(), tonic::Code::InvalidArgument); + assert_eq!( + late_update_error.message(), + "configuration for a terminating instance can not be changed" + ); + + // Model the machine controller winning the site lock and completing its + // atomic hard-delete phase while admin cleanup is outside the database for + // UFM work. The fixture uses a shared segment, so there is no generated + // instance-owned segment for that owner to mark alongside the row delete. + let mut controller_cleanup = env.pool.begin().await.unwrap(); + db::routing_safety::lock_site_mutation(controller_cleanup.as_mut()) + .await + .unwrap(); + db::instance::delete(instance_id, controller_cleanup.as_mut()) + .await + .unwrap(); + controller_cleanup.commit().await.unwrap(); + + partition_lock.commit().await.unwrap(); + + let response = tokio::time::timeout(std::time::Duration::from_secs(60), force_delete_task) + .await + .expect("force delete did not complete after the config update released the site lock") + .unwrap() + .expect("force delete completes after the config update") + .into_inner(); validate_delete_response(&response, Some(&mh.id), &mh.dpu().id); - // after host deleted, ib port should be removed from UFM + // The cleanup snapshot must include the just-committed GUID. + let filter = ib::Filter { + guids: Some(HashSet::from_iter([updated_guid])), + pkey: Some(pkey), + state: Some(model::ib::IBPortState::Active), + }; + assert_eq!(ib_fabric.find_ib_port(Some(filter)).await.unwrap().len(), 0); + + // after host deleted, the original ib port should remain absent from UFM let filter = ib::Filter { guids: Some(guids.iter().cloned().collect()), pkey: Some(pkey), diff --git a/crates/api-core/src/tests/machine_network.rs b/crates/api-core/src/tests/machine_network.rs index f5ae9f0971..2074240e4b 100644 --- a/crates/api-core/src/tests/machine_network.rs +++ b/crates/api-core/src/tests/machine_network.rs @@ -981,7 +981,7 @@ async fn test_managed_host_network_config_omits_admin_fnn_vrf_loopback_by_defaul let env = api_fixtures::create_test_env_with_overrides(pool, overrides).await; // Attach the FNN admin VPC because test env setup does not run production setup hooks. - crate::db_init::create_admin_vpc(&env.pool, Some(10000)) + crate::db_init::create_admin_vpc(&env.pool, &env.config, Some(10000)) .await .unwrap(); crate::db_init::update_network_segments_svi_ip(&env.pool) @@ -1230,7 +1230,7 @@ async fn test_managed_host_network_config_multi_dpu_fnn_ipv6_loopbacks(pool: sql routing_profile: FnnRoutingProfileConfig::default(), }); let env = api_fixtures::create_test_env_with_overrides(pool, overrides).await; - crate::db_init::create_admin_vpc(&env.pool, Some(10000)) + crate::db_init::create_admin_vpc(&env.pool, &env.config, Some(10000)) .await .unwrap(); crate::db_init::update_network_segments_svi_ip(&env.pool) diff --git a/crates/api-core/src/tests/network_segment.rs b/crates/api-core/src/tests/network_segment.rs index c847cc52b1..b0fbb51cae 100644 --- a/crates/api-core/src/tests/network_segment.rs +++ b/crates/api-core/src/tests/network_segment.rs @@ -1817,7 +1817,7 @@ async fn test_update_svi_ip_admin_segment( let env = create_test_env(pool).await; // This should create VPC for admin segment - db_init::create_admin_vpc(&env.pool, Some(10600)).await?; + db_init::create_admin_vpc(&env.pool, &env.config, Some(10600)).await?; let mut txn = env.pool.begin().await?; let admin_segments = db::network_segment::admin(&mut txn).await?; @@ -2445,6 +2445,63 @@ async fn attach_host_inband_segment_to_flat_vpc_succeeds( Ok(()) } +#[crate::sqlx_test] +async fn attach_host_inband_segment_rejects_new_routed_overlap( + pool: sqlx::PgPool, +) -> Result<(), Box> { + let env = create_test_env_with_overrides(pool, TestEnvOverrides::no_network_segments()).await; + let (vpc_id, _vpc) = + common::api_fixtures::vpc::create_flat_vpc(&env, "flat-overlap".to_string(), None).await; + env.api + .create_vpc_prefix(Request::new(rpc::forge::VpcPrefixCreationRequest { + id: None, + prefix: String::new(), + vpc_id: Some(vpc_id), + site_prefix_id: None, + config: Some(rpc::forge::VpcPrefixConfig { + prefix: "192.1.4.0/24".to_string(), + }), + metadata: Some(rpc::forge::Metadata { + name: "flat overlap parent".to_string(), + description: "attachment admission regression".to_string(), + labels: vec![], + }), + })) + .await?; + let segment = create_unattached_segment( + &env, + "ATTACH_HOST_INBAND_OVERLAP", + "192.1.4.0/24", + "192.1.4.1", + rpc::forge::NetworkSegmentType::HostInband, + ) + .await?; + let segment_id = segment.id.expect("created segment has an ID"); + + let error = attach_network_segment_to_vpc(&env, segment_id, vpc_id, false) + .await + .expect_err("binding an unbound prefix must run candidate admission"); + + assert_eq!(error.code(), tonic::Code::InvalidArgument); + assert_eq!( + error.message(), + "requested address space overlaps existing routed address space" + ); + let mut txn = env.pool.begin().await?; + let persisted = db::network_segment::find_by( + txn.as_mut(), + ObjectColumnFilter::One(db::network_segment::IdColumn, &segment_id), + network_segment::NetworkSegmentSearchConfig::default(), + ) + .await? + .pop() + .expect("segment remains after rejected attachment"); + assert_eq!(persisted.config.vpc_id, None); + txn.rollback().await?; + + Ok(()) +} + #[crate::sqlx_test] async fn attach_host_inband_segment_to_same_vpc_is_idempotent( pool: sqlx::PgPool, diff --git a/crates/api-core/src/tests/vpc.rs b/crates/api-core/src/tests/vpc.rs index f504a89e42..743bc99364 100644 --- a/crates/api-core/src/tests/vpc.rs +++ b/crates/api-core/src/tests/vpc.rs @@ -20,7 +20,7 @@ use std::ops::DerefMut; use carbide_network::virtualization::VpcVirtualizationType; use carbide_test_support::Outcome::FailsWith; use carbide_test_support::{Case, check_cases_async}; -use carbide_uuid::vpc::VpcId; +use carbide_uuid::vpc::{VpcId, VpcPrefixId}; use common::api_fixtures::{create_test_env, populate_network_security_groups}; use config_version::ConfigVersion; use db::vpc::{self}; @@ -1589,7 +1589,7 @@ async fn vpc_deletion_is_idempotent(pool: sqlx::PgPool) -> Result<(), eyre::Repo async fn create_admin_vpc(pool: sqlx::PgPool) -> Result<(), eyre::Report> { let env = create_test_env(pool).await; let vni = 10000; - db_init::create_admin_vpc(&env.pool, Some(vni)).await?; + db_init::create_admin_vpc(&env.pool, &env.config, Some(vni)).await?; let mut txn = env.pool.begin().await?; let mut admin_vpc = db::vpc::find_by_vni(&mut txn, vni as i32).await?; @@ -1610,6 +1610,135 @@ async fn create_admin_vpc(pool: sqlx::PgPool) -> Result<(), eyre::Report> { Ok(()) } +#[crate::sqlx_test] +async fn create_admin_vpc_preserves_legacy_prefix_containment( + pool: sqlx::PgPool, +) -> Result<(), eyre::Report> { + let env = create_test_env(pool).await; + let tenant_prefix = "192.0.2.0/25".parse::()?; + let (tenant_vpc_id, _) = common::api_fixtures::vpc::create_flat_vpc( + &env, + "legacy-admin-containment".to_string(), + None, + ) + .await; + env.api + .create_vpc_prefix(tonic::Request::new(rpc::forge::VpcPrefixCreationRequest { + id: None, + prefix: String::new(), + vpc_id: Some(tenant_vpc_id), + site_prefix_id: None, + config: Some(rpc::forge::VpcPrefixConfig { + prefix: tenant_prefix.to_string(), + }), + metadata: Some(rpc::forge::Metadata { + name: "legacy admin containment".to_string(), + description: "startup compatibility regression".to_string(), + labels: vec![], + }), + })) + .await?; + + let mut txn = env.pool.begin().await?; + let admin_segments = db::network_segment::admin(&mut txn).await?; + assert!(admin_segments.iter().any(|segment| { + segment.config.vpc_id.is_none() + && segment.prefixes.iter().any(|prefix| { + prefix.prefix.prefix() < tenant_prefix.prefix() + && prefix.prefix.contains(tenant_prefix.network()) + }) + })); + txn.rollback().await?; + + db_init::create_admin_vpc(&env.pool, &env.config, Some(10000)).await?; + + let mut txn = env.pool.begin().await?; + let admin_vpc = db::vpc::find_by_vni(&mut txn, 10000) + .await? + .pop() + .expect("admin VPC is created"); + for admin_segment in db::network_segment::admin(&mut txn).await? { + assert_eq!(admin_segment.config.vpc_id, Some(admin_vpc.id)); + } + + Ok(()) +} + +#[crate::sqlx_test] +async fn create_admin_vpc_validates_before_committing( + pool: sqlx::PgPool, +) -> Result<(), eyre::Report> { + let env = create_test_env(pool).await; + let prefix = "198.51.100.0/24".parse::()?; + let version = ConfigVersion::initial(); + let vpc_ids = [VpcId::new(), VpcId::new()]; + let mut setup = env.pool.begin().await?; + + // This seam cannot create tenant-prefix reuse itself. Reconstruct the + // post-exclusion database state to prove its mutation still validates + // before commit when an unsafe exact cross-tenant duplicate is retained. + sqlx::query( + "ALTER TABLE network_vpc_prefixes \ + DROP CONSTRAINT network_vpc_prefixes_globally_unique", + ) + .execute(setup.as_mut()) + .await?; + for (index, (vpc_id, organization_id)) in vpc_ids + .into_iter() + .zip(["routing-safety-a", "routing-safety-b"]) + .enumerate() + { + sqlx::query( + "INSERT INTO tenants (organization_id, organization_name, version) \ + VALUES ($1, $1, $2)", + ) + .bind(organization_id) + .bind(version) + .execute(setup.as_mut()) + .await?; + sqlx::query( + "INSERT INTO vpcs \ + (id, name, organization_id, version, network_virtualization_type, vni, status) \ + VALUES ($1, $2, $3, $4, 'fnn', $5, jsonb_build_object('vni', $5))", + ) + .bind(vpc_id) + .bind(format!("routing-safety-{index}")) + .bind(organization_id) + .bind(version) + .bind(11001 + index as i32) + .execute(setup.as_mut()) + .await?; + sqlx::query( + "INSERT INTO network_vpc_prefixes (id, prefix, name, vpc_id) \ + VALUES ($1, $2, $3, $4)", + ) + .bind(VpcPrefixId::new()) + .bind(prefix) + .bind(format!("unsafe-duplicate-{index}")) + .bind(vpc_id) + .execute(setup.as_mut()) + .await?; + } + setup.commit().await?; + + let vni = 10000; + let error = db_init::create_admin_vpc(&env.pool, &env.config, Some(vni)) + .await + .expect_err("admin VPC reconciliation must reject unsafe retained tenant reuse"); + assert!(matches!(error, crate::CarbideError::FailedPrecondition(_))); + + // Validation runs in the mutation transaction, so neither the new admin + // VPC nor its segment attachments survive the failed reconciliation. + assert!(db::vpc::find_by_name(&env.pool, "admin").await?.is_empty()); + let mut txn = env.pool.begin().await?; + assert!(db::vpc::find_by_vni(&mut txn, vni as i32).await?.is_empty()); + for admin_segment in db::network_segment::admin(&mut txn).await? { + assert!(admin_segment.config.vpc_id.is_none()); + } + + Ok(()) +} + #[crate::sqlx_test] async fn create_admin_vpc_updates_existing_admin_vpc_vni( pool: sqlx::PgPool, @@ -1619,7 +1748,7 @@ async fn create_admin_vpc_updates_existing_admin_vpc_vni( let updated_vni = 10001; // Create the initial admin VPC and verify the admin segments attach to it. - db_init::create_admin_vpc(&env.pool, Some(initial_vni)).await?; + db_init::create_admin_vpc(&env.pool, &env.config, Some(initial_vni)).await?; let mut txn = env.pool.begin().await?; let mut initial_admin_vpcs = db::vpc::find_by_vni(&mut txn, initial_vni as i32).await?; assert_eq!(initial_admin_vpcs.len(), 1); @@ -1630,7 +1759,7 @@ async fn create_admin_vpc_updates_existing_admin_vpc_vni( txn.commit().await?; // Change the configured VNI and run startup reconciliation again. - db_init::create_admin_vpc(&env.pool, Some(updated_vni)).await?; + db_init::create_admin_vpc(&env.pool, &env.config, Some(updated_vni)).await?; // Fetch from the DB to verify the existing admin VPC was updated in place. let mut txn = env.pool.begin().await?; @@ -1688,7 +1817,7 @@ async fn create_admin_vpc_rejects_existing_tenant_vpc_vni( txn.commit().await?; // Seeding the admin VPC must fail instead of adopting the tenant VPC. - let err = db_init::create_admin_vpc(&env.pool, Some(vni)) + let err = db_init::create_admin_vpc(&env.pool, &env.config, Some(vni)) .await .expect_err("admin VPC seeding should reject an already-used tenant VNI"); assert!( diff --git a/crates/api-core/src/tests/vpc_prefix.rs b/crates/api-core/src/tests/vpc_prefix.rs index 81ea45213b..6d69a16746 100644 --- a/crates/api-core/src/tests/vpc_prefix.rs +++ b/crates/api-core/src/tests/vpc_prefix.rs @@ -889,9 +889,13 @@ async fn test_ipv6_vpc_prefix_linknet_counters_cap_large_prefix( Ok(()) } -#[crate::sqlx_test] -async fn test_overlapping_vpc_prefixes(pool: PgPool) -> Result<(), Box> { - let env = create_test_env(pool).await; +async fn assert_same_vpc_overlap_is_sanitized( + pool: PgPool, + tenant_prefix_overlap_enabled: bool, +) -> Result<(), Box> { + let mut config = api_fixtures::get_config(); + config.tenant_prefix_overlap_enabled = tenant_prefix_overlap_enabled; + let env = create_test_env_with_overrides(pool, TestEnvOverrides::with_config(config)).await; env.create_vpc_and_tenant_segment().await; let vpc_id = get_vpc_fixture_id(&env).await; @@ -937,12 +941,34 @@ async fn test_overlapping_vpc_prefixes(pool: PgPool) -> Result<(), Box Result<(), Box> { + assert_same_vpc_overlap_is_sanitized(pool, true).await +} + +#[crate::sqlx_test] +async fn same_vpc_overlap_is_invalid_argument_with_gate_disabled( + pool: PgPool, +) -> Result<(), Box> { + assert_same_vpc_overlap_is_sanitized(pool, false).await +} + #[crate::sqlx_test] async fn test_reject_create_with_invalid_metadata( pool: PgPool, diff --git a/crates/api-db/src/lib.rs b/crates/api-db/src/lib.rs index 88bf446c75..f268617c09 100644 --- a/crates/api-db/src/lib.rs +++ b/crates/api-db/src/lib.rs @@ -86,6 +86,7 @@ pub mod redfish_actions; pub mod resource_pool; pub mod retained_boot_interface; pub mod route_servers; +pub mod routing_safety; pub mod secrets; pub mod site_exploration_report; pub mod site_explorer_run_status; diff --git a/crates/api-db/src/routing_safety.rs b/crates/api-db/src/routing_safety.rs new file mode 100644 index 0000000000..5cf323b74d --- /dev/null +++ b/crates/api-db/src/routing_safety.rs @@ -0,0 +1,387 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +//! Transactional database view of the site routing-safety graph. +//! +//! Routing writers acquire one site-local transaction lock before any +//! resource-specific row lock. Address rows are loaded first; peerings, +//! security groups, and retained instance transitions are loaded only when +//! exact tenant address reuse makes policy analysis necessary. Soft-deleted +//! rows remain in the snapshots while controllers may still route or drain +//! them, so admission cannot forget an in-flight path. + +use model::DeletedFilter; +use model::instance::config::network::{InstanceNetworkConfig, InstanceNetworkConfigUpdate}; +use model::network_security_group::NetworkSecurityGroup; +use model::network_segment::{NetworkSegment, NetworkSegmentSearchConfig}; +use model::site_prefix::{SitePrefix, SitePrefixSearchFilter}; +use model::vpc::{Vpc, VpcPeering}; +use model::vpc_prefix::VpcPrefix; +use sqlx::PgConnection; + +use crate::{DatabaseError, DatabaseResult, ObjectColumnFilter}; + +const SITE_ROUTING_SAFETY_LOCK: &str = "site-routing-safety"; + +/// The address inventory needed to decide whether policy paths can possibly +/// matter for the current mutation. +#[derive(Debug)] +pub struct RoutingAddressSnapshot { + /// Every VPC row, including soft-deleted parents retained by routed children. + pub vpcs: Vec, + /// SitePrefix roots used to prove tenant ownership and routing scope. + pub site_prefixes: Vec, + /// VPC prefixes, including soft-deleted rows that have not finished draining. + pub vpc_prefixes: Vec, + /// Segments and their direct prefixes, including soft-deleted retained rows. + pub network_segments: Vec, +} + +/// A routing-only instance projection. Retained current and pending old/new +/// network configs stay present without hydrating OS and unrelated status. +#[derive(Debug, sqlx::FromRow)] +pub struct RoutingInstance { + /// Instance identity retained for operator-only failure diagnostics. + pub id: carbide_uuid::instance::InstanceId, + /// Current network attachment set. + pub network_config: sqlx::types::Json, + /// Pending old/new union that remains routable during an update. + pub update_network_config_request: Option>, + /// Instance-level NSG that can add reachability to any retained path. + pub network_security_group_id: + Option, +} + +/// Policy and retained-path rows loaded only after exact tenant reuse is present. +#[derive(Debug)] +pub struct RoutingPolicySnapshot { + /// Every persisted peering edge used by the DPU routing renderer. + pub peerings: Vec, + /// All NSGs, including soft-deleted groups retained by a routing attachment. + pub network_security_groups: Vec, + /// Routing-only current and pending instance projections, including soft-deleted rows. + pub instances: Vec, +} + +/// `lock_site_mutation` serializes site-wide routing graph mutations for the +/// duration of the caller's transaction. +/// +/// This must be the transaction's first database lock. No short lock timeout is +/// applied: losing a routing update to ordinary contention would weaken atomic +/// admission. The wait belongs to the caller's request or controller-task +/// lifecycle; canceling that work rolls back the transaction, and commit or +/// rollback releases the lock normally. +pub async fn lock_site_mutation(txn: &mut PgConnection) -> DatabaseResult<()> { + let query = "SELECT pg_advisory_xact_lock(hashtextextended($1, 0))"; + sqlx::query(query) + .bind(SITE_ROUTING_SAFETY_LOCK) + .execute(txn) + .await + .map(|_| ()) + .map_err(|error| DatabaseError::query(query, error)) +} + +/// `load_addresses` reads the authoritative address inventory after the site +/// mutation lock has been acquired. +pub async fn load_addresses(txn: &mut PgConnection) -> DatabaseResult { + // VPC deletion is soft. Retain the parent while a soft-deleted segment or + // instance transition can still refer to it. + let vpc_query = "SELECT * FROM vpcs ORDER BY id"; + let vpcs = sqlx::query_as(vpc_query) + .fetch_all(&mut *txn) + .await + .map_err(|error| DatabaseError::query(vpc_query, error))?; + + let site_prefix_ids = + crate::site_prefix::find_ids(&mut *txn, SitePrefixSearchFilter::default()).await?; + let site_prefixes = crate::site_prefix::find_by_ids(&mut *txn, &site_prefix_ids).await?; + + let vpc_prefixes = crate::vpc_prefix::get_by_id( + &mut *txn, + ObjectColumnFilter::All::, + DeletedFilter::Include, + ) + .await?; + + // `find_by` deliberately has no deleted predicate; the segment controller + // keeps a soft-deleted segment routed until final deletion. + let network_segments = crate::network_segment::find_by( + &mut *txn, + ObjectColumnFilter::All::, + NetworkSegmentSearchConfig::default(), + ) + .await?; + + Ok(RoutingAddressSnapshot { + vpcs, + site_prefixes, + vpc_prefixes, + network_segments, + }) +} + +/// `load_policy_paths` reads policy and retained paths after the site mutation +/// lock is held. +/// +/// All peerings are included. Soft-deleted security groups and instances remain +/// included so draining resources continue to constrain admission. +pub async fn load_policy_paths(txn: &mut PgConnection) -> DatabaseResult { + let peering_ids = crate::vpc_peering::find_ids(&mut *txn, None).await?; + let peerings = crate::vpc_peering::find_by_ids(&mut *txn, peering_ids).await?; + + // An NSG can be soft-deleted after its last active attachment disappears, + // while a retained VPC or instance still references that policy during drain. + let network_security_group_query = "SELECT * FROM network_security_groups ORDER BY id"; + let network_security_groups = sqlx::query_as(network_security_group_query) + .fetch_all(&mut *txn) + .await + .map_err(|error| DatabaseError::query(network_security_group_query, error))?; + + // Release preserves these routing columns until hard delete. Avoid the + // full instance/OS snapshot on every graph mutation. + let instance_query = "SELECT id, network_config, update_network_config_request, \ + network_security_group_id FROM instances ORDER BY id"; + let instances = sqlx::query_as(instance_query) + .fetch_all(&mut *txn) + .await + .map_err(|error| DatabaseError::query(instance_query, error))?; + + Ok(RoutingPolicySnapshot { + peerings, + network_security_groups, + instances, + }) +} + +#[cfg(test)] +mod tests { + use std::time::Duration; + + use carbide_uuid::machine::{MachineId, MachineIdSource, MachineType}; + use model::instance::config::network::{ + InstanceNetworkAutoConfig, InstanceNetworkConfig, InstanceNetworkConfigUpdate, + }; + use model::network_prefix::NewNetworkPrefix; + use model::network_segment::{ + AllocationStrategy, NetworkSegmentControllerState, NetworkSegmentType, NewNetworkSegment, + }; + use sqlx::PgPool; + + use super::*; + + #[crate::sqlx_test] + async fn site_mutation_lock_serializes_transactions( + pool: PgPool, + ) -> Result<(), Box> { + let mut first = pool.begin().await?; + let first_pid: i32 = sqlx::query_scalar("SELECT pg_backend_pid()") + .fetch_one(first.as_mut()) + .await?; + lock_site_mutation(first.as_mut()).await?; + + let (pid_sender, pid_receiver) = tokio::sync::oneshot::channel(); + let second_pool = pool.clone(); + let mut second = tokio::spawn(async move { + let mut txn = second_pool + .begin() + .await + .map_err(|error| error.to_string())?; + let pid: i32 = sqlx::query_scalar("SELECT pg_backend_pid()") + .fetch_one(txn.as_mut()) + .await + .map_err(|error| error.to_string())?; + pid_sender + .send(pid) + .map_err(|_| "could not report second transaction pid".to_string())?; + lock_site_mutation(txn.as_mut()) + .await + .map_err(|error| error.to_string())?; + txn.commit().await.map_err(|error| error.to_string()) + }); + + let second_pid = pid_receiver.await?; + tokio::time::timeout(Duration::from_secs(5), async { + loop { + let blocked_by_first: bool = + sqlx::query_scalar("SELECT $1 = ANY(pg_blocking_pids($2))") + .bind(first_pid) + .bind(second_pid) + .fetch_one(&pool) + .await?; + if blocked_by_first { + return Ok::<(), sqlx::Error>(()); + } + tokio::task::yield_now().await; + } + }) + .await + .map_err(|_| std::io::Error::other("second transaction did not wait on site lock"))??; + + first.commit().await?; + tokio::time::timeout(Duration::from_secs(5), &mut second) + .await + .map_err(|_| std::io::Error::other("site lock remained blocked after commit"))? + .map_err(|error| std::io::Error::other(error.to_string()))? + .map_err(std::io::Error::other)?; + Ok(()) + } + + #[crate::sqlx_test] + async fn snapshot_retains_soft_deleted_routing_parents_and_children( + pool: PgPool, + ) -> Result<(), Box> { + let vpc_id = carbide_uuid::vpc::VpcId::new(); + let old_vpc_id = carbide_uuid::vpc::VpcId::new(); + let new_vpc_id = carbide_uuid::vpc::VpcId::new(); + let network_security_group_id: carbide_uuid::network_security_group::NetworkSecurityGroupId = + "retained-routing-policy".parse()?; + let machine_id = MachineId::new( + MachineIdSource::ProductBoardChassisSerial, + [0x39; 32], + MachineType::Host, + ); + let mut setup = pool.begin().await?; + sqlx::query( + "INSERT INTO vpcs (id, name, organization_id, version) \ + VALUES ($1, $2, $3, $4)", + ) + .bind(vpc_id) + .bind("retained-routing-parent") + .bind("tenant-a") + .bind(config_version::ConfigVersion::initial()) + .execute(setup.as_mut()) + .await?; + sqlx::query( + "INSERT INTO tenants (organization_id, organization_name, version) \ + VALUES ($1, $2, $3)", + ) + .bind("tenant-a") + .bind("Tenant A") + .bind(config_version::ConfigVersion::initial()) + .execute(setup.as_mut()) + .await?; + sqlx::query("INSERT INTO machines (id, dpf) VALUES ($1, '{}'::jsonb)") + .bind(machine_id) + .execute(setup.as_mut()) + .await?; + let current_network = InstanceNetworkConfig { + interfaces: vec![], + auto_config: Some(InstanceNetworkAutoConfig { vpc_id }), + }; + let update = InstanceNetworkConfigUpdate { + old_config: InstanceNetworkConfig { + interfaces: vec![], + auto_config: Some(InstanceNetworkAutoConfig { vpc_id: old_vpc_id }), + }, + new_config: InstanceNetworkConfig { + interfaces: vec![], + auto_config: Some(InstanceNetworkAutoConfig { vpc_id: new_vpc_id }), + }, + }; + let instance_id: carbide_uuid::instance::InstanceId = sqlx::query_scalar( + "INSERT INTO instances \ + (machine_id, tenant_org, os_ipxe_script, network_config, \ + update_network_config_request, nvlink_config, deleted) \ + VALUES ($1, $2, $3, $4, $5, '{\"gpu_configs\": []}'::jsonb, NOW()) RETURNING id", + ) + .bind(machine_id) + .bind("tenant-a") + .bind("#!ipxe retained-routing-instance") + .bind(sqlx::types::Json(¤t_network)) + .bind(sqlx::types::Json(&update)) + .fetch_one(setup.as_mut()) + .await?; + sqlx::query( + "INSERT INTO network_security_groups (id, tenant_organization_id, name, deleted) \ + VALUES ($1, $2, $3, NOW())", + ) + .bind(&network_security_group_id) + .bind("tenant-a") + .bind("retained-routing-policy") + .execute(setup.as_mut()) + .await?; + let segment = crate::network_segment::persist( + NewNetworkSegment { + id: carbide_uuid::network::NetworkSegmentId::new(), + name: "retained-routing-child".to_string(), + subdomain_id: None, + vpc_id: Some(vpc_id), + mtu: 1500, + prefixes: vec![NewNetworkPrefix { + prefix: "192.0.2.0/24".parse()?, + gateway: None, + dhcpv6_link_address: None, + num_reserved: 0, + }], + vlan_id: None, + vni: None, + segment_type: NetworkSegmentType::Tenant, + can_stretch: None, + allocation_strategy: AllocationStrategy::Dynamic, + infer_slaac_eui64_addresses: false, + }, + setup.as_mut(), + NetworkSegmentControllerState::Ready, + ) + .await?; + crate::network_segment::mark_as_deleted(&segment, setup.as_mut()).await?; + sqlx::query("UPDATE vpcs SET deleted=NOW() WHERE id=$1") + .bind(vpc_id) + .execute(setup.as_mut()) + .await?; + setup.commit().await?; + + let mut txn = pool.begin().await?; + lock_site_mutation(txn.as_mut()).await?; + let addresses = load_addresses(txn.as_mut()).await?; + let policy = load_policy_paths(txn.as_mut()).await?; + assert!( + addresses.vpcs.iter().any(|vpc| vpc.id == vpc_id), + "soft-deleted VPC must remain available to resolve retained routing children" + ); + let retained_segment = addresses + .network_segments + .iter() + .find(|candidate| candidate.id == segment.id) + .expect("soft-deleted segment must remain in the routing snapshot"); + assert_eq!(retained_segment.config.vpc_id, Some(vpc_id)); + assert_eq!(retained_segment.prefixes.len(), 1); + assert!(retained_segment.prefixes[0].vpc_prefix_id.is_none()); + assert!( + policy + .network_security_groups + .iter() + .any(|group| group.id == network_security_group_id), + "soft-deleted NSG must remain available for retained routing attachments" + ); + let retained_instance = policy + .instances + .iter() + .find(|instance| instance.id == instance_id) + .expect("soft-deleted instance must remain in the routing snapshot"); + assert_eq!(retained_instance.network_config.0, current_network); + assert_eq!( + retained_instance + .update_network_config_request + .as_ref() + .map(|request| &request.0), + Some(&update) + ); + txn.commit().await?; + Ok(()) + } +} diff --git a/crates/api-db/src/vpc_prefix.rs b/crates/api-db/src/vpc_prefix.rs index cc8ed8379a..a1591f1ba0 100644 --- a/crates/api-db/src/vpc_prefix.rs +++ b/crates/api-db/src/vpc_prefix.rs @@ -423,20 +423,6 @@ pub async fn persist( Ok(vpc_prefix) } -/// Checks for existing or deleting VPC prefixes using any of the address space. -pub async fn probe( - network: IpNetwork, - txn: &mut PgConnection, -) -> Result, DatabaseError> { - // Include soft-deleted rows because the global exclusion constraint still reserves them. - let query = "SELECT * FROM network_vpc_prefixes WHERE prefix && $1"; - sqlx::query_as(query) - .bind(network) - .fetch_all(txn) - .await - .map_err(|e| DatabaseError::query(query, e)) -} - // Given a new VPC prefix which has been not been persisted yet, find the // network segment prefixes that overlap with it, along with the VPC ID each // one is associated with. The caller should use this information to reject diff --git a/crates/machine-controller/src/handler.rs b/crates/machine-controller/src/handler.rs index d0f8a5cac4..df70adaddf 100644 --- a/crates/machine-controller/src/handler.rs +++ b/crates/machine-controller/src/handler.rs @@ -8456,6 +8456,8 @@ impl StateHandler for InstanceStateHandler { // Delete from database now. Once done, reboot and move to next state. let mut txn = ctx.services.db_pool.begin().await?; + // Serialize the graph removal before resource-specific locks. + db::routing_safety::lock_site_mutation(&mut txn).await?; db::instance::delete(instance.id, &mut txn) .await .map_err(|err| StateHandlerError::GenericError(err.into()))?; @@ -8831,6 +8833,9 @@ async fn handle_instance_network_config_update_request( // Update requested network config and increment version. let mut txn = ctx.services.db_pool.begin().await?; + // The API admitted both retained configs before storing the + // request; serialize their promotion with routing graph writers. + db::routing_safety::lock_site_mutation(&mut txn).await?; db::instance::update_network_config( txn.as_mut(), instance.id, @@ -8872,6 +8877,8 @@ async fn handle_instance_network_config_update_request( } NetworkConfigUpdateState::ReleaseOldResources => { let mut txn = ctx.services.db_pool.begin().await?; + // Serialize the graph contraction before pool and row locks. + db::routing_safety::lock_site_mutation(&mut txn).await?; // Identify all the resources which have to be released. // Release Ips. // Release segments. diff --git a/crates/machine-controller/tests/integration/rack_firmware_upgrade.rs b/crates/machine-controller/tests/integration/rack_firmware_upgrade.rs index 83c7776a3e..ff394abca6 100644 --- a/crates/machine-controller/tests/integration/rack_firmware_upgrade.rs +++ b/crates/machine-controller/tests/integration/rack_firmware_upgrade.rs @@ -227,6 +227,7 @@ async fn assigned_host_returns_to_assigned_ready_on_completion(pool: PgPool) { .await .unwrap(); assert_eq!(attached_instance, Some(instance_id)); + txn.rollback().await.unwrap(); } #[sqlx_test] diff --git a/crates/network-segment-controller/src/handler.rs b/crates/network-segment-controller/src/handler.rs index 6cd4f02c0f..4055391ef0 100644 --- a/crates/network-segment-controller/src/handler.rs +++ b/crates/network-segment-controller/src/handler.rs @@ -171,6 +171,8 @@ impl StateHandler for NetworkSegmentStateHandler { } NetworkSegmentDeletionState::DBDelete => { let mut txn = ctx.services.db_pool.begin().await?; + // Serialize the graph removal before pool and row locks. + db::routing_safety::lock_site_mutation(&mut txn).await?; if let Some(vni) = state.status.vni.take() { db::resource_pool::release(&self.pool_vni, &mut txn, vni).await?; } diff --git a/crates/vpc-prefix-controller/src/handler.rs b/crates/vpc-prefix-controller/src/handler.rs index 9cf0d741cf..ea670f4c5a 100644 --- a/crates/vpc-prefix-controller/src/handler.rs +++ b/crates/vpc-prefix-controller/src/handler.rs @@ -123,6 +123,8 @@ impl StateHandler for VpcPrefixStateHandler { } VpcPrefixDeletionState::DBDelete => { let mut txn = ctx.services.db_pool.begin().await?; + // Serialize the graph removal before resource-specific locks. + db::routing_safety::lock_site_mutation(&mut txn).await?; // Remove the prefix row after all network_prefix references are gone. tracing::info!(