Support dynamic client certificate resolution (ResolvesClientCert) - #1340
Support dynamic client certificate resolution (ResolvesClientCert)#1340brucearctor wants to merge 7 commits into
Conversation
|
Will be giving this a review very soon, thanks for your patience. |
|
i'm in no rush |
chris-olszewski
left a comment
There was a problem hiding this comment.
Overall I think this design makes sense to me. The largest thing for me is making sure we feature gate it.
|
@chris-olszewski - i think addressed? |
chris-olszewski
left a comment
There was a problem hiding this comment.
Looks good to me. Only a few questions on some of the new tests and a suggestion that we mark this variant as experimental. Thanks for your patience on the review process.
780b89d to
c44c0d5
Compare
|
@chris-olszewski — rebased onto main and addressed your outstanding comments. Summary:Removed the re_exports_are_accessible test — you were right that use super::* imports parent imports, not just public exports, so it wasn't actually testing the re-export surface. The re-exports are exercised implicitly by the other dynamic-tls tests. Merge conflict:Rebased onto current main. The only conflict was integrating #1422's connect_timeout — it slots in between Endpoint::from_shared() and add_tls_to_channel() in the dynamic-tls path. IPv6 fixes (new): While doing a deeper review of the connector code, caught two IPv6 bugs:
All 166 tests pass with --features dynamic-tls, 154 pass on default features. |
…ClientCert) Add a new client_cert_resolver field to TlsOptions that accepts an Arc<dyn ResolvesClientCert> for dynamic per-handshake client certificate resolution. This enables transparent mTLS certificate rotation without requiring a process restart -- useful for short-lived certificates managed by Vault agents, sidecars, or HSMs. Key changes: - Add client_cert_resolver: Option<Arc<dyn ResolvesClientCert>> to TlsOptions (mutually exclusive with static client_tls_options) - Re-export ResolvesClientCert from the crate root for ergonomic use - When a resolver is set, bypass tonic's static Identity path and build a rustls::ClientConfig manually with with_client_cert_resolver() - Introduce DynamicTlsConnector (tower::Service<Uri>) that performs TLS via tokio_rustls::TlsConnector with the custom config - Use Endpoint::connect_with_connector() to wire it into tonic's channel - Add validation: error when both static and dynamic client certs are set - DNS load balancing returns a clear error for now (not yet supported with dynamic cert resolution) - Update C bridge and envconfig with client_cert_resolver: None Tests (13 new): - Mutual exclusion validation (static + dynamic = error) - CustomConnector result with explicit and inferred domain - Resolver + custom ServerCertVerifier combination - Resolver + custom CA certificate - build_custom_rustls_config with/without resolver/verifier - Debug output for TlsOptions with resolver - DynamicTlsConnector Clone requirement - Default TlsOptions has no resolver - No-TLS passthrough still returns Standard Closes: temporalio#1338
Fixes from 4-reviewer deep code review (Temporal, Rust, Systems, Security): Must Fix: - Proxy + cert resolver: add validation to prevent silent discard of resolver when http_connect_proxy is also set (was silently ignoring the dynamic cert resolver) - Empty SNI domain: fail early with clear error instead of producing an empty string that causes a confusing late error from ServerName - Localhost fallback: DynamicTlsConnector no longer falls back to "localhost" when URI has no host — returns a clear error instead - Connect timeout: wrap TCP connect in 30s timeout to prevent hanging on unreachable hosts (tonic's built-in connector handles this but custom connectors must do it themselves) - CA cert parsing: check roots.is_empty() after add_parsable_certificates to catch cases where all provided CA certs are malformed - Dead code: remove unused CountingCertResolver test helper Should Fix: - Tracing: add debug! logging to DynamicTlsConnector for TCP connect and TLS handshake completion - Re-exports: add CertifiedKey and SignatureScheme re-exports so users can implement ResolvesClientCert without adding tokio-rustls directly - Debug impl: add manual Debug for DynamicTlsConnector showing domain - Nodelay comment: document why set_nodelay(true) is set
Co-authored-by: Chris Olszewski <chrisdolszewski@gmail.com>
Co-authored-by: Chris Olszewski <chrisdolszewski@gmail.com>
The test used `use super::*` which imports parent's imports, not just public re-exports, so it didn't actually verify the public API surface. The re-exports (ResolvesClientCert, CertifiedKey, SignatureScheme) are implicitly exercised by the other dynamic-tls tests.
…raction - Use (host, port) tuple for TcpStream::connect instead of format string, which breaks for IPv6 addresses (e.g. '::1:443' is ambiguous) - Strip brackets from IPv6 URI hosts before ServerName::try_from, since uri.host() returns '[::1]' but rustls expects raw '::1'
c44c0d5 to
c277cf1
Compare
Support dynamic client certificate resolution (ResolvesClientCert)
What
Add a
client_cert_resolverfield toTlsOptionsthat accepts anArc<dyn ResolvesClientCert>for dynamic, per-handshake client certificate selection. This enables transparent mTLS certificate rotation without requiring a process restart — useful for short-lived certificates managed by Vault agents, cert-manager sidecars, or HSM-backed signers.Closes #1338
Why
The existing
client_tls_optionsrequires static(cert, key)bytes at connection time. Users with rotating certificates (e.g., Vault-issued certs with 24h TTLs) must restart the entire Temporal client to pick up new material. The Go SDK solves this withtls.Config.GetClientCertificate; this PR brings equivalent functionality to the Rust SDK.How
Since
tonic0.14.6 does not exposerustls::ClientConfig::with_client_cert_resolver(), the implementation bypasses tonic's TLS layer when a resolver is present:build_custom_rustls_config()— manually constructs arustls::ClientConfigwith the user'sResolvesClientCert, replicating tonic's security defaults (protocol versions, ALPN, native/custom CA roots)DynamicTlsConnector— atower::Service<Uri>that wraps TCP with TLS usingtokio_rustls::TlsConnector, including connect timeouts and tracingEndpoint::connect_with_connector()— used instead ofconnect()to wire the custom connector into tonic's channelThe resolver fires on each new TLS handshake (reconnections), not per-RPC over an existing HTTP/2 connection.
Key design decisions
client_tls_optionsandclient_cert_resolverreturnsInvalidConfigResolvesClientCert,CertifiedKey, andSignatureSchemeare re-exported at the crate root so users don't need to depend ontokio-rustlsdirectlybalance_channelAPI constraints)client_cert_resolver: None(dynamic resolution from C callers needs a callback design in a follow-up)Changes
crates/client/src/options_structs.rsclient_cert_resolverfield, updateDebugimplcrates/client/src/lib.rsTlsConfigResultenum,add_tls_to_channelbranching,build_custom_rustls_config,DynamicTlsConnector, proxy/DNS validation, tracingcrates/client/src/dns.rsTlsConfigResult, reject resolver + DNS LBcrates/client/src/envconfig.rsclient_cert_resolver: Nonecrates/client/Cargo.tomlrustls-native-certsdepcrates/sdk-core-c-bridge/src/client.rsclient_cert_resolver: Nonecrates/sdk-core/tests/common/mod.rsCHANGELOG.md[Unreleased] > AddedTests (15 new, 134 total pass)
ServerCertVerifiercombinationbuild_custom_rustls_configwith/without resolver/verifierTlsOptionswith resolverDynamicTlsConnectorisClone+DebugTlsOptionshas no resolverStandardCertifiedKey,SignatureScheme)Future work
FileWatchingCertResolverbatteries-included implementation — Provide batteries-included FileWatchingCertResolver for common mTLS rotation #1345