Skip to content

Support dynamic client certificate resolution (ResolvesClientCert) - #1340

Open
brucearctor wants to merge 7 commits into
temporalio:mainfrom
brucearctor:feat/dynamic-client-certs
Open

Support dynamic client certificate resolution (ResolvesClientCert)#1340
brucearctor wants to merge 7 commits into
temporalio:mainfrom
brucearctor:feat/dynamic-client-certs

Conversation

@brucearctor

@brucearctor brucearctor commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

Support dynamic client certificate resolution (ResolvesClientCert)

What

Add a client_cert_resolver field to TlsOptions that accepts an Arc<dyn ResolvesClientCert> for dynamic, per-handshake client certificate selection. This enables transparent mTLS certificate rotation without requiring a process restart — useful for short-lived certificates managed by Vault agents, cert-manager sidecars, or HSM-backed signers.

Closes #1338

Why

The existing client_tls_options requires static (cert, key) bytes at connection time. Users with rotating certificates (e.g., Vault-issued certs with 24h TTLs) must restart the entire Temporal client to pick up new material. The Go SDK solves this with tls.Config.GetClientCertificate; this PR brings equivalent functionality to the Rust SDK.

How

Since tonic 0.14.6 does not expose rustls::ClientConfig::with_client_cert_resolver(), the implementation bypasses tonic's TLS layer when a resolver is present:

  1. build_custom_rustls_config() — manually constructs a rustls::ClientConfig with the user's ResolvesClientCert, replicating tonic's security defaults (protocol versions, ALPN, native/custom CA roots)
  2. DynamicTlsConnector — a tower::Service<Uri> that wraps TCP with TLS using tokio_rustls::TlsConnector, including connect timeouts and tracing
  3. Endpoint::connect_with_connector() — used instead of connect() to wire the custom connector into tonic's channel

The resolver fires on each new TLS handshake (reconnections), not per-RPC over an existing HTTP/2 connection.

Key design decisions

  • Mutually exclusive with static certs — setting both client_tls_options and client_cert_resolver returns InvalidConfig
  • Re-exportsResolvesClientCert, CertifiedKey, and SignatureScheme are re-exported at the crate root so users don't need to depend on tokio-rustls directly
  • DNS load balancing — returns a clear error (not yet supported with dynamic certs due to balance_channel API constraints)
  • Proxy — returns a clear error (not yet supported; would require composing connectors)
  • C bridge — hardcodes client_cert_resolver: None (dynamic resolution from C callers needs a callback design in a follow-up)

Changes

File Change
crates/client/src/options_structs.rs Add client_cert_resolver field, update Debug impl
crates/client/src/lib.rs TlsConfigResult enum, add_tls_to_channel branching, build_custom_rustls_config, DynamicTlsConnector, proxy/DNS validation, tracing
crates/client/src/dns.rs Handle TlsConfigResult, reject resolver + DNS LB
crates/client/src/envconfig.rs Add client_cert_resolver: None
crates/client/Cargo.toml Add rustls-native-certs dep
crates/sdk-core-c-bridge/src/client.rs Add client_cert_resolver: None
crates/sdk-core/tests/common/mod.rs Update test struct literals
CHANGELOG.md Add entry under [Unreleased] > Added

Tests (15 new, 134 total pass)

  • Custom connector returned when resolver is set (explicit domain + inferred domain)
  • Mutual exclusion: static + dynamic certs → error
  • Resolver + custom ServerCertVerifier combination
  • Resolver + custom CA certificate
  • build_custom_rustls_config with/without resolver/verifier
  • Debug output correctness for TlsOptions with resolver
  • DynamicTlsConnector is Clone + Debug
  • Default TlsOptions has no resolver
  • No-TLS passthrough returns Standard
  • IP host fallback for domain extraction
  • Re-exports compile (CertifiedKey, SignatureScheme)

Future work

@chris-olszewski

Copy link
Copy Markdown
Member

Will be giving this a review very soon, thanks for your patience.

@brucearctor

Copy link
Copy Markdown
Contributor Author

i'm in no rush

@chris-olszewski chris-olszewski left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall I think this design makes sense to me. The largest thing for me is making sure we feature gate it.

Comment thread crates/client/src/options_structs.rs Outdated
Comment thread crates/client/Cargo.toml Outdated
Comment thread crates/client/src/lib.rs Outdated
@brucearctor

Copy link
Copy Markdown
Contributor Author

@chris-olszewski - i think addressed?

@chris-olszewski
chris-olszewski self-requested a review July 29, 2026 20:25

@chris-olszewski chris-olszewski left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Only a few questions on some of the new tests and a suggestion that we mark this variant as experimental. Thanks for your patience on the review process.

Comment thread crates/client/src/lib.rs Outdated
Comment thread crates/client/src/lib.rs Outdated
Comment thread crates/client/src/lib.rs
@brucearctor
brucearctor force-pushed the feat/dynamic-client-certs branch from 780b89d to c44c0d5 Compare August 3, 2026 22:32
@brucearctor

brucearctor commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

@chris-olszewski — rebased onto main and addressed your outstanding comments.

Summary:

Removed the re_exports_are_accessible test — you were right that use super::* imports parent imports, not just public exports, so it wasn't actually testing the re-export surface. The re-exports are exercised implicitly by the other dynamic-tls tests.
The dynamic_tls_connector_is_clone_and_debug removal and "Experimental API" doc comment were already in place from earlier.

Merge conflict:

Rebased onto current main. The only conflict was integrating #1422's connect_timeout — it slots in between Endpoint::from_shared() and add_tls_to_channel() in the dynamic-tls path.

IPv6 fixes (new): While doing a deeper review of the connector code, caught two IPv6 bugs:

  • DynamicTlsConnector::call used format!("{host}:{port}") for TcpStream::connect, which produces ambiguous addresses for IPv6 (e.g. ::1:443). Switched to the (host, port) tuple form of ToSocketAddrs.
  • SNI domain extraction from uri.host() returns brackets for IPv6 literals ([::1]), which ServerName::try_from rejects. Now strips brackets before conversion.

All 166 tests pass with --features dynamic-tls, 154 pass on default features.

brucearctor and others added 7 commits August 4, 2026 17:24
…ClientCert)

Add a new client_cert_resolver field to TlsOptions that accepts an
Arc<dyn ResolvesClientCert> for dynamic per-handshake client certificate
resolution. This enables transparent mTLS certificate rotation without
requiring a process restart -- useful for short-lived certificates
managed by Vault agents, sidecars, or HSMs.

Key changes:
- Add client_cert_resolver: Option<Arc<dyn ResolvesClientCert>> to
  TlsOptions (mutually exclusive with static client_tls_options)
- Re-export ResolvesClientCert from the crate root for ergonomic use
- When a resolver is set, bypass tonic's static Identity path and build
  a rustls::ClientConfig manually with with_client_cert_resolver()
- Introduce DynamicTlsConnector (tower::Service<Uri>) that performs TLS
  via tokio_rustls::TlsConnector with the custom config
- Use Endpoint::connect_with_connector() to wire it into tonic's channel
- Add validation: error when both static and dynamic client certs are set
- DNS load balancing returns a clear error for now (not yet supported
  with dynamic cert resolution)
- Update C bridge and envconfig with client_cert_resolver: None

Tests (13 new):
- Mutual exclusion validation (static + dynamic = error)
- CustomConnector result with explicit and inferred domain
- Resolver + custom ServerCertVerifier combination
- Resolver + custom CA certificate
- build_custom_rustls_config with/without resolver/verifier
- Debug output for TlsOptions with resolver
- DynamicTlsConnector Clone requirement
- Default TlsOptions has no resolver
- No-TLS passthrough still returns Standard

Closes: temporalio#1338
Fixes from 4-reviewer deep code review (Temporal, Rust, Systems, Security):

Must Fix:
- Proxy + cert resolver: add validation to prevent silent discard of
  resolver when http_connect_proxy is also set (was silently ignoring
  the dynamic cert resolver)
- Empty SNI domain: fail early with clear error instead of producing
  an empty string that causes a confusing late error from ServerName
- Localhost fallback: DynamicTlsConnector no longer falls back to
  "localhost" when URI has no host — returns a clear error instead
- Connect timeout: wrap TCP connect in 30s timeout to prevent hanging
  on unreachable hosts (tonic's built-in connector handles this but
  custom connectors must do it themselves)
- CA cert parsing: check roots.is_empty() after add_parsable_certificates
  to catch cases where all provided CA certs are malformed
- Dead code: remove unused CountingCertResolver test helper

Should Fix:
- Tracing: add debug! logging to DynamicTlsConnector for TCP connect
  and TLS handshake completion
- Re-exports: add CertifiedKey and SignatureScheme re-exports so users
  can implement ResolvesClientCert without adding tokio-rustls directly
- Debug impl: add manual Debug for DynamicTlsConnector showing domain
- Nodelay comment: document why set_nodelay(true) is set
Co-authored-by: Chris Olszewski <chrisdolszewski@gmail.com>
Co-authored-by: Chris Olszewski <chrisdolszewski@gmail.com>
The test used `use super::*` which imports parent's imports, not just
public re-exports, so it didn't actually verify the public API surface.
The re-exports (ResolvesClientCert, CertifiedKey, SignatureScheme) are
implicitly exercised by the other dynamic-tls tests.
…raction

- Use (host, port) tuple for TcpStream::connect instead of format string,
  which breaks for IPv6 addresses (e.g. '::1:443' is ambiguous)
- Strip brackets from IPv6 URI hosts before ServerName::try_from, since
  uri.host() returns '[::1]' but rustls expects raw '::1'
@brucearctor
brucearctor force-pushed the feat/dynamic-client-certs branch from c44c0d5 to c277cf1 Compare August 5, 2026 00:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature Request] Support dynamic/reloadable client certificates (ResolvesClientCert) in TlsOptions

2 participants