Shared config.API_HOST_URL races across controllers: a cached SDK client can be bound to the wrong Infisical host permanently

[m-minasyan]

Summary

config.API_HOST_URL is a package-level global that three controllers write, unsynchronised, from their own reconcile goroutines. getResourceVariables reads it to build an SDK client and caches that client for the lifetime of the process, keyed only by the CR's UID. When a cluster uses more than one Infisical host - the common case being self-hosted plus Cloud - a resource whose client is built in the window where another controller owns the global is bound to the wrong host permanently. Every retry re-authenticates through the same cached client, so controller-runtime backoff never repairs it.

We hit this in production on v0.10.32. It is unchanged on v0.11.5 and on main.

The code path

The global, with no mutex anywhere near it:

• internal/config/config.go:58 - var API_HOST_URL string = GetDefaultHostAPI()

Three writers, each called from its own controller's Reconcile:

• internal/services/infisicalsecret/handler.go:35,37 (SetupAPIConfig)
• internal/services/infisicalpushsecret/handler.go:34,36 (SetupAPIConfig)
• internal/services/infisicaldynamicsecret/handler.go:38,40 (SetupAPIConfig)

Three readers that bake the value into a cached client:

• internal/services/infisicalpushsecret/reconciler.go:41 - SiteUrl: config.API_HOST_URL
• internal/services/infisicalsecret/reconciler.go:626
• internal/services/infisicaldynamicsecret/reconciler.go:121

In InfisicalPushSecretReconciler.Reconcile the sequence is handler.SetupAPIConfig(...) (writes the global) → handler.ReconcileInfisicalPushSecret(...) → getResourceVariables(...) (reads it back). Any other controller's goroutine can write the global in between. The three controllers are registered separately in cmd/main.go, so each has its own worker goroutine and they genuinely run concurrently even at the default MaxConcurrentReconciles: 1.

Because the client is stored in infisicalPushSecretResourceVariablesMap[string(UID)] and only ever removed by the UpdateFunc/DeleteFunc predicates, a client built with the wrong SiteUrl is reused until the CR changes or the process restarts.

Symptom

We run a one-way mirror: InfisicalSecret resources pull from Infisical Cloud (https://app.infisical.com/api) and InfisicalPushSecret resources push to a self-hosted instance, each with spec.hostAPI set correctly on the CR.

After an operator restart, one push resource began logging:

POST https://app.infisical.com/api/v1/auth/universal-auth/login
[status-code=401] [message="Invalid credentials"]

• Its spec.hostAPI is the in-cluster self-hosted Service, not Cloud.
• Its machine-identity credentials are valid - the identity exists on the self-hosted instance, not on Cloud, so Cloud correctly rejects them.
• It broke 41 seconds after the operator started and was still broken 3h21m later. It is not transient.
• A distinctive fingerprint: sibling calls in the same reconcile that read the global live (e.g. ExtractProjectIdFromSlug) name the self-hosted host in the very same log, while the cached SDK client keeps hitting Cloud. Two different hosts for one resource in one reconcile is the tell.

A second push resource had been caught by the identical race on an earlier restart.

Reproduction

1. Create at least one InfisicalSecret with spec.hostAPI: https://app.infisical.com/api and at least one InfisicalPushSecret with spec.hostAPI pointing at a self-hosted instance, with credentials valid only on their respective hosts. A handful of each makes it likelier to land.
2. Restart the operator.
3. Some fraction of the push resources authenticate against Cloud and 401 forever. Which ones differ per restart, which is what identifies it as a race rather than a misconfiguration.

Workaround

Changing any annotation on the affected CR makes UpdateFunc call rv.CancelCtx() and delete the map entry, so the next reconcile rebuilds the client:

kubectl -n <ns> annotate infisicalpushsecret <name> evict="$(date -u +%s)" --overwrite

Restarting the operator also clears it, but it is strictly worse - it drops every cached client and re-runs the race across all resources at once, which is how the poisoning happens in the first place.

Suggested fix

The host is per-resource state being passed through a process-global, so the fix is to stop doing that:

1. Have SetupAPIConfig return the resolved host instead of assigning to config.API_HOST_URL, and thread it through to getResourceVariables as a parameter. That removes the race entirely rather than narrowing it.
2. Key the cached client on (UID, siteURL), or store the siteURL alongside the client and rebuild when it differs. This is worth doing regardless: today, editing spec.hostAPI on a CR is picked up only because the generation change happens to evict the cache, and any future eviction-path change would silently reintroduce a stale host.
3. If the global has to stay for internal/api/api.go's legacy service-token calls, guard it with a mutex held across the write-and-read, so at least the window closes.

Related, and probably worth its own fix

infisicalPushSecretResourceVariablesMap (and its infisicalsecret / infisicaldynamicsecret equivalents) is a plain map written from the reconcile goroutine in getResourceVariables/updateResourceVariables and deleted from the event-source goroutine in the UpdateFunc/DeleteFunc predicates, with no lock. That is a concurrent map read and write, which Go can turn into a hard fatal error: concurrent map read and map write rather than a recoverable panic. We have not observed it, but the data race is there in the same code as the bug above.

Environment

• Operator v0.10.32 (chart secrets-operator 0.10.32); code re-read at v0.11.5 and main, both unchanged in this respect.
• Kubernetes v1.35.2.
• ~21 InfisicalPushSecret (self-hosted) and ~36 InfisicalSecret (Cloud) in one cluster, one operator replica, leader election on.

[Adisa-Shobi]

Would like to take this if it's still unclaimed. Read through the write/read sites you listed — agree the fix is threading the resolved host through SetupAPIConfig's return value instead of the config.API_HOST_URL global, and keying the cached client on (UID, siteURL) rather than UID alone. Happy to leave the unlocked map read/write as a separate follow-up rather than bundling it in. Let me know if this is wanted from an outside PR.