Replace JWT plugin catalog refresh hook with versioned OpenBao plugin lifecycle

[nvjmcnamee]

Description

OpenBao CVE remediation moved the server and bundled JWT secrets plugin forward. During upgrade testing, the JWT plugin catalog needed a tactical refresh so OpenBao would use the upgraded binary digest after the pod restarted.

That tactical repair is acceptable as a short-term fix, but the long-term model should not depend on a Helm hook rewriting an unversioned plugin catalog entry. The JWT plugin is a runtime dependency for signed JWT issuance. Its deployed version, configured mount version, running mount version, and rollback target should be explicit.

Today the JWT plugin lifecycle is anchored on an unversioned catalog entry:

• the OpenBao image carries vault-plugin-secrets-jwt;
• existing migrations enable mounts by plugin name without plugin_version;
• OpenBao stores the plugin catalog and mount metadata in its own storage;
• Kubernetes image/config rollback and OpenBao catalog rollback are separate state changes.

This makes upgrades fragile. If the binary at an unversioned command path changes without the catalog SHA being updated, OpenBao can reject or fail to load the plugin. If a mount is later pinned to a version, rolling back to an image that does not carry that versioned command path can leave the mount present but unusable.

Move the JWT plugin to an explicit versioned lifecycle:

1. Package the OpenBao image with versioned plugin command paths.
2. Declaratively register plugin versions through OpenBao server config.
3. Add upgrade-driver tooling that pins existing mounts to an explicit old version, then tunes them to the target version.
4. Update new mount creation to use bao secrets enable -plugin-version=<target>.
5. Keep old plugin versions through a rollback window.
6. Remove old binary paths and catalog entries only after validation proves no mount is configured or running on the old/unversioned entry.

The tooling should be runnable outside Helm hook semantics. Deployment automation or a manual upgrade procedure can invoke it, but correctness should come from OpenBao state checks rather than hook ordering.

Bridge image layout:

/openbao/plugins/vault-plugin-secrets-jwt
/openbao/plugins/vault-plugin-secrets-jwt-vOLD
/openbao/plugins/vault-plugin-secrets-jwt-vNEW

Bridge server config:

plugin_directory = "/openbao/plugins"
plugin_auto_register = true

plugin "secret" "vault-plugin-secrets-jwt" {
  command   = "vault-plugin-secrets-jwt-vOLD"
  version   = "vOLD"
  sha256sum = "<old plugin sha>"
}

plugin "secret" "vault-plugin-secrets-jwt" {
  command   = "vault-plugin-secrets-jwt-vNEW"
  version   = "vNEW"
  sha256sum = "<new plugin sha>"
}

Existing mounts should be migrated by explicit tune operations:

bao secrets tune -plugin-version=vOLD jwt/
# restart or reload OpenBao so Running Version becomes vOLD

bao secrets tune -plugin-version=vNEW jwt/
# restart or reload OpenBao so Running Version becomes vNEW

New mounts should be created version-pinned:

bao secrets enable -path=<mount> -plugin-version=vNEW vault-plugin-secrets-jwt

Cleanup should be gated by mount metadata:

bao secrets list -detailed

Do not remove vOLD or the unversioned catalog entry while any JWT mount has configured or running version n/a or vOLD.

Proof notes from local testing:

• Declarative plugin_auto_register can register explicit plugin versions while preserving the existing unversioned catalog entry.
• Existing unversioned mounts continue to work after the bridge restart if the image keeps the old binary at the original unversioned command path.
• bao secrets tune -plugin-version=... changes mount metadata but does not immediately switch the running backend. The running version changes after restart/reload.
• Clearing a mount back to unversioned after plugin_version is set did not work cleanly; rollback should use explicit old and new plugin versions.
• Rolling back to a pre-bridge image after a mount is version-pinned is unsafe unless that image also carries the configured versioned plugin command path.
• After all mounts were tuned to the target version, a target-only image containing only the target command path continued to serve existing and new JWT mounts.
• Deregistering the old versioned catalog entry and the old unversioned catalog entry was safe after the metadata gate passed.

The important release-engineering conclusion is that setup/tune and cleanup should not happen in the same release. Introduce a bridge release first, keep the old explicit version through a rollback window, and remove old entries in a later release after validation.

Definition of Done

• Existing unversioned JWT mounts survive upgrade to the bridge image/config.
• Existing JWT mounts can be pinned to the explicit old version and then tuned to the target version.
• New JWT mounts are created with an explicit target plugin version.
• Rollback instructions are explicit: rollback between versioned plugin entries is supported; rollback to a pre-bridge image after version pinning is not supported.
• Cleanup refuses to remove old plugin entries while any JWT mount still uses the old/unversioned plugin state.
• Cleanup can remove old catalog entries and old binary paths after all mounts run the target version.
• The process is not dependent on Helm hook ordering.
• The upgrade guide includes preflight, upgrade, rollback, and cleanup commands.
• The implementation handles per-architecture plugin SHA values or documents why the deployed image/config does not need them.

Resources (optional)

• fix(openbao): refresh JWT plugin catalog on upgrade #1628
• build(openbao): update runtime to 2.6.2 #1620
• fix(openbao): refresh vulnerable server and plugin dependencies #1477