fix(secret): resolve remaining secrets when one getSecretValues value cannot be decrypted

[yjyspree]

Fixes #19362

Problem

GetSecretValuesResolver decrypts all requested secrets inside a single stream wrapped in one try/catch. A single undecryptable stored value — e.g. a secret encrypted under a previous SECRET_SERVICE_ENCRYPTION_KEY before a DB migration, or corrupted Base64 — fails the entire getSecretValues batch.

For managed ingestion this means every secret referenced by the recipe (including perfectly healthy ones) resolves to an empty string, and the run fails with a misleading downstream auth error (OAuthError: invalid_client, Access denied ... (using password: NO)) that never mentions secrets. Deleting and re-creating the suspected secret doesn't help while any other secret in the same recipe is still undecryptable.

Change

• Catch decryption failures per secret in GetSecretValuesResolver: log an error naming the failing secret (with guidance that the value is likely encrypted with a different key and should be re-created) and omit only that secret from the response.
• Healthy secrets in the same batch keep resolving; the executor's existing per-secret warning (Secret 'X' not found in secret stores or environment, using empty string) then points at the actual broken secret.
• If every stored value in the batch fails to decrypt, throw IllegalStateException listing the failed secrets, so a systemic issue (e.g. a rotated SECRET_SERVICE_ENCRYPTION_KEY) surfaces loudly instead of returning an empty list.
• This matches the resolver's documented contract that missing secrets are simply omitted.

Before / after

Before — the executor log never mentions secrets; the only visible symptom is a downstream auth error:

Secret masking enabled: registered 0 secret(s)
...
OAuthError: invalid_client

After — GMS names the broken secret and how to fix it, and the executor's per-secret warning identifies it on the ingestion side:

# GMS
Failed to decrypt secret 'BAD_SECRET' (urn: urn:li:dataHubSecret:BAD_SECRET). The stored value is likely corrupted or was encrypted with a different encryption key; delete and re-create the secret to fix it. Omitting it from the response.

# executor
Secret 'BAD_SECRET' not found in secret stores or environment, using empty string

Healthy secrets in the same recipe keep working. When every stored value in the batch is undecryptable, the call fails loudly instead:

Failed to decrypt all 2 stored secret value(s): [GOOD_SECRET, BAD_SECRET]. The values were likely encrypted with a different SECRET_SERVICE_ENCRYPTION_KEY (e.g. the key changed during a migration); delete and re-create the secrets to fix them.

Testing

• Added testGetPartialDecryptionFailure: a batch with one healthy and one undecryptable secret returns the healthy secret and does not throw.
• Added coverage for the all-failure case (throws with the failing secret names) and for aspect-less secrets being omitted.
• Existing GetSecretValuesResolverTest cases pass.
• The failure mode is from a real v1.6 deployment where secrets had been encrypted under a rotated SECRET_SERVICE_ENCRYPTION_KEY: the executor-side log lines quoted above are taken from that environment's execution reports.
• Verified the fix end-to-end on a local quickstart: created two secrets, rotated SECRET_SERVICE_ENCRYPTION_KEY and restarted GMS, re-created one secret under the new key. The healthy secret keeps resolving while the broken one is omitted with the GMS error log naming it; when both are broken, the call fails loudly with the secret names instead of returning an empty list.

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly PR Title Format)
• Links to related issues
• Tests for the changes have been added/updated
• Docs related to the changes have been added/updated (n/a)
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub (n/a)

[github-actions]

Linear: CAT-3036

Thanks for your contribution! We have created an internal ticket to track this PR. A member of the core DataHub team will be assigned to review it within the next few business days - you will get a follow-up comment once a reviewer is assigned.

[cubic-dev-ai]

All reported issues were addressed across 2 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[codecov]

Bundle Report

Bundle size has no change ✅

[codecov]

Codecov Report

❌ Patch coverage is 96.42857% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...solvers/ingest/secret/GetSecretValuesResolver.java	96.42%	0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!