Implement fine-grained API token authentication

[lotgon]

Context

Implement the authentication foundation for the AI-manageable HSM API: fine-grained opaque API tokens (PAT-style), not OAuth/JWT.

Normative source:

• Implementation task Markdown
• Documentation PR: Document fine-grained API token architecture #1355

The linked initiative is authoritative and contains the complete design, risks, tests, acceptance criteria, and focused PR sequence.

Core authorization invariant

allowed(operation, resource)
    = ownerCurrentlyAllows(operation, resource)
    AND tokenGrantAllows(operation, currentAuthorizationBoundary(resource))

Persist explicit operation + Product/Folder/global-boundary grant pairs. Never store permissions and scope as freely recombinable sets.

Use the actual HSM model:

• IsAdmin is global.
• ProductManager and ProductViewer are per Product/Folder.
• No global Client role or disabled/blocked user state exists.
• Owner deletion, downgrade, or resource-role removal affects the next request.
• An IsAdmin owner can create a read-only token whose covered mutations all return 403.

An existing token cannot expand operation/boundary pairs, including through rotation. An explicitly confirmed Folder grant intentionally includes current and future descendants; this is the only dynamic-membership exception.

Token and verifier contract

• Format: hsm_pat_v1_..
• TokenId: 128 random bits, exactly 22 canonical unpadded Base64URL characters.
• Secret: 256 random bits from RandomNumberGenerator.GetBytes(32), exactly 43 canonical unpadded Base64URL characters.
• Reject padding, wrong lengths/alphabet, aliases, and non-canonical encoding.
• hsm_pat_v1_ maps only to versionByte 0x01 and must match the stored record.
• Persist only:
  SHA-256(ASCII("HSM-API-TOKEN") || 0x00 || versionByte[1] || tokenId[16] || secret[32]).
• No pepper, PepperKeyId, pepper configuration/rotation, or separate verification secret.
• Always hash the presented candidate. Compare against stored-or-fixed-dummy verifier with FixedTimeEquals. A missing-record marker forces generic failure after comparison.
• Server generates every secret; user-selected secrets are forbidden.

Credential-safe v1 capability boundary

The v1 token catalog contains no users:, access-keys:, credentials:, or server-settings: capability.

Grantable management APIs:

• use dedicated credential-free DTOs;
• never serialize User, UserViewModel, AccessKeyModel, or AccessKeyViewModel;
• never return User.Password, operative access-key Id/master key, token, verifier, secret, or reversible authentication material.

A future metadata-only user/access-key/settings capability requires a separate threat review and purpose-built DTO.

Principal and listener isolation

• Cookie remains DefaultAuthenticateScheme, DefaultChallengeScheme, and the scheme pinned by DefaultPolicy/bare Authorize.
• HsmApiToken is never default, forwarded, or a policy scheme. It runs only from explicit management policies.
• Ordinary data-management /api/v1 routes require only HsmApiToken and reject cookie-only principals.
• /api/v1/api-tokens is cookie-only and rejects all API-token principals.
• A non-/api/v1 hsm_pat_ bearer guard returns non-redirecting generic 401 before MVC/Razor/BaseController, performs no token lookup, and never produces 500.
• A token principal has exactly one HsmApiToken identity. Mixed/multiple identities fail closed.
• UserProcessorMiddleware never replaces a token principal and the handler never sets Identity.Name to a login.
• Management controllers derive from ControllerBase, not BaseController.
• /api/v1 is SitePort-only and rejected on SensorPort through fail-closed endpoint metadata.
• Preserve existing collector/Grafana OpenAPI, Swagger UI, and Key/ClientName behavior on both listeners. Management v1 has a separate SitePort-only document and does not inherit DataRequestHeaderSwaggerFilter.

Persistence identity and insertion

ApiTokenEntity includes:

• EntityId: stable GUID for lifecycle routes, in-memory entity identity, and journal.
• TokenId: public 128-bit authentication lookup key.
• VersionByte, verifier, owner, grants, timestamps/revocation metadata.
• GlobalRevocationGenerationAtIssue and OwnerRevocationGenerationAtIssue.

Lifecycle routes use {entityId}; list/detail returns EntityId plus a safe display hint, never full TokenId/secret/verifier.

Implement dedicated persist-first TryInsertApiToken:

• existence check and LevelDB write are serialized atomically inside the database worker/store boundary;
• do not compose unlocked read + Put;
• do not use generic ConcurrentStorage.TryAdd for creation;
• publish to the authentication index only after persistence;
• write failure leaves neither durable nor live state;
• collision retries a completely new ID/secret pair and never overwrites.

Emergency revocation

Cookie-only IsAdmin routes:

• POST /api/v1/api-tokens/emergency/users/{ownerUserId}/revoke
• POST /api/v1/api-tokens/emergency/revoke-all

Require CSRF, sanitized reason, and explicit target confirmation. Remain available when ApiTokens.Enabled=false.

Mechanism:

• revoke-user atomically advances one durable owner revocation generation;
• revoke-all atomically advances the durable global generation;
• persist before publishing to authentication;
• every authentication compares token issuance generations with current values;
• missing/corrupt/regressed generation state rejects authentication;
• successful generation change invalidates the whole target set before 204;
• reconcile per-token RevokedAtUtc metadata afterward in bounded retryable batches;
• generation write failure starts no cleanup and returns 503 with correlation;
• post-persistence publication failure makes token authentication unavailable until authoritative generations reload.

HTTP mapping: 204 success, 400 confirmation/reason, 401 no cookie, 403 non-admin, 404 unknown owner, 503 storage/publication failure.

Generation-invalidated records immediately stop counting toward MaxTokensPerUser, even before per-record reconciliation.

Lifecycle rules

• Create returns the full token exactly once with Cache-Control: no-store.
• Restrict only removes grant pairs or shortens expiry.
• Rotate preserves/reduces grants and expiry, creates fresh EntityId/TokenId/secret, atomically revokes the old token, and cannot make finite expiry unlimited.
• API tokens cannot create/list/restrict/rotate/revoke tokens.
• ApiTokens.Enabled=false rejects token auth and issuance/rotation/restrict while cookie list/revoke and emergency actions remain available.
• MaxTokensPerUser counts only unexpired, individually active, generation-current records.
• Retention cleanup is bounded and independent from lifecycle/security-event audit retention.
• Current owner rights and current hierarchy are checked on every request.
• Folder grants store stable Folder kind/ID, warn/confirm dynamic descendant membership, and fail closed after deletion/recreation with another ID.

Delivery tasks

• ADR, /api/v1 convention, capability/role matrix, operation-boundary catalog, and separate management OpenAPI group.
• Versioned token entity/store, EntityId/TokenId indexes, persist-first TryInsertApiToken, revocation generations, authentication index, retention/orphan cleanup.
• Strict token parser/verifier, dummy path, cookie/token scheme isolation, legacy-route bearer guard, listener guard, current-owner/resource authorization.
• Cookie-only create/list/restrict/rotate/revoke and IsAdmin emergency routes with CSRF, confirmation, audit, one-time disclosure.
• Dedicated credential-free management DTOs, redaction, invalid-attempt limiting, security-event sink, emergency recovery.
• Crypto, collision/write-failure, generation/publication-failure, quota/retention, lifecycle, privilege-reduction, hierarchy, listener/OpenAPI, legacy MVC, and compatibility tests.
• Unattended read-only monitoring journey and canonical docs/ADR/glossary updates.

Mandatory acceptance examples

• IsAdmin-owned read-only token cannot mutate or call lifecycle routes.
• No grantable response returns user/access-key credentials.
• Valid HSM bearer on legacy MVC/BaseController returns 401, performs no token lookup, and never renders/returns 500.
• Unknown TokenId follows dummy verifier and cannot authenticate.
• At quota: emergency generation advance allows immediate new issuance before reconciliation.
• Existing cookie, collector access-key, Grafana, and default Swagger behavior remains compatible.
• Management routes and management OpenAPI are unavailable on SensorPort.

Use the complete acceptance and verification matrix in the linked initiative. Deliver the focused PR sequence rather than one large implementation PR.

[GonzoSpire]

Closing — the full sequence landed on master:

• Token material/parser/verifier, entity store, revocation generations, retention — PR API tokens: domain, verifier and persist-first LevelDB store (#1356 step 2) #1360.
• HsmApiToken scheme, cookie/token isolation, area/listener/legacy guards, authorization evaluator, security-event sink, ADR-0003 — PR API tokens: HsmApiToken scheme, fail-closed /api/v1 area and authorization foundation (#1356 step 3) #1362.
• Retention/throttling config + hourly cleaner — PR API tokens: retention cleanup and invalid-attempt throttling (#1356, step-4 prerequisite) #1363.
• First read-only /api/v1 endpoints — PRs REST CRUD for alert templates (#1351) #1364, REST read-only for alert schedules (#1352) #1365; OpenAPI + uniform JSON error contract — PR OpenAPI publication + uniform JSON error contract for /api/v1 (#1353) #1366 (+ fix PR fix: keep the web UI 'API' link on the SitePort (swagger 404 after #1368) #1369).
• Cookie-only personal token management UI + ApiTokens.* config (step 4 PR A) — PR Profile page and personal API token management UI (#1356 step 4, PR A) #1368; IsAdmin emergency revoke (step 4 PR B) — PR feat(#1367): IsAdmin emergency revoke-user/revoke-all (step 4 PR B) #1370 (closed API tokens: profile page and personal token management UI (#1356 step 4) #1367).
• Docs: aicontext/features/server/api-tokens/, aicontext/features/server/management-api/, glossary updates.

Epic #1347 closed as complete as well. Carry-over nits from #1366 round 3 → #1371.

[saetto]

✅ Manually tested — token lifecycle on Profile: create (with the read-only checkbox), the one-time secret disclosure modal, rename, rotate (fresh id/secret, old one dies immediately), revoke, the 10-token quota (Create button disables at the cap, with a tooltip explaining why), and not_found on foreign/unknown/already-revoked ids for rename/rotate/revoke. Also verified the disabled-mode banner and degraded behavior when "Enable API tokens" is off. All works as specced.

Two follow-up findings from this pass, filed separately (not regressions of this issue): #1422 (Esc doesn't close the Create/Revoke confirmation modals, unlike the one-time secret modal which is correctly Esc-proof by design) and #1423 (the disabled-tokens banner is easy to miss — proposing to bump its severity styling and clarify the copy).

[saetto]

✅ Manually tested the authentication perimeter (checklist block 2), on top of the earlier lifecycle pass on this issue:

• Bearer token on /api/v1 (SitePort, 44333) → 200.
• Same bearer on the sensor port (44330) → 404, not 401 — the area is invisible there, confirms it never leaks endpoint existence.
• Same bearer on a legacy route (/Home) → 401 with a JSON body ("A management API token is not accepted on this route."), no redirect to login, no 500.
• Cookie session (no bearer) on /api/v1/products → 401, a logged-in cookie session is never treated as a token principal.
• Bearer on the cookie-only token-management surface (/Profile/CreateToken) → 401, confirming a token can never manage tokens. Note: the literal /api/v1/api-tokens prefix (ManagementApiGuardMiddleware.CookieOnlyRouteRoot) has no controller mapped to it today — consistent with epic Epic: REST management API for AI agents (API keys + alert templates/schedules) #1347's explicit "Out of scope: token management is cookie-only," so this is forward-looking defensive code, not a gap.
• Read-only token, POST /api/v1/alertTemplates → 403, confirming the read-only flag is enforced at the policy level regardless of which endpoint is hit.

All as specced.

[saetto]

✅ Manually tested emergency revoke (checklist block 5):

• Users page → key icon on a user row → "Revoke user tokens": requires typing the exact username, requires a reason, revokes every live token of that user immediately.
• Configuration → Server → "Revoke all API tokens": requires typing the literal phrase revoke-all, requires a reason, revokes every live token of every user immediately.
• Both work with "Enable API tokens" turned off — the emergency lever is correctly independent of the kill switch.
• After revoke-all, Create token works again and new tokens authenticate normally.
• Cross-checked a revoked token directly with curl: GET /api/v1/products → 401 {"error":"unauthorized",...}. Note for future testers: Swagger UI's "Authorize" button only stores the token string client-side — it does not round-trip to the server, so it will happily accept an already-revoked token without complaint. Only a real request ("Try it out") proves whether a token is actually still live.

All as specced, no findings.