Fix/trashcan restore wal barrier

[atauov]

MR title

fix(crud): prevent trash-can restore from wedging WAL publishing

Summary

Fix WAL barrier bookkeeping during trash-can restore and add diagnostics for detecting a stalled WAL publisher.

This MR:

• balances activeOps when one handler acquires graph locks in multiple passes;
• prevents child handlers from consuming their parent's WAL barrier mark;
• adds regression coverage for both trash-can restore entry points;
• adds WAL backlog, barrier age, and publish outcome metrics.

Problem

Graph write locks participate in the WAL barrier:

1. A handler acquiring a write lock calls MarkOperationActive(opTime).
2. Its single operationKeysMutexUnlock calls MarkOperationDone(opTime).

Trash-can restore broke this pairing.

CreateObject and the UpdateObject upsert path first acquired the object's write lock, then restoreObjectFromTrashCan acquired additional edge locks. Both lock passes marked the same operation active, but the handler performed only one unlock:

MarkOperationActive  +1
MarkOperationActive  +1
MarkOperationDone    -1
                        = orphaned activeOps entry

The orphan permanently pinned the WAL barrier. The publisher stopped draining pendingTxs, causing unbounded memory growth and repeated sorting and scanning of the growing backlog.

A related issue allowed __key_lock_time to propagate into child handler payloads. The child then consumed the parent's mark during its own unlock, releasing the WAL barrier while the parent was still writing.

Fix

Balanced per-handler marking

Both graph lock helpers now use markOperationActiveOnce.

A handler acquiring locks several times before one unlock contributes only one activeOps mark. If a later lock pass unexpectedly carries an older opTime, the mark is moved to the older timestamp by activating it before releasing the newer one, so the barrier is never temporarily empty.

Invocation-local child payloads

injectParentHoldsLocks still promotes held locks into __parent_holds_locks, but now removes both per-handler fields from the child payload:

• __key_locks
• __key_lock_time

The child therefore knows which locks its parent holds without inheriting bookkeeping that its own unlock could consume.

This remains local to graph CRUD and does not introduce global payload filtering in the statefun runtime.

Restore contract guard

restoreObjectFromTrashCan documents its locking contract and emits a warning when called without an existing operation mark.

Observability

Added the following per-cache Prometheus gauges:

• cache_pending_transactions
• cache_active_operations
• cache_oldest_pending_age_seconds
• cache_oldest_active_operation_age_seconds
• cache_wal_publish_total
• cache_wal_publish_errors_total

The age metrics distinguish a temporarily busy publisher from a wedged one. With a stuck barrier, the oldest pending age keeps increasing while the successful publish count stays flat.

StoreStatsForTest exposes the same snapshot data for deterministic regression and leak tests.

Tests

Added:

• unit tests for child-payload bookkeeping isolation;
• runtime regression tests for restore through CreateObject;
• runtime regression tests for restore through UpdateObject(..., upsert=true);
• the S14 leak scenario covering repeated park -> restore -> purge cycles;
• Prometheus registration and value tests for all six WAL metrics.

Verification performed:

go test ./statefun/cache/... ./embedded/graph/crud/... -count=1
scripts/run-leak-tests.sh --scenario s14

S14 result:

PASS=14
FAIL=0
cache_active_ops: PASS
cache_pending_txs: PASS
graph_keymutex_entries: PASS

Commits

• 827957a fix(crud): balance WAL barrier across trash restores
• 04ed9e2 feat(cache): expose WAL barrier metrics