feat(app-hosting): dedicated always-on tier (ships dark) - #2504
Conversation
|
Warning Review limit reachedNext included review available in 56 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (36)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: dec6ab2ab1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const existing = await findDedicatedSubscriptionForApp(input.publishedAppId); | ||
| if (existing) return { ok: false, reason: 'already_subscribed' }; |
There was a problem hiding this comment.
When two POSTs for the same app overlap, both can observe no mirror row here and then independently execute stripe.subscriptions.create. The later database upsert merely replaces the app's pointer with one subscription ID, leaving the other Stripe subscription untracked and potentially charging the customer recursively. Use a durable per-app claim or a stable Stripe idempotency key rather than relying on this preflight read.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 2689e88 — you were right that the preflight read cannot close this.
The window is between the read and Stripe, so no amount of checking first removes it; two POSTs read before either writes, both pass, and both mint a subscription. The mirror's UNIQUE on publishedAppId then keeps one pointer and the other charges the customer monthly with nothing in our database naming it.
subscriptions.create now carries an idempotency key derived from what the racing requests agree on — the app id and the previous subscription id each of them read:
pgs-dedicated:<publishedAppId>:<previousSubscriptionId ?? 'none'>
Two overlapping purchases compute the same key, so Stripe answers both with one subscription. It is deliberately not constant per app, which matters in the other direction: a legitimate re-buy after a cancellation follows a different previous id, gets its own key, and is not silently answered with the dead subscription.
I preferred this over a durable claim row because the mirror's NOT NULL columns (stripeSubscriptionId, both period bounds) have no honest placeholder values before Stripe has replied.
Three tests in apps/web/src/lib/app-hosting/__tests__/dedicated-subscription.test.ts: the key is sent, two racing calls compute one key, a re-buy computes a different one. Mutation-proved twice — drop the key, and make it constant per app; each turns the matching tests red.
| loggers.ai.warn('Published app tier could not follow its dedicated subscription', { | ||
| publishedAppId: mirror.publishedAppId, | ||
| stripeSubscriptionId, | ||
| status, | ||
| target, | ||
| reason: result.reason, | ||
| }); | ||
| return { outcome: 'tier_change_refused', publishedAppId: mirror.publishedAppId, reason: result.reason }; |
There was a problem hiding this comment.
When a canceled or unpaid subscription belongs to a larger dedicated-only guest, setPublishedAppTier(..., 'metered') returns guest_preset_not_allowed; this branch only logs and returns the refusal. The webhook is then acknowledged while the app remains dedicated, so it retains the always-on configuration and is excluded from both awake and storage metering indefinitely despite having no paying subscription. This terminal-status path needs to stop/disable the app or otherwise enforce a non-entitled state.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 2689e88. You are right, and this was the strongest of the four: I had turned this path into a warning-level Sentry capture, which makes the leak visible but does not make it stop — and as you note, no further Stripe event is ever coming for a dead subscription, so nothing would reconsider it.
enforceUnpaidDedicated now runs when a non-entitled subscription's downgrade is refused for guest_preset_not_allowed. The order is the whole design, because each step is what makes the next one safe:
- Stop the machine. This is what actually ends the cost, and it is first so everything after it happens to an app that is already down.
operator, notinsolvent—insolventlands inparked, andparkedis metered-only, so that transition would be refused while the app is stilldedicatedand the machine would stay up. - Tier and guest in one statement. Neither shape is legal alone:
published_apps_metered_guest_presetmakes a metered row on a larger guest unrepresentable, so "set the tier, then resize" is two statements of which the first cannot commit. - Push
min_machines_running: 0. Without this the row says metered while the live machine config still says keep-one-up, and Fly's proxy restarts the machine we just stopped. The row alone never reaches Fly.
A failed stop aborts rather than resizing a running machine — that would leave a machine whose row promises a guest it is not on, still always-on.
On the resize, which is the one part the customer did not ask for: it is defensible only because step 1 already stopped the app, so no live machine is interrupted and the smaller guest is what the next wake creates. Nothing is lost with it — a published app's machine has no volume; its filesystem comes from the image. It is gated on !entitled, so a paying customer can never lose their guest as a side effect of an ordinary billing event; there is a test asserting exactly that.
Six tests in dedicated-tier-service.test.ts. Mutation-proved twice: revert to the logged refusal → the enforcement test goes red; skip the stop → three go red.
75c3c28 to
626b1bf
Compare
The flat monthly SKU on the metered pipeline: skip the balance gate, keep one machine up, exempt from the idle reaper, larger guests priced per size. The tier is the only difference between the two products, and it changes four things: no balance gate at the router (already there from #2491), no awake-seconds drain, `min_machines_running: 1`, and reaper exemption. Policy lives in one pure module (`dedicated-tier.ts`); the database enforces the one shape that must never exist (a parked dedicated row) and a new CHECK confines METERED apps to the v1 guest, because the awake meter prices every second at one fixed shape and a metered app on a bigger guest would be silently under-billed. Billing is a per-app Stripe subscription mirrored in a new `published_app_subscriptions` table, deliberately NOT in `subscriptions`: that table is the ACCOUNT PLAN mirror, and a hosting subscription routed through `handleSubscriptionChange` derives an account tier of `free` from its unmapped price and writes it over a paying customer's tier — permanently, since the reconcile cron then reads that entitled unmapped row as `indeterminate` and refuses to repair it. The webhook forks on `metadata.kind` before the account handler, fail-closed: no metadata takes the existing path byte-identically, an unrecognised kind logs and takes it too. Two double-charges the tier would have introduced, found and fixed here: the awake-seconds meter and the rootfs storage drain both billed credits tier-blind, so a dedicated app would have paid a flat price AND been metered. Both row sources now filter to `tier = 'metered'`. Everything is dark behind APP_HOSTING_ENABLED, and inert where isBillingEnabled() is false (tenant, onprem) — a no-op flag with no Stripe call. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NTETMrmm8ukvBZP7xN4TSp
…subscriptions The Art 15 coverage gate refuses any schema table that is neither read by an export collector nor listed with a stated reason — the ratchet that exists because agent_workspaces and ai_stream_sessions once went missing from every subject access request. Excluded as ORGANISATION_OWNED, the same answer as `subscriptions` and the credit ledger beside it: the row records a recurring charge for a DRIVE's infrastructure (which app is always-on, at which guest size, on which price), and `userId` is the payer denormalized from the drive owner — who is billed for the drive's machine, not anything the subject authored. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NTETMrmm8ukvBZP7xN4TSp
…tryable hosting events, revenue-leak signal F3 — `already_subscribed` gated on LIVE status, not row existence. The mirror row outlives its subscription on purpose (it is what explains why an app went back to metered), so refusing on existence meant one abandoned checkout or one ordinary cancellation permanently bricked the SKU for that app — and the cancel escape hatch could not help either, since cancelling a terminal Stripe subscription errors. Now only a paying subscription (active/trialing/past_due) or an OPEN checkout (incomplete, where the customer has a payment sheet in front of them) blocks a second charge. F1 — the dedicated webhook fork runs inside `withFundingRetry`. Without it a throwing hosting event was still marked processed, so Stripe's redelivery classified as a duplicate and was acked, leaving an app set `dedicated` forever with no paying subscription behind it and nothing left to repair it. The handler is idempotent (an upsert plus a tier write guarded on the tier it planned against), so reprocessing is safe. F4 — `tier_change_refused` now raises a warning-level, cause-fingerprinted Sentry capture. That outcome is a pure revenue leak (a subscription stopped paying but the app runs a guest the metered tier may not run, so the downgrade is refused rather than forced), and it persists until a human acts — a loggers.ai.warn is not an operator signal for that. Three mutation proofs added: restore the existence check and the three re-buy cases go red; drop the retry wrapper and the marker-release case goes red; silence the capture and the operator-signal case goes red. F2 (late-event reordering guard) is held for the migration renumber, per the review's merge-order instruction. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NTETMrmm8ukvBZP7xN4TSp
…tion mirror (F2)
Stripe does not order webhook deliveries. A customer.subscription.updated carrying
`active` can arrive AFTER the `deleted` that ended the subscription — a redelivery,
a retry after a timeout, or two events simply racing. Written blindly that late
event re-entitles the app, and it re-entitles it FOREVER: nothing further will
arrive to correct it, because the subscription is already dead. The result is an
always-on machine nobody is paying for and nothing that knows.
BOTH GUARDS SHIP, because neither closes the case alone:
- Terminal statuses are ABSORBING (load-bearing). Once a row is canceled /
unpaid / incomplete_expired, only a DIFFERENT stripeSubscriptionId — an actual
new purchase — may re-entitle that app. No clock involved, and it matches the
purchase model: re-buying always mints a new subscription.
- Monotonic `stripeEventCreated` stamps (general). These order everything the
terminal rule says nothing about — a stale `past_due` overwriting a fresh
`active`, two ordinary updates arriving backwards.
The stamp cannot replace the terminal rule: `event.created` has ONE-SECOND
resolution, so a `deleted` and an `updated` emitted a few hundred milliseconds
apart compare as equal and the stamp admits the later-arriving one either way.
That case is covered by a dedicated test.
The mirror write is now a locked read plus a decision plus an upsert in one
transaction — the ordering decision must not itself be subject to the reordering
it exists to refuse — and it returns the AUTHORITATIVE row on every outcome,
including refusals. `syncAppTierToSubscription` now takes that row instead of an
event's status, which makes the real hazard unexpressible: syncing from the event
would have let the write be refused and the tier move anyway, which is worse than
no guard because it looks defended.
Each half is mutation-proved separately: remove the terminal rule and 2 go red
(including the same-second case); remove the stamp and 1 goes red.
Migration regenerated in place (unreleased); renumber to 0276 follows the rebase
onto post-#2503 master.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NTETMrmm8ukvBZP7xN4TSp
…he reaper exemption once MIGRATION (5). My 0275 is regenerated as 0276 on top of #2503's 0275, same prevId chain, `drizzle-kit check` clean. The regenerated snapshot carries #2503's columns (lastHitAt, awakeSecondsDay/Today) alongside this branch's table, and the published-apps constraints block now holds both sides: #2503's counter CHECKs and this branch's guest-preset pair. REAPER REWIRE (6). I land second, so the duplicated rule is mine to collapse. `IDLE_REAPER_EXEMPT_TIERS` is now the single source: `isIdleReaperExempt` tests against it, `planDailyAwakeCap` asks through that predicate instead of an inline `!== 'metered'`, and the reaper's candidate query builds its `notInArray` from the same array — SQL cannot call the function, so it reads the constant rather than spelling the rule a third time. `DailyAwakeCapInput.tier` is narrowed from `string` to `PublishedAppTier` so the predicate could be used at all; "any string" was the wrong domain for a question about whether an app may be switched off. Proved rather than performed: empty that array and exactly three tests go red — the reaper's candidate source (against a real Postgres), the daily cap, and the predicate itself. SURVIVAL CHECKS (7). Both double-charge predicates survived the merge and are re-verified against a real database: `listRunningApps` and `listPublishedAppRootfs` still filter `tier = 'metered'`. The wake seam kept BOTH sides of its conflict — #2503's daily-cap check, which parks before the ledger is touched, and this branch's metered-only gate — and the `gate.holdId` → `holdId` refactor compiles clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NTETMrmm8ukvBZP7xN4TSp
… and make the purchase race unwinnable Two P1s from the Codex review, both real. ENFORCE THE UNPAID DOWNGRADE. A dedicated app may run a guest the metered tier is forbidden, so when its subscription stops paying `setPublishedAppTier` correctly refuses — and stopping there was the trap. The row stayed `dedicated`, kept `min_machines_running: 1`, stayed out of BOTH the awake meter and the rootfs storage drain, and did it INDEFINITELY, because a dead subscription sends no further events. An unpaid machine running forever, invisible to both meters. A Sentry warning made that visible; it did not make it stop. `enforceUnpaidDedicated` now runs on that path, in the order the constraints require: STOP the machine first (this is what ends the cost, and it makes everything after it happen to an app that is already down), then move tier AND guest in one statement (neither shape is legal alone), then push `min_machines_running: 0` — without which Fly's proxy restarts the machine we just stopped, because the row alone never reaches Fly. A failed stop ABORTS rather than resizing a running machine. The resize is the only part the customer did not ask for, and it is defensible only because the app is already stopped; nothing is lost with it, since a published app's filesystem comes from its image. MAKE THE PURCHASE RACE UNWINNABLE. Two POSTs for the same app both pass the live-subscription check — they read before either writes — and both call `subscriptions.create`. Stripe mints two recurring subscriptions; the mirror's UNIQUE keeps one pointer and the other bills the customer monthly with nothing in our database naming it. No preflight read can close that window. The create now carries an idempotency key derived from what racing requests AGREE on (the app and the previous subscription they each saw), so they compute the same key and Stripe answers both with one subscription — while a genuine re-buy follows a different previous id, gets its own key, and is not silently answered with the dead one. Four mutation proofs: revert the enforcement to a logged refusal, skip the stop, drop the idempotency key, make the key constant per app — each turns exactly the tests naming that mechanism red. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NTETMrmm8ukvBZP7xN4TSp
…r arrives Self-review of the enforcement path landed in 2689e88 found the same shape of defect the F2 fix had: a guard that reads as defended and is not. `enforceUnpaidDedicated` wrapped its stop in try/catch and aborted on a throw. But `stopPublishedApp` NEVER throws — it reports every refusal as a value, including the two that matter: `stop_failed` (Fly refused; the machine is very likely still running) and `lock_busy` (the awake meter's advisory lock was held, so nothing was read, stopped or billed at all). So the abort could not fire for either real failure mode, and the path would have gone on to resize a running machine and declare it downgraded. The dep now answers the question instead of hiding it in an exception that never arrives: `stopApp` returns `{ stopped, error? }`, and the default binding maps the outcomes explicitly — `stopped` and an already-stopped app count as down; `stop_failed`, `lock_busy` and every other refusal do not. The try/catch stays for a deps implementation that does throw, but it is no longer the guard. That mapping is now tested directly against `defaultDedicatedTierDeps`, which it was not before: every other test injects its own `stopApp`, so a binding that answered "stopped" to everything passed the entire suite. Mutation-proved — make it unconditional and three of the five mapping tests go red. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NTETMrmm8ukvBZP7xN4TSp
bb1ee2a to
826ad26
Compare
`enforceUnpaidDedicated` reported `same_tier` as a refusal, which raised an operator alert. Stripe redelivers events, so a second `deleted` for a subscription already enforced arrives with nothing left to do — and an app that is already stopped and already metered is exactly the state the function exists to reach. Alerting a human on every redelivery, for work that is finished, is how a real signal gets ignored. Unreachable through the sync path today (the outer tier change answers `same_tier` first), but the function is exported and tests call it directly; an exported function should not carry a landmine for its next caller. Mutation-proved: remove the check and the redelivery test goes red. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NTETMrmm8ukvBZP7xN4TSp
1 participant
The dedicated tier is the flat monthly SKU on the metered publish pipeline: same provisioner, same builder, same blue/green deploy, same router. Ships dark behind
APP_HOSTING_ENABLED, and inert whereisBillingEnabled()is false (tenant, onprem) — there it is a no-op flag with no Stripe call.The tier changes exactly four things
All four live in one pure module,
services/app-hosting/dedicated-tier.ts:parkPublishedApp, which the status machine correctly refuses (parked_is_metered_only), leaving an app that is neither woken nor parked, warns on every request, and that the customer is paying for.min_machines_running: 1, applied at build time bybuildMachineConfigand pushed to a live machine throughupdateMachineConfig— the fetch→merge→send path, and only that path.tier(see coordination below).Two double-charges this would have introduced, fixed here
Both row sources billed credits tier-blind, so a dedicated app would have paid twice:
awake-meter'slistRunningApps— the awake-seconds drain.sandbox-storage-billing'slistPublishedAppRootfs— the rootfs storage drain. This is my only edit in that shared file: oneand(..., eq(tier,'metered'))plus a docblock.Both now filter to
tier = 'metered', both covered by new cases inawake-metering.integration.test.tsagainst a real Postgres.A third, subtler one: an app upgraded while awake carried an open metering window and the wake's credit hold. The moment its tier changed the meter stopped listing it, so nothing would ever settle that window or release that hold — the reservation would suppress the payer's spendable balance for its whole TTL against a charge that was never coming. The upgrade now closes the window in the same statement as the tier and returns the hold after commit.
Billing: a separate table, and why
The Stripe subscription is mirrored in a new
published_app_subscriptionstable, deliberately not insubscriptions. That table is the ACCOUNT PLAN mirror:deriveTierFromSubscriptionswalks a user's rows there and writes the winner intousers.subscriptionTier. A hosting subscription put in it would be an entitled row on a price the tier map has never seen, which has two independent consequences:customer.subscription.createdderivesfreeand writes it over a paying Pro/Founder/Business customer's tier — a paying user demoted by buying more.indeterminate, andisTierDriftRepairabledeliberately refuses to auto-repair such a user. The demotion is permanent and invisible to the machinery built for exactly that failure.So subscriptions are stamped
metadata.kind = 'published_app_dedicated'and the webhook forks on it before the account handler — forsubscription.created/updated/deletedand forinvoice.paid/invoice.payment_failed(the invoice fork matters just as much:applyStripeFundingrefills the monthly AI-credit bucket on every paid subscription invoice, so an unforked hosting invoice grants a free allowance refill every month on its own billing anchor).Fail-closed means the OLD behaviour, not no behaviour. A subscription with no metadata takes the existing path byte-identically. An unrecognised kind is logged and takes it too — diverting it would mean a typo silently stops maintaining a real customer's tier, with nothing left for the reconcile cron to repair from.
Audited every consumer of a customer's subscription list, not just the webhook: all nine (
subscriptions/status,stripe/{reactivate,cancel,update,cancel-schedule,upcoming-invoice},monitoring-queries,credit-gate,subscription-tier-reconcile) read the localsubscriptionstable filtered by userId; none callsstripe.subscriptions.list. A separate table therefore makes the skip structural rather than a filter each of them must remember. The reconcile cron needs no change and got none.Guest presets
published_apps_guest_preset_allowedwidens to four sizes; a newpublished_apps_metered_guest_presetCHECK confines metered apps to the v1shared-cpu-1x-512. That is an economics constraint, not a preference: the awake meter prices every second at one fixed shape (PUBLISHED_APP_GUEST_SHAPE), so a metered app on a bigger guest would be under-billed by exactly the difference — silently, with no error and no drift signal. Sizes are unlocked by moving to the tier whose flat price is derived from the size it sells. Migration0275(generated;drizzle-kit checkclean).Pricing
No list price is hardcoded.
calculateDedicatedMonthlyFloorCentsderives a FLOOR from the existing constants (MACHINE_RATES× 730 h/month ×MACHINE_MARKUP_BPS), and its only runtime use is a guard: at purchase the Stripe price is fetched and refused unless it is active, monthly-recurring, USD, and at or above that floor. That makes the 1.5× substrate-cost rule actually bind on a flat SKU instead of being documentation.MACHINE_RATESis the Sprites active-hour rate for bursty sandbox runtime; across a whole month it pricesshared-cpu-1x-512at ~$100 against a real Fly cost of roughly $3. Reusing it is the founder-economics interim decision, and it errs in the safe direction (a floor set too high refuses a sale; too low sells at a loss). When hosting gets its own rate table, that function is the one place that changes.Price ids come from env (
DEDICATED_PRICE_ID_<PRESET>), fail-closed: an unconfigured preset refuses the purchase withprice_not_configuredand makes no Stripe call.stripe-config.tshardcodes account-plan ids because they areNEXT_PUBLICclient-bundle values; these are server-side only and one per size, so env lets the SKU be switched on by creating prices and setting config, with no rebuild.Decisions taken, both ratified on the questions page
past_duekeeps an app up. Losing a plan feature during dunning is an inconvenience; taking a customer's production app to scale-to-zero over a card that will retry successfully is an outage they did not cause. The free ride is bounded by Stripe's dunning ending atcanceled/unpaid— but that bound is a Stripe account setting, not code. SosurveyDedicatedDunning()counts apps overdue past 7 days into the hosting cron's log, audit row, response and a warning-level, cause-fingerprinted Sentry capture. It never fails the tick (this is an operator signal, not an incident) and cannot break the meter (.catch()-ed, counters default to 0).incompletedoes not entitle, so an always-on machine is not available to anyone who starts a checkout and abandons it.Known gap, deliberately not closed here
Unpublishing does not cancel the Stripe subscription. Deleting the
published_appsrow cascades the mirror row away and the customer keeps paying monthly for an app that no longer exists. Not fixed because the honest fix is a Stripe-side reclaim outbox (theapp_hosting_reclaimspattern, which exists precisely so a delete never strands a billing resource), Stripe is unreachable frompackages/lib, and there is no unpublish route yet to hang it off. Documented at the FK so nobody reads the cascade as if it cancelled anything; now a binding requirement on the Publish-surface task.Coordination
Reaper exemption is keyed on
tierso the two branches compose:isIdleReaperExempt()is exported here andpu/idle-reaperfilterstier = 'metered'in its own row source. Per the orchestrator's merge-time rule, whichever of the two lands second rewires that predicate to the shared helper and adds the direct exemption test. Until then this export is intentionally caller-less in its own branch.No changelog entry: the whole publish stack ships dark and has none, and nothing here is user-visible yet.
Verification
Six mutation proofs, each breaking the mechanism and confirming the right tests go red:
classifySubscriptionKindstops recognising the hosting kindminMachinesRunningForalways returns 0services" failure)Review rounds
Four P1s from the Codex review, all real, all fixed — two of them (terminal re-buy, stale-event reordering) had already been closed by earlier commits before that review landed — plus three from the orchestrator's pass (F1/F3/F4) and one, F2, whose own hole I caught while writing it.
The two that mattered most, because both were silent revenue leaks that no test and no alert would have surfaced:
min_machines_running: 1, stayed out of both meters, and stayed that way indefinitely, since a dead subscription sends no further events.enforceUnpaidDedicatednow stops the machine, moves tier and guest together (neither shape is legal alone), and pushesmin_machines_running: 0— the row alone never reaches Fly.subscriptions.createnow carries an idempotency key derived from what racing requests agree on, so they collapse to one — while a genuine re-buy computes a different key and is not answered with the dead subscription.Verification
Local gate run (build slot, this worktree):
typecheck17/17,lint15/15,knip:check4 issues all within the 4 baseline,drizzle-kit checkclean. Test suites: lib 10,766 passed and web 18,890 passed; the 55 failures across both are all the missing local test Postgres (role "test" does not exist/relation ... does not exist) — every failing file is either*.integration.test.tsor DB-gated, none touch this change, and CI provides the database. The hosting integration tests were run against a REAL Postgres (a private migrated database in the shared test container, since dropped):awake-metering.integration.test.ts27/27, which is where the two double-charge guards and the four CHECK constraints are proved.Mutation proofs: 29. Each breaks one mechanism and confirms exactly the tests naming it go red — including the two double-charge guards and the reaper exemption against a real Postgres. Notable ones: empty
IDLE_REAPER_EXEMPT_TIERS→ the reaper's candidate source, the daily cap and the predicate all go red; breakclassifySubscriptionKind→ the account-tier clobber and credit-refill guards go red while the four fail-closed cases stay green; revert the unpaid enforcement to a logged refusal, or drop the Stripe idempotency key → the tests naming those go red.Self-review found two more, both the same shape
Neither was reported by anyone, and both are the shape worth naming: a guard that reads as defended and is not.
enforceUnpaidDedicatedwrapped its stop intry/catchand aborted on a throw — butstopPublishedAppnever throws. It reports every refusal as a value, including the two that matter (stop_failed,lock_busy). The abort could not fire for either real failure mode, so the path would have resized a still-running machine and declared it downgraded. The dep now answers the question ({ stopped, error? }) instead of hiding it in an exception that never arrives — and that mapping is now tested against the default binding, which every other test bypasses by injecting its own stub.same_tier) as a refusal, raising an operator alert on every Stripe redelivery for work that was already finished.Both mutation-proved.
CI note
This PR's
pull_request: synchronizeevents stopped being delivered — three pushes produced zero check runs while other branches triggered normally. Aworkflow_dispatchrun proved Actions itself was healthy, andgh pr close && gh pr reopenre-armed it; it recurs per push, so a close/reopen may be needed after future pushes. All 11 checks are green on the current head.