[simplex]: Keep posted limit orders alive with heartbeats, renewal and reconciliation

[dharjeezy]

Section 7 of #1267. A posting is not something the orderbook keeps for you: it suspends a solver it has not heard from, an entry expires on its own clock, and a crash between two requests leaves one side holding what the other does not. LimitOrderLifecycle runs the three jobs that keep the two copies together, and bootFiller starts and stops it with the filler.

heartbeat() signs Heartbeat(solver, timestamp) on the same domain as CancelOrder and goes out on half the interval serverInfo asks for. It stays quiet until something is posted, because the orderbook only knows a solver whose order it has accepted, and it fires immediately on a posting that comes back surfaced: false since that is the orderbook saying the solver is still suspended.

renewExpiring() replaces an entry before it expires rather than extending it, which is not possible: the op carries its own deadline and the orderbook remembers every op hash it has taken, so renewal is a fresh op on a new nonce. reconcile() pages through the solver's orders and cancels an entry nothing local owns, reposts an order whose entry has gone, and leaves a resized or backed: false entry alone with UNDER_FUNDED on the row, since reposting it would only have it cut down again. A row that went to resizing in the last two minutes is skipped, because that is a repost in flight rather than a crash, and posting a second entry for one liability is the thing section 6 went out of its way to avoid.

An orderbook that is down at boot does not stop the filler starting. It prices from the local limit orders either way, a pass that throws is logged and its clock carries on, and a tick is skipped while the previous pass is still running.

orderbook.renewMarginSecs and orderbook.reconcileIntervalSecs were already accepted and validated but nothing read them. They now default to 120 and 300 seconds.

The last acceptance criterion on #1267 is also covered here. src/tests/fixtures/orderbook-userops.json is a copy of the orderbook's own golden vectors, 14 signed fillOrder UserOps whose descriptions each name the verdict they expect. posted-userop.test.ts reads an op the way the orderbook does and returns the refusal it would earn, all 14 vectors get the verdict they name, and the op prepareLimitOrderUserOp builds then goes through the same function and is accepted. Checking our op against a hand-written list of rules would only ever prove we agree with ourselves.

Two checks were added on top of that, and each found a bug the rest of the suite could not see, because nothing else here reads a query or talks to a server. schema.test.ts validates every document the client sends against the orderbook's published schema.graphql: submitOrder was asking for code on both OrderRejected and OrderSubmissionFailed, which return different enums, so a server would have refused the whole mutation and no order would ever have been posted. orderbook.live.test.ts takes one limit order through a real orderbook, and found that CancelOrder was signed without the solver field the server's struct carries, so every cancel came back SOLVER_MISMATCH: withdrawing left the entry live, and a resize would have put a second entry behind the same liability.

It also found that backed: false does not mean under-funded. It is false before any balance cycle has read the order too, and the order surfaces at its full quoted size until one does, so reconciliation was stamping UNDER_FUNDED on healthy postings seconds after they went up. PostedOrder now carries validatedAt and only a backed: false a cycle actually decided counts.

The live test is skipped unless HYPERFX_ORDERBOOK_URL or HYPERFX_ORDERBOOK_BIN says where to find a server, so it does not run in CI. It needs no chain: the config it writes runs the server the way its own config.dev.toml does, with the protocol fee a constant and the validation cycle reading nothing.

A review of the whole stack then turned up four defects, fixed in the last commit. An expired limit order was read only by the matcher, so it stopped matching while renewal kept its posting alive and reconciliation put it back, advertising depth the filler would always refuse; expireStale() now withdraws it and the row moves to a new expired status. The matcher ranked on the offer rather than on min(offer, remaining - reserved), so an order quoting a good rate with nothing behind it beat one that could cover the swap, and a pre-filter on the offer made a same-chain partial fill impossible whenever the price rather than the size fell short. And the catch around executeOrder released a reservation directly even after the bid row had taken ownership of it, which gave the same hold back twice when a listener threw.

The last commit closes the rest of that review. Same-asset quoting comes back as a local limit order: no book trades a symbol against itself, so one that takes in and pays out the same symbol is stored and priced here and never sent anywhere, and it has to sit at or below par, which is the rule the curves expressed as ask-only and priced under 1. An operator whose [[pairs]] still carries curve keys is warned at boot that their prices are no longer read, since the config parses cleanly and quotes nothing. A cross-chain order nothing can value in dollars now waits the deepest confirmation its curve allows rather than the shallowest. expiresAt is validated on the way in, and the amount helpers handle a token with more than 18 decimals instead of raising RangeError.

A second read of the stack found one more, fixed in the last commit. setPosting and setStatus wrote whatever the caller asked, so a posting already in flight when the operator cancelled the order, or when the expiry sweep took it down, wrote the row back to open with a fresh commitment and a live entry: the cancel was undone and the order matched swaps again. Both writes now take the statuses the row must still hold and answer null when it has moved on, and post withdraws the entry the orderbook just accepted when its write does not apply.

One change outside the feature: a test in log-store.test.ts starts two LogStores in the same tick and assumed the second would lose the wx race, but two concurrent O_EXCL opens finish in whichever order the pool serves them, so it now waits for the first launch file before starting the second. The stale pairs.setCurve example in the src/index.ts module doc, left behind by the curve removal, is replaced with a limit order.

Stacked on #1271. Review the last seven commits only, or wait for #1271 to merge and this rebases onto main.

The last commit answers the review here. The grace period that keeps reconciliation from posting over a repost in flight only covered resizing rows, but create posts after its insert and renewExpiring posts after its cancel, and both leave a live row with no entry to find. The guard is on updatedAt alone now, for any live row whose entry is missing, and repost marks the row for the duration so the renewal path says what it is doing. The decision note is rewritten around postings rather than resizes.

The latest round answers the second review. The bid is clamped to the ask, since targetOutput is solverAmount and the gateway hands everything above totalRequired to the beneficiary and the protocol while debiting the solver the lot; payoutSurplusUsd was booking that donation as the profit that justified the fill. offer >= requestedOutput is back as a participation condition, because proportional escrow release means every fill settles at T / I whatever fraction it covers, so an order below the ask would pay above the rate it signed. A fill settles its holds in one store transaction rather than three writes. The pinned schema is refreshed, token decimals now come from the orderbook's own registry instead of an RPC read per token, and a workflow diffs the pin against the server so it cannot drift silently again.

Two decisions in there are mine rather than the reviewer's, and both are worth disagreeing with if the reasoning is wrong. Orders are drawn on tightest first, not best price first: the rate is the swapper's whichever funds the slice, so the choice only decides what stays resting, and a generous order qualifies for every swap a tighter one does plus swaps it cannot serve, at the same cost per unit. And a swap draws on several orders through one bid with slices, not one bid per order: clamping to the ask means the total is the ask, so a single bid covers it, which removes the shared nonce key, the head-of-line blocking and the dependency on #1259 entirely.

abb922e0 answers the newest comment: no path reposts an expired limit order. The check now sits in repost, which every posting goes through, rather than in the sweep, since reconciliation runs on its own clock and a late-settling fill never swept at all. An expired order reaching it is withdrawn and retired the way the sweep does it, through the same function. Reconciliation also counts a repost only when a posting actually landed, since a retired order was being reported as one.

Part of #1267.

[Wizdave97]

Reviewed against #1267 §7 and hyperfx-orderbook@main.

The lifecycle itself is solid: three independent clocks, a running guard so a slow orderbook can't overlap passes, failures confined to their own clock, and boot not blocking on the first reconcile. Halving heartbeatIntervalSecs so one lost request isn't a suspension is the right margin. expireStale before renewExpiring is the right order and the comment says why.

Two checks I ran that came back clean and are worth recording:

• myOrders selects no status, and that's correct — push_order_filter with status: None emits no status clause, so the page covers ACTIVE and SUSPENDED alike (crates/store/src/orders.rs::solver_order_pages). Reconciliation sees suspended entries and won't repost duplicates for them.
• underFunds() gating backed === false on validatedAt being present is right: per the schema backed is also false before any cycle has read a balance, and an order surfaces at its quoted size meanwhile. Reporting that as under-funded would have been noise on every fresh posting.

One comment below. Note that the two other findings I had here — repost() discarding the cancel result, and the transport-error/UNKNOWN_ORDER conflation that makes it fire — belong to #1271 and #1270 respectively, where those lines were introduced. They both bear directly on the decision doc added in this PR.

[Wizdave97]

Re-reviewed at 5abbf901 against hyperfx-orderbook@main (now 9121a06). Every one of the 15 comments from the last pass is addressed, and several are fixed better than I suggested — POST_GRACE_MS generalising the reconciliation guard to all three posting paths rather than just the two I named, holdAll taking every hold or none, and 5d06ef80 catching a late posting resurrecting a cancelled row, which I had missed. drawDown/release now throw on a lost CAS, orderAt handles ORDER_EXISTS properly, transport failures are their own kind, and serverInfo.chains is validated. 279 tests pass locally.

The multi-order work has a correctness bug in the pricing, though, and the orderbook has moved under the pinned schema. Three comments below.

[Wizdave97]

Design direction for multi-order fills, plus a correction to something I got wrong in my first review.

The target shape: when several limit orders serve one incoming order, sort them best price to worst, keeping only those that match the incoming order's rate; price the full order input against each one; and submit a separate bid per limit order rather than one bid for a combined amount.

Reading the gateway on current main makes the case for that stronger than I put it, and shows that a rate filter is not optional — see the comment below.

The correction. In my first review I said dropping the issue's offer >= reqOut match condition (§3 rule 5) was a safe, better-reasoned relaxation, on the grounds that ranking by payout never loses a fill the rule would have kept. That reasoning only holds for selecting a single order. It is wrong about whether an order should participate at all, because of how escrow is released. offer >= reqOut is a genuine participation condition and should come back.

[dharjeezy]

Second round is in 104c939e.

The overfill donation is fixed by clamping the bid to min(offered, output.amount). Your reading of the gateway is right, and the part that stung is that payoutSurplusUsd was booking the donation as the profit justifying the fill. I checked #964 before touching it: it disabled the clamp for curve-priced legs, where the figure was a curve's answer for that leg, which does not carry over to a rate applied to the whole input.

offer >= requestedOutput is back as a participation condition, on your derivation. Worth noting it was live on the single-order path too, as you said, not only on the combined one.

A fill now settles its holds in one store transaction. I had argued ordering was enough because a transaction would span two stores; that was wrong, SqliteDataStore hands the same bids.db connection to both, so §6's requirement was always reachable. settleFill splits as a result: the draw-down goes inside the transaction, resize stays outside because it is a round trip.

Schema pin refreshed, and .github/workflows/check-orderbook-schema.yml diffs it against the server weekly and on any change here. Token decimals come from serverInfo.chains now rather than an RPC read per token per post. That change broke the live orderbook test against my locally built binary, which predated the field, which is a decent argument for the workflow existing.

Two decisions in there are mine rather than yours, and both are worth arguing with:

Tightest first, not best price first. Since every fill settles at T / I whatever funds the slice, consumption order does not change what this swap earns, only what stays resting. A more generous order qualifies for every swap a tighter one does and for swaps it cannot serve, at the same cost per unit, so the tight end is what to spend. This is the case you flagged as "if the intent is to preserve the generous end, say so and it should be the reverse".

One bid with slices, not one bid per order. With the bid clamped to the ask, the total is the ask, so a single bid covers it: each order funds a slice, is drawn down by that slice and receives that fraction of the input, so all of them settle at T / I inside their own terms. That removes the shared nonce key, the head-of-line blocking and the #1259 dependency rather than solving them. If you think the per-bid shape buys something this misses, I would rather hear it now than after #1259 lands.

[Wizdave97]

Simplex should not repost a limit order whose operator expiry has passed — and today all three repost paths can.

expireStale is the only code that reads expiresAt, and it is a separate sweep rather than a guard. repost() is the shared entry point for every posting and never checks it, so whether an expired order gets put back on the book depends on which clock fires first.