Default-config validators: no fast recovery from missed commit message, plus silent late-signature exclusion — both losing real signing %

[hedgeme]

Summary

Running a mainnet shard-1 validator (v2026.1.0-gdb22d51, "Bloom") on a config
verified byte-identical to harmony config dump -n mainnet from this exact
binary (only difference: inert ShardData.ShardCount). Over the past ~24
hours we've captured hard evidence of two distinct mechanisms causing real
signing-percentage loss, both reproducible with log traces, neither explained
by local config, hardware, or network issues (all independently ruled out —
detail below). Filing since together they may point at a real gap: default
mainnet config (Sync.Client = false, i.e. not using the experimental
streamsync client) has no fast reactive-recovery path when a single consensus
message is missed, and a substantial fraction of signing loss appears to
come from the FBFT tight-deadline design working exactly as intended but with
no operator-facing signal.

Mechanism 1: missed commit-confirmation message → no fast recovery on default config

Traced second-by-second from zerolog-harmony.log, 2026-07-31 23:00:00–23:06:51 UTC:

1. 23:00:04.438 — validator sends its own FBFT Commit vote for block
   94505738, on schedule.
2. The network's aggregated Committed confirmation for that specific
   block never arrives at this node.
3. 23:00:06+ — Announce/Prepared messages for the next blocks (739, 740,
   741, ...) arrive while still parked waiting on 738's commit confirmation.
   Each rejected: [VerifyNewBlock] Cannot Verify New Block!!!, blockHeight 94505739, myHeight 94505737 / error: unknown ancestor.
4. consensus.OnCommitted correctly detects this
   ([OnCommitted] low consensus block number. Spin up state sync) and calls
   consensus.spinUpStateSync() immediately (not on a timer).
5. However, spinUpStateSync() only does consensus.dHelper.DownloadAsync(),
   and consensus.dHelper is only initialized via SetDownloader(), which
   cmd/harmony/main.go gates behind if hc.Sync.Client && .... Since
   Sync.Client = false is the binary's own default (confirmed via fresh
   harmony config dump -n mainnet diff), this fast path appears to be a
   no-op on any node running default config.
6. Actual recovery instead comes from a separate, independent, much slower
   periodic legacysync check (getMaxPeerHeight / EPOCHSYNC cycle,
   observed running on a ~20-40s cadence). Full recovery took until
   23:06:51 — ~6.5 minutes, during which the node correctly detects
   OUT OF SYNC repeatedly but cannot sign.
7. Network-wide block cadence stayed completely normal throughout (204
   blocks in 407s, ~2.0s/block, cross-checked against public RPC) — only
   this node fell behind. Confirmed identical signature (unknown ancestor
   cascade + ~5-7 min self-heal) present in this node's logs going back to
   at least 2026-07-28, so not a new regression, and reproduced 7 times in
   19 hours of monitoring before this report.

Is this a known limitation of running with default (non-streamsync) config?
Is there a non-experimental way to shorten this recovery window, or is a
faster reactive path planned for the legacysync/default path specifically
(not just the experimental streamsync client)?

Mechanism 2: silent late-signature exclusion, zero local trace

Observed live during this report, 2026-08-01 02:00:02–02:05:02 UTC:

• Two consecutive 5-minute sampling ticks: 28/127 missed (22.0%), then
  30/129 missed (23.3%).
• No unknown ancestor errors, no stall — local block height advanced
  normally throughout, gap vs. public reference stayed at 0 (briefly -9,
  i.e. momentarily ahead of a lagging public RPC, not behind).
• Only routine timeout reason was overwritten phase-timer log lines
  present — nothing that would let an operator distinguish this from
  healthy operation without external signing-percent monitoring.

This matches Harmony's own "1-Second Finality" design notes: FBFT proceeds
once 2/3 of signatures arrive per phase and explicitly excludes late
signatures past a tight per-block deadline, with no error generated on the
excluded validator's side. Understood this is by design, not a bug — but
raising because it means a well-connected, fully-synced, default-config
validator can lose 20%+ of an epoch's signing with zero actionable local
signal, which seems worth documenting/exposing (e.g. a debug-level metric
or log line specifically for "signature arrived but too late to include")
so operators aren't left inferring this purely from external explorers.

Environment

• v2026.1.0-gdb22d51 (built 2026-07-08, current mainnet release)
• Config: harmony config dump -n mainnet from this binary, zero
  modifications besides inert ShardData.ShardCount
• Wired gigabit, no Wi-Fi hop, TCP 9000 forwarded and externally verified
• p2p-connectivity.connected: 368-621 throughout every incident (healthy)
• NIC hardware/driver error counters (rx_errors, rx_crc_errors,
  rx_missed_errors, etc.): all 0
• No NVMe/ATA/MCE/EDAC kernel warnings in dmesg for the full incident window
• chrony synced, ~32µs offset from NTP at time of worst incident

What we're not asking

Not asking for the streamsync client itself to be un-experimental — we
understand that's a deliberate, ongoing effort. Asking specifically about
the default (legacysync) path's recovery behavior and whether Mechanism 2's
silence is expected/by-design or something worth better operator-facing
visibility for.

Happy to provide full raw logs for either window if useful.

[GheisMohammadi]

FBFT dropping late votes is by design; But, we can improve operator visibility (log/metric)

[GheisMohammadi]

#5093

[hedgeme]

Thanks for confirming this is by design. Follow-up question, since we've
spent the last week ruling out everything on our own end (local config —
byte-identical to harmony config dump -n mainnet; hardware — thermal,
NVMe, memory, disk, NIC error counters all clean; CPU/IRQ scheduling —
tested isolating our NIC's interrupt handling from the validator's
threads, no durable improvement, reverted; connectivity — peer counts and
p2p-connectivity healthy throughout every incident):

Is there anything on the operator side that reduces exposure to the
late-signature drop — peering strategy, network placement, a
non-experimental config lever we haven't found — or is proximity/RTT to
the gossip mesh simply not a meaningful factor given the deadline's
size? We measured real RTT to our currently-connected shard-1 peers at
45-197ms (typical for a residential connection reaching global
datacenter-hosted peers), and are trying to understand whether that's
plausibly relevant to occasionally missing the deadline, or whether the
margin is generous enough that only pathological (multi-hundred-ms+)
cases would ever matter.

Trying to distinguish "nothing more an operator can do" from "there's a
lever we haven't found" before we consider this fully closed on our end.

[hedgeme]

Following up on the question above (2026-08-06) — has anyone had a chance
to look at whether there's an operator-side lever for this, or should we
treat it as "nothing more to do, this is the deadline's margin working as
designed"? Happy to provide any additional data if it would help — we've
got detailed tick-level signing/mechanism data going back to 08-01 and
can share more of it if useful.

No urgency on our end, just checking in since it's been a couple of days.
Thanks for the help so far.

[hedgeme]

For additional context while this is pending — a short log from today
(2026-08-08, UTC) showing what a typical mechanism-2 day looks like on
our end:

Time	Missed this 5-min tick
00:10	30.7% (39/127) — worst single tick we've recorded
01:15	13.2% (17/129)
04:05	10.9% (21/192)
07:35	7.0% (9/128)
09:00	14.6% (28/192)
09:25	8.6% (11/128)
09:30	12.5% (24/192)
09:45	9.4% (12/128)
10:00–13:05	clean, 0-1.6% every tick

No restarts, no config changes, no crashes at any point in this window —
sync stayed fully caught up throughout (gap 0-1 blocks vs public
reference), and every tagged incident traces to the same mechanism (no
local error trace, height stays in sync, consistent with the late-vote
deadline). The clustering (several bad ticks within a short window, then
hours of clean signing) has been a consistent pattern since we started
tracking this in detail — not a steady/constant rate, but recurring
bursts.

[hedgeme]

Additional data since our last update, plus a new data source that might
be useful context for whoever picks this up.

Real epoch-over-epoch decline: epochs 2996-2997 signed at 99.40%;
the two most recent epochs (2998-2999) dropped to 98.88% / 99.00%,
driven partly by a fresh mechanism-1 cluster (2 stuck-block/
unknown ancestor incidents within ~2.5 hours today) after that
mechanism had been quiet for a few days.

New: we noticed harmony exposes a Prometheus /metrics endpoint
(port 9900) with real consensus-internal histograms we hadn't been
using — sharing in case it's useful signal for you too. Cumulative
since our last restart (~3.3 days, 139,970 blocks observed):

• hmy_consensus_finality histogram: 96.69% of blocks finalize ≤800ms,
  but there's a sharp secondary cluster — 2.40% land in the
  1.56-1.95s bucket, with almost nothing in the 1-1.56s range just
  before it. That gap-then-spike shape looks like it could be the
  direct fingerprint of retry/delayed-round blocks (the ones that would
  plausibly cost a validator its signature under the deadline
  discussed in this issue), rather than a smooth tail.
• hmy_consensus_viewchange: 47 started / 46 finished in the same
  window (~14/day).
• hmy_consensus_sync: 42 out-of-sync transitions vs. 33 in-sync
  (~12.7/day) — notably more frequent than the incidents big enough to
  show up in our own tick-level signing-miss sampling, suggesting a lot
  of brief desyncs aren't visible from the RPC side at all.

Following up again since it's been a few days — happy to pull more from
this metrics endpoint if it would help narrow this down further, just
let us know what would be useful.

[hedgeme]

Another root-cause pass since our last update, in case it's useful before
this gets picked up.

Full official docs review (every page under
docs.harmony.one/home/network/validators, not just the ones we'd
happened across before) — nothing beyond what we'd already checked.
The troubleshooting guide's guidance ("verify CPU/memory/hard
disk/internet issues") has been verified clean repeatedly on our end.

New check: leader correlation for the two most recent mechanism-1
incidents (2026-08-08, 15:49:36 and 18:04:41 UTC — the missed-commit-
confirmation → unknown ancestor cascade, not the by-design late-drop
discussed above). Pulled the exact stuck block numbers from our app log
and queried miner via hmy_getBlockByNumber:

• Incident 1: stuck at height 94835935, next block 94835937 mined by
  one1l0wwv67lfelxhjvu6dkm9t8j5gsxev44lm6s02
• Incident 2: stuck at height 94839939, mined by
  one1jfe5c30pcyxnzktqfv7w7qqnxysm5r3hlgcgq4

Different leaders both times — rules out "one specific leader we have a
bad connection to" as an explanation, at least for this pair. Also
notable: incident 2 had already fallen 15 blocks behind by the time it
was detected (vs. 1-2 for incident 1) — same mechanism, more severe.

At this point we've checked hardware, OS/kernel logs, binary integrity,
config, connectivity (aggregate and real per-peer path latency),
bandwidth, TCP health, and now leader correlation — all clean or
inconclusive. We genuinely can't identify a further local lever. Would
appreciate any input on whether this rings a bell on your end, or
confirmation that this is as far as operator-side diagnosis can take it.

[hedgeme]

Update in case it's useful signal for anyone else hitting this: also
upgraded to v2026.1.1 today (thanks for the fix).

Separately, found and reverted a change we'd made speculatively during
our own outage response back in July: we'd added inbound port-forwarding
for our P2P port, deviating from over a year of prior outbound-only
operation. Removed it 2026-08-11. Since then: 11 straight hours at
0.50-0.73% missed per hour, the best sustained window we've measured —
though we can't yet fully separate that from an unrelated restart that
happened around the same time (documented on our side, still an open
question). Mentioning in case inbound reachability vs. outbound-only
posture is a variable worth knowing about for anyone else chasing
occasional missed-signature issues — still watching ourselves before
drawing a firm conclusion.

No rush on the original question above — still curious whether there's
a known operator-side lever, but wanted to share this in case it's
useful either way.