Skip to content

Log EL SYNCING status at debug instead of warn#706

Open
rocker-zhang wants to merge 1 commit into
grandinetech:developfrom
rocker-zhang:log-el-syncing-at-debug
Open

Log EL SYNCING status at debug instead of warn#706
rocker-zhang wants to merge 1 commit into
grandinetech:developfrom
rocker-zhang:log-el-syncing-at-debug

Conversation

@rocker-zhang

Copy link
Copy Markdown

While the execution layer is syncing, engine_forkchoiceUpdated and engine_newPayload return SYNCING, which is a normal and expected status. Grandine continues optimistically and control flow is unchanged, so logging it at warn is just noise during EL sync (the subject of #217).

This downgrades both SYNCING branches in execution_service.rs from warn_with_peers! to debug_with_peers!. The INVALID and call-failure branches stay at warn. The detail is still available via GRANDINE_LOG=eth1_api=debug (the default filter is eth1_api=info).

Re #217.

Scope note: #217 also mentions a request_with_fallback warning in eth1_api/http_api.rs that fires on a JSON-RPC -32001 "Receipt not available" error during EL sync. I left that path unchanged on purpose: it is a generic RPC-error handler that also marks the endpoint offline and increments error metrics, and there is no EL-sync-state signal at that layer — downgrading it via error-code/string matching would hide genuine endpoint failures. A proper fix would plumb EL-sync state into that layer; happy to follow up separately, so I've left #217 open.

(Saw @DarkLord017 expressed interest earlier, but the issue was unassigned with no follow-up — happy to coordinate, and the deferred request_with_fallback case is up for grabs.)

@CLAassistant

CLAassistant commented May 22, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@sauliusgrigaitis

Copy link
Copy Markdown
Member

Hey @rocker-zhang, currently those SYNCING warnings work to inform that EL is syncing. It's not ideal, but we can't simply silence it.

@rocker-zhang

Copy link
Copy Markdown
Author

Hey @rocker-zhang, currently those SYNCING warnings work to inform that EL is syncing. It's not ideal, but we can't simply silence it.

Thanks for taking a look, @sauliusgrigaitis — and you're right that we shouldn't lose the signal entirely. The pain point in #217 isn't the warning itself, it's that on a long EL sync this line repeats once per forkchoiceUpdated / newPayload for the entire duration, which both drowns out other WARNs in journalctl -p warning and trains operators to ignore the level. A middle ground that keeps the signal but kills the noise — happy to revise the PR along any of these, whichever you prefer:

  1. State-transition logging. Track the last-seen EL status on ExecutionService. Log at WARN (or INFO) the first time it enters SYNCING, and again when it transitions back to VALID. Repeats stay at DEBUG. Operators get a clear "EL is syncing" / "EL is back" pair instead of N hundred identical lines.
  2. Throttled WARN. Keep the level but rate-limit to e.g. one line per epoch while consecutively SYNCING.
  3. Demote to INFO. Less invasive than DEBUG; still in the default log stream but no longer pollutes the WARN bucket.
    My preference is (1) since it gives the most useful operator signal for the least noise, but any of them is straightforward — let me know which shape you'd accept and I'll push an update.

@rocker-zhang

Copy link
Copy Markdown
Author

@sauliusgrigaitis any preference among the three options above? I'll push the change once you pick one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants