Lodestar v1.47.0 with ethpillar and CSM Nimbus: Nimbus validator client fails with "Could not obtain time configuration parameters"

[merylstreet]

Describe the bug

I am not really sure where the bug lies but I believe it's coming from Lodestar.

Describe the bug

After upgrading Lodestar consensus client from v1.45.0 to v1.47.0, the Nimbus validator client (used in ethpillar for CSM validator duties) fails to initialise.

Nimbus exits almost immediately with:

Validator client initialization failed err_name=ValidatorClientError err_msg="Could not obtain time configuration parameters"

The validator service then crash-loops.

Versions tested

Nimbus validator	Lodestar consensus	Result
v26.7.0	1.45.0	works
v26.8.0	1.45.0	works
v26.7.0	1.47.0	fails
v26.8.0	1.47.0	fails

Workaround

Roll back Lodestar consensus client to v1.45.0 and everything goes back to normal.

Expected behavior

Nimbus validator client should connect to Lodestar’s Beacon API and initialise normally with Lodestar v1.47.0, as it does with Lodestar v1.45.0.

Steps to reproduce

• Network: Ethereum mainnet
• Consensus client: Lodestar v1.47.0
• Validator client: Nimbus v26.8.0 and v26.7.0
• Orchestration: ethpillar
• Execution client: Besu
• OS: Debian 13

Additional context

Logs

Sep 08 19:56:01 node nimbus_validator_client[3478491]: NTC 2026-09-08 19:56:01.887+02:00 Beacon node initialized                    node=http://127.0.0.1:5052 node_index=0 node_roles=AGBSDT
Sep 08 19:56:01 node nimbus_validator_client[3478491]: INF 2026-09-08 19:56:01.887+02:00 Initializing genesis                       nodes_count=1
Sep 08 19:56:01 node nimbus_validator_client[3478491]: ERR 2026-09-08 19:56:01.899+02:00 Validator client initialization failed     err_name=ValidatorClientError err_msg="Could not obtain time configuration parameters"
Sep 08 19:56:01 node systemd[1]: csm_nimbusvalidator.service: Deactivated successfully.
Sep 08 19:56:07 node systemd[1]: csm_nimbusvalidator.service: Scheduled restart job, restart counter is at 441.
Sep 08 19:56:07 node systemd[1]: Started csm_nimbusvalidator.service - Nimbus CSM Validator Client.
Sep 08 19:56:07 node nimbus_validator_client[3478614]: NTC 2026-09-08 19:56:07.123+02:00 Launching validator client                 version=v26.8.0-404a00-stateofus cmdParams="["--data-dir=/opt/ethpillar/plugin-csm/csm_nimbus_validator", "--payload-builder=true", "--beacon-node=http://127.0.0.1:5052", --metrics, "--metrics-port=8108", "--metrics-address=127.0.0.1", "--suggested-fee-recipient=0xxxxxxx", "--graffiti=\x", "--doppelganger-detection=off", --non-interactive]" config="(configFile: none(InputFile), logLevel: INFO, logFormat: auto, dataDirFlag: some(/opt/ethpillar/plugin-csm/csm_nimbus_validator), doppelgangerDetection: false, nonInteractive: true, validatorsDirFlag: none(InputDir), verifyingWeb3Signers: [], provenBlockProperties: [], web3signerUpdateInterval: 3600, web3Signers: [], secretsDirFlag: none(InputDir), restRequestTimeout: 0, restMaxRequestBodySize: 16384, restMaxRequestHeadersSize: 64, suggestedFeeRecipient: some(0xxxxxxxx), suggestedGasLimit: 60000000, keymanagerEnabled: false, keymanagerPort: 5062, keymanagerAddress: 127.0.0.1, keymanagerAllowedOrigin: none(string), keymanagerTokenFile: none(InputFile), metricsEnabled: true, metricsAddress: 127.0.0.1, metricsPort: 8108, graffiti: some(0xxxxxx), stopAtEpoch: 0, payloadBuilderEnable: true, distributedEnabled: false, builderBoostFactor: 100, beaconNodes: [http://127.0.0.1:5052], beaconNodeMode: bestscore, monitoringType: event, batchAttestations: false, logFile: none(OutFile))" beacon_nodes_count=1

Operating system

Linux

Lodestar version or commit hash

v1.47.0

[nflaig]

A fix was merged on the nimbus side here status-im/nimbus-eth2#8997, and we also merged a workaround #10004 on the lodestar side, neither is part of a release yet though. I don't think there is any workaround for it, maybe --params.GAS_LIMIT_SCHEDULE= works, ie. setting it to an non array value, I haven't tried that, maybe @lodekeeper can check if there is a workaround we can use, and confirm it

[lodekeeper]

Checked the --params.GAS_LIMIT_SCHEDULE= idea against the v1.47.0 source — it won't work, and the reason is a type constraint on the Lodestar side, so it fails regardless of how the Nimbus VC orders its own config parsing:

• GET /eth/v1/config/spec serializes GAS_LIMIT_SCHEDULE through serializeSpecValue(..., "gas_limit_schedule") in packages/config/src/chainConfig/json.ts, which always emits a JSON array (value.map(...)). The emitted type is fixed by the key, not by the value — there's no value that makes it serialize as a scalar.
• Trying to set it to a non-array doesn't help either: deserializeGasLimitSchedule rejects anything that isn't an array (Invalid GAS_LIMIT_SCHEDULE value ... expected array), so --params.GAS_LIMIT_SCHEDULE= (empty/scalar) makes Lodestar reject the config rather than emit a scalar.
• And any array value — including the mainnet default [], which is exactly what v1.47.0 already sends — still serializes to a JSON array, which is what the affected Nimbus VC versions (v26.7.0 / v26.8.0) reject. The root cause is the type (array vs the scalar their decoder allowlist expects), and no value override changes the emitted type.

So there's no BN-side flag workaround on v1.47.0. Until a release carries either fix (#10004 Lodestar-side, or nimbus-eth2#8997 Nimbus-side), the working interim options are:

1. roll the BN back to a Lodestar version that didn't emit GAS_LIMIT_SCHEDULE in /config/spec (v1.45.0, as noted above, works), or
2. run a Nimbus VC build that includes fix(vc): decode GAS_LIMIT_SCHEDULE arrays in getSpec status-im/nimbus-eth2#8997 (e.g. a nightly/unstable build past that merge).

I confirmed this from source rather than a live run since the serializer/deserializer both hard-require an array (there's no code path to a scalar gas_limit_schedule); happy to stand up a v1.47.0 BN + Nimbus VC repro if you'd like empirical confirmation too.

[nflaig]

sadly what I was expecting as well, another workaround would be to start nimbus vc with lodestar bn 1.45, then upgrade to 1.47 as nimbus vc only checks the config on it's own startup, the risk with this is though that if nimbus vc restarts for some reason, it will be stuck in a restart loop again

[nflaig]

v1.48.0-rc.0 has our workaround, will probably release it some time next week

[nflaig]

resolved with v1.48.0 since we added a workaround on lodestar side