feat: adding `telemetry-trace-sample-rate` cli arg and set default to 1% #8561

0xmrree · 2025-12-10T18:14:34Z

Adding a new telemetry-trace-sample-rate cli arg for both VC and BN to adjust sampling rate for OpenTelemetry tracing spans. Default will now be 1%.

#8554

Proposed Changes

Simply add the cli arg then set the sampler arg in the open telemetry object constructor via below variant
https://docs.rs/opentelemetry_sdk/0.30.0/opentelemetry_sdk/trace/enum.Sampler.html#variant.TraceIdRatioBased

Additional Info

Didn't add unit tests for this given low risk and low ROI to added meaningfully test in this case

Testing

ran cargo nextest run -p lighthouse --release and all tests passed, see below

...
        PASS [   0.021s] lighthouse::lighthouse_tests validator_manager::validator_import_defaults
        PASS [   0.022s] lighthouse::lighthouse_tests validator_manager::validator_import_misc_flags
        PASS [   0.013s] lighthouse::lighthouse_tests validator_manager::validator_import_missing_both_file_flags
        PASS [   0.017s] lighthouse::lighthouse_tests validator_manager::validator_import_missing_token
        PASS [   0.014s] lighthouse::lighthouse_tests validator_manager::validator_import_using_both_file_flags
        PASS [   0.019s] lighthouse::lighthouse_tests validator_manager::validator_list_defaults
        PASS [   0.019s] lighthouse::lighthouse_tests validator_manager::validator_move_count
        PASS [   0.019s] lighthouse::lighthouse_tests validator_manager::validator_move_defaults
        PASS [   0.021s] lighthouse::lighthouse_tests validator_manager::validator_move_misc_flags_0
        PASS [   0.021s] lighthouse::lighthouse_tests validator_manager::validator_move_misc_flags_1
        PASS [   0.023s] lighthouse::lighthouse_tests validator_manager::validator_move_misc_flags_2
        PASS [  25.886s] lighthouse::lighthouse_tests beacon_node::validator_monitor_file_flag
        PASS [  26.837s] lighthouse::lighthouse_tests beacon_node::validator_monitor_metrics_threshold_custom
        PASS [  26.960s] lighthouse::lighthouse_tests beacon_node::validator_monitor_metrics_threshold_default
        PASS [  23.081s] lighthouse::lighthouse_tests beacon_node::validator_monitor_pubkeys_flag
        PASS [  47.206s] lighthouse::lighthouse_tests beacon_node::test_builder_disable_ssz_flag
        PASS [  20.507s] lighthouse::lighthouse_tests beacon_node::wss_checkpoint_flag
        PASS [  15.336s] lighthouse::lighthouse_tests beacon_node::zero_ports_flag
────────────
     Summary [ 696.662s] 310 tests run: 310 passed, 0 skipped

Adding a `new telemetry-trace-sample-rate` cli arg for both VC and BN to adjust sampling rate for OpenTelemetry tracing spans sigp#8554

eserilev

Looks good, just a few minor things and it should be ready for another round of review. Thanks!

eserilev · 2025-12-11T03:50:45Z

lighthouse/src/main.rs

+            let sample_rate = matches
+                .get_one::<f64>("telemetry-trace-sample-rate")


i think we should keep this a usize and allow values between 0 - 100. we try to avoid using f64 in our codebase in general and i dont think we need more granularity here than full percentage points

Hmm ok i'm down to convert it to u8 but this needs to be convert to ratio in f64 when we set the variant.

see below

TraceIdRatioBased([f64](https://doc.rust-lang.org/nightly/std/primitive.f64.html)) Sample a given fraction of traces. Fractions >= 1 will always sample. If the parent span is sampled, then it’s child spans will automatically be sampled. Fractions < 0 are treated as zero, but spans may still be sampled if their parent is. Note: If this is used then all Spans in a trace will become sampled assuming that the first span is sampled as it is based on the trace_id not the span_id

gonna use u8 instead of usize given ValueParserFactory does not support usize for range, if its cool hehe

eserilev · 2025-12-11T03:51:20Z

lighthouse/src/main.rs

                .display_order(0)
        )
+        .arg(
+            Arg::new("telemetry-trace-sample-rate")


can you please write a test case for this flag. we have some example flag specific tests you can use as an example

eserilev · 2025-12-11T03:52:30Z

lighthouse/src/main.rs

+                )
+                .requires("telemetry-collector-url")
+                .value_parser(clap::value_parser!(f64))
+                .action(ArgAction::Set)


lets set the default here explicitly via .default_value("1")

I think we can also set a min and max value explicitly at the flag level

sure and we can use the range of value parser for min and max

eserilev · 2025-12-11T03:53:52Z

book/src/help_general.md

          Override the OpenTelemetry service name. Defaults to 'lighthouse-bn'
          for beacon node, 'lighthouse-vc' for validator client, or 'lighthouse'
          for other subcommands.
+      --telemetry-trace-sample-rate <RATE>


in order to update this file you should actually just run make cli-local.

eserilev · 2025-12-11T03:55:31Z

tagging as backwards incompatible because the default trace span sampling is reduced from 100% to 1%

Fix an issue where a kurtosis testnet script was failing because no supernodes were provided ``` There was an error interpreting Starlark code Evaluation error: fail: Fulu fork is enabled (epoch: 0) but no supernodes are configured, no nodes have 128 or more validators, and perfect_peerdas_enabled is not enabled. Either configure a supernode, ensure at least one node has 128+ validators, or enable perfect_peerdas_enabled in network_params with 16 participants. at [github.com/ethpandaops/ethereum-package/main.star:83:57]: run at [github.com/ethpandaops/ethereum-package/src/package_io/input_parser.star:377:17]: input_parser at [0:0]: fail ``` Co-Authored-By: Eitan Seri-Levi <[email protected]> Co-Authored-By: Pawan Dhananjay <[email protected]>

…8559) Co-Authored-By: Tan Chee Keong <[email protected]>

0xmrree · 2025-12-12T08:03:29Z

@eserilev should be good to go for another review. Just used the config in the BN to keep the tests simple and following similar pattern as the other cli flag tests.

feat: trace sampling cli arg

d3b16dc

Adding a `new telemetry-trace-sample-rate` cli arg for both VC and BN to adjust sampling rate for OpenTelemetry tracing spans sigp#8554

eserilev approved these changes Dec 11, 2025

View reviewed changes

eserilev added UX-and-logs waiting-on-author The reviewer has suggested changes and awaits thier implementation. tracing backwards-incompat Backwards-incompatible API change labels Dec 11, 2025

eserilev and others added 5 commits December 11, 2025 23:58

Do not request attestation data when attestation duty is empty (sigp#…

f3f343c

…8559) Co-Authored-By: Tan Chee Keong <[email protected]>

add tests and change to u8 from f64

93aaca2

refactor: remove mock and use config for test

20d7332

added float tolerance to tests

e039381

0xmrree requested a review from eserilev December 12, 2025 21:06

0xmrree added 2 commits December 17, 2025 21:07

Merge branch 'unstable' into trace_sampling_default

97e06e1

Merge branch 'unstable' into trace_sampling_default

3cf9c1a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: adding `telemetry-trace-sample-rate` cli arg and set default to 1% #8561

feat: adding `telemetry-trace-sample-rate` cli arg and set default to 1% #8561

Uh oh!

0xmrree commented Dec 10, 2025

Uh oh!

eserilev left a comment

Uh oh!

eserilev Dec 11, 2025

Uh oh!

0xmrree Dec 11, 2025 •

edited

Loading

Uh oh!

0xmrree Dec 11, 2025 •

edited

Loading

Uh oh!

eserilev Dec 11, 2025

Uh oh!

eserilev Dec 11, 2025

Uh oh!

eserilev Dec 11, 2025

Uh oh!

0xmrree Dec 11, 2025

Uh oh!

eserilev Dec 11, 2025 •

edited

Loading

Uh oh!

eserilev commented Dec 11, 2025

Uh oh!

0xmrree commented Dec 12, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		let sample_rate = matches
		.get_one::<f64>("telemetry-trace-sample-rate")

feat: adding telemetry-trace-sample-rate cli arg and set default to 1% #8561

Are you sure you want to change the base?

feat: adding telemetry-trace-sample-rate cli arg and set default to 1% #8561

Uh oh!

Conversation

0xmrree commented Dec 10, 2025

Proposed Changes

Additional Info

Testing

Uh oh!

eserilev left a comment

Choose a reason for hiding this comment

Uh oh!

eserilev Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

0xmrree Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

0xmrree Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eserilev Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

eserilev Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

eserilev Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

0xmrree Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

eserilev Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eserilev commented Dec 11, 2025

Uh oh!

0xmrree commented Dec 12, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

feat: adding `telemetry-trace-sample-rate` cli arg and set default to 1% #8561

feat: adding `telemetry-trace-sample-rate` cli arg and set default to 1% #8561

0xmrree Dec 11, 2025 •

edited

Loading

0xmrree Dec 11, 2025 •

edited

Loading

eserilev Dec 11, 2025 •

edited

Loading