Skip to content

[nodejs] fix remote config polling slowing down multiple tests#6808

Open
rochdev wants to merge 9 commits intomainfrom
rochdev/fix-slowest-nodejs-tests
Open

[nodejs] fix remote config polling slowing down multiple tests#6808
rochdev wants to merge 9 commits intomainfrom
rochdev/fix-slowest-nodejs-tests

Conversation

@rochdev
Copy link
Copy Markdown
Member

@rochdev rochdev commented Apr 24, 2026

Motivation

That was slowing everything down.

Changes

Fix remote config polling slowing down multiple tests.

Explanation from Claude when I originally asked it to fix APPSEC_RUNTIME_ACTIVATION and APPSEC_AUTO_EVENTS_RC being the slowest tests:


Two changes made:

utils/_context/_scenarios/init.py — Added DD_REMOTE_CONFIG_POLL_INTERVAL_SECONDS: "0.5" to APPSEC_RUNTIME_ACTIVATION. Without this, each apply() call waited for the default ~5s poll cycle before the tracer acknowledged the config. The scenario had 58 total apply() calls across its test files (34 in test_remote_config_rule_changes.py alone), making this the dominant bottleneck.

utils/_remote_config.py:144 — Reduced time.sleep(2) to time.sleep(0.5). This sleep runs after every send_state() call to give subprocesses time to catch up. With a 0.5s poll interval, 0.5s (one poll cycle) is sufficient — any worker process that hadn't applied yet will poll within that window. The old 2s value was actually already too short for scenarios using the default 5s interval, but 4× too long for scenarios using 0.5s.

Estimated speedup:

APPSEC_RUNTIME_ACTIVATION: each apply goes from ~5s + 2s = ~7s → ~0.5s + 0.5s = ~1s — roughly 7× faster
APPSEC_AUTO_EVENTS_RC: each apply goes from ~0.5s + 2s = ~2.5s → ~0.5s + 0.5s = ~1s — roughly 2.5× faster


I then asked it to apply the same fix to everything else.

Workflow

  1. ⚠️ Create your PR as draft ⚠️
  2. Work on you PR until the CI passes
  3. Mark it as ready for review
    • Test logic is modified? -> Get a review from RFC owner.
    • Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

  • Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
  • A docker base image is modified?
    • the relevant build-XXX-image label is present
  • A scenario is added, removed or renamed?

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 24, 2026

CODEOWNERS have been resolved as:

utils/_context/_scenarios/endtoend.py                                   @DataDog/system-tests-core
utils/_remote_config.py                                                 @DataDog/system-tests-core

@datadog-prod-us1-5
Copy link
Copy Markdown

datadog-prod-us1-5 Bot commented Apr 24, 2026

Tests

Fix all issues with BitsAI or with Cursor

⚠️ Warnings

🧪 2 Tests failed

tests.debugger.test_debugger_pii.Test_Debugger_PII_Redaction.test_pii_redaction_method_full[poc] from system_tests_suite   View in Datadog   (Fix with Cursor)
AssertionError: The following probes are not emitting: {'log170aa-acda-4453-9111-1478a6method': 'INSTALLED'}
assert not {'log170aa-acda-4453-9111-1478a6method': 'INSTALLED'}

self = <tests.debugger.test_debugger_pii.Test_Debugger_PII_Redaction object at 0x7fc6f2b6dfd0>

    @slow
    def test_pii_redaction_method_full(self):
>       self._assert()

tests/debugger/test_debugger_pii.py:247: 
...
tests.remote_config.test_remote_configuration.Test_RemoteConfigurationUpdateSequenceLiveDebugging.test_tracer_update_sequence[php-fpm-8.5] from system_tests_suite   View in Datadog   (Fix with Cursor)
Exception: ("client is not expected to have cached config but is reporting cached config: [{'path': 'datadog/2/LIVE_DEBUGGING/metricProbe_33a64d99-fbed-5eab-bb10-80735405c09b/config', 'length': 360, 'hashes': [{'algorithm': 'sha256', 'hash': '6daaa0eb13996d340d99983bb014ef17453bad39edf19041f24a87a159ff94fe'}]}]", 'SUCCESS - Initial Request, bump TUF version but nothing else. RFC about integrating with remote-config: https://docs.google.com/document/d/1u_G7TOr8wJX0dOM_zUDKuRJgxoJU_hVTd5SeaMucQUs')

self = <tests.remote_config.test_remote_configuration.Test_RemoteConfigurationUpdateSequenceLiveDebugging object at 0x7f92c0ef3290>

    def test_tracer_update_sequence(self):
        """Test update sequence, based on a scenario mocked in the proxy"""
    
        # Index the request number by runtime ID so that we can support applications
        # that spawns multiple worker processes, each running its own RCM client.
        request_number: dict = defaultdict(int)
...

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: fa0032e | Docs | Datadog PR Page | Give us feedback!

@rochdev rochdev marked this pull request as ready for review April 25, 2026 01:00
@rochdev rochdev requested a review from a team as a code owner April 25, 2026 01:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant