Skip to content

fix: improve monitor correctness and robustness - #3

Merged
staryxchen merged 1 commit into
mainfrom
fix-correctness-robustness
Jul 10, 2026
Merged

fix: improve monitor correctness and robustness#3
staryxchen merged 1 commit into
mainfrom
fix-correctness-robustness

Conversation

@staryxchen

Copy link
Copy Markdown
Owner

Summary

Four correctness/robustness fixes to nvlink_monitor, all identified by reading the existing source. No behavioral change to nvlink_bw_test.

Fixes

1. Async-signal-safe signal handler (monitor/nvlink_monitor.{h,cpp})

  • g_running: volatile boolvolatile sig_atomic_t (the standard-mandated type for writes from a signal handler).
  • signal_handler() no longer calls std::cout — output streams are not async-signal-safe and can deadlock if the signal interrupts the main thread mid-output. The handler now only flips the flag.
  • The "Received stop signal, exiting..." notice is printed from main() after the monitoring loop observes the flag, covering both continuous and single modes.

2. Timestamp consistency in runContinuousMonitoring (monitor/nvlink_monitor.cpp)

The per-iteration timestamp was recorded before getNvLinkData(), while the pre-loop lastTime was recorded after its read. This made iteration 1's actualInterval exclude the read duration ε while iteration 2+ included ε + the previous iteration's calculate/print time, producing inconsistent bandwidth denominators.

Fixed by recording currentTime after getNvLinkData() returns, so both lastTime and currentTime mark "moment a counter read completed" and every actualInterval exactly equals the observation window between two reads (which is what NVML counters actually reflect).

3. NVML fallback counter unit caveat (monitor/nvlink_monitor.cpp)

The nvmlDeviceGetNvLinkUtilizationCounter fallback returns raw counters whose units depend on nvmlDeviceSetNvLinkUtilizationCounter config and are not guaranteed to be KiB like NVML_FI_DEV_NVLINK_THROUGHPUT_DATA_TX/RX. The code feeds them through the same KiB→GiB conversion regardless.

  • Added a code comment documenting the caveat.
  • Gated the per-link fallback debug print behind --verbose (it previously spammed stdout unconditionally every sample in continuous mode).
  • The verbose warning now notes the bandwidth estimate may be inaccurate on this path.

4. Counter reset wording & handling (monitor/bandwidth_calc.cpp)

A negative delta on the 64-bit KiB throughput counters (~8 EiB wrap range) almost always indicates a driver counter reset, not arithmetic overflow. The old warning said "overflow", which is misleading.

  • Renamed warning "overflow" → "counter reset" and included the delta value.
  • Added a comment explaining no meaningful rate can be computed across a reset, so the sample is clamped to 0.
  • Renamed test CounterOverflowClampedToZeroCounterResetClampedToZero.
  • Added CounterResetVerboseWarningDoesNotCrash to exercise the verbose stderr branch.

Verification

  • make check-format — clean
  • make test59/59 pass (was 58; +1 new test)
  • make monitor — builds with -Wall -Wextra, no warnings (NVML link verified)

Files changed

 monitor/bandwidth_calc.cpp   | 19 +++++++++++++-----
 monitor/nvlink_monitor.cpp   | 53 ++++++++++++++++++++++++++++++++++++++++++-----------
 monitor/nvlink_monitor.h     |  6 ++++--
 test/test_bandwidth_calc.cpp | 16 ++++++++++++++--
 4 files changed, 75 insertions(+), 20 deletions(-)

Four fixes to nvlink_monitor:

1. Async-signal-safe signal handler (monitor/nvlink_monitor.{h,cpp}):
   g_running changed from volatile bool to volatile sig_atomic_t so
   writes from the signal handler are well-defined. The handler no
   longer calls std::cout (not async-signal-safe; can deadlock if the
   signal interrupts the main thread mid-output) -- it only flips the
   flag. The "exiting" notice is printed from main() after the
   monitoring loop observes the flag, covering both continuous and
   single modes.

2. Timestamp consistency in runContinuousMonitoring
   (monitor/nvlink_monitor.cpp): the per-iteration timestamp is now
   recorded AFTER getNvLinkData() returns, matching the pre-loop
   snapshot's timestamp. Previously the timestamp was taken before the
   read, so iteration 1 excluded the read duration while iteration 2+
   included it, producing inconsistent actualInterval deltas. Now every
   actualInterval exactly equals the observation window between two
   read completions.

3. NVML fallback counter unit caveat (monitor/nvlink_monitor.cpp): the
   traditional nvmlDeviceGetNvLinkUtilizationCounter fallback returns
   counters whose units depend on nvmlDeviceSetNvLinkUtilizationCounter
   config and are not guaranteed to be KiB like the Field Values API.
   Documented this in a code comment and gated the per-link debug print
   behind verbose mode (it previously spammed stdout unconditionally in
   continuous mode) with a warning noting the bandwidth estimate may
   be inaccurate.

4. Counter reset wording and handling (monitor/bandwidth_calc.cpp): a
   negative delta on the 64-bit KiB throughput counters (~8 EiB wrap
   range) almost always indicates a driver counter reset, not arithmetic
   overflow. Renamed the warning from "overflow" to "counter reset",
   added the delta value to the message, and added a comment explaining
   that no meaningful rate can be computed across a reset so the sample
   is clamped to 0. Renamed the test CounterOverflowClampedToZero to
   CounterResetClampedToZero and added a
   CounterResetVerboseWarningDoesNotCrash test exercising the verbose
   stderr branch.

Tests: 59 pass (was 58; +1 new). make check-format clean. make monitor
builds with -Wall -Wextra and no warnings.

Signed-off-by: staryxchen <staryxchen@tencent.com>
@staryxchen
staryxchen merged commit 7f014c3 into main Jul 10, 2026
2 checks passed
@staryxchen
staryxchen deleted the fix-correctness-robustness branch July 10, 2026 08:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants