fix: improve monitor correctness and robustness - #3
Merged
Conversation
Four fixes to nvlink_monitor:
1. Async-signal-safe signal handler (monitor/nvlink_monitor.{h,cpp}):
g_running changed from volatile bool to volatile sig_atomic_t so
writes from the signal handler are well-defined. The handler no
longer calls std::cout (not async-signal-safe; can deadlock if the
signal interrupts the main thread mid-output) -- it only flips the
flag. The "exiting" notice is printed from main() after the
monitoring loop observes the flag, covering both continuous and
single modes.
2. Timestamp consistency in runContinuousMonitoring
(monitor/nvlink_monitor.cpp): the per-iteration timestamp is now
recorded AFTER getNvLinkData() returns, matching the pre-loop
snapshot's timestamp. Previously the timestamp was taken before the
read, so iteration 1 excluded the read duration while iteration 2+
included it, producing inconsistent actualInterval deltas. Now every
actualInterval exactly equals the observation window between two
read completions.
3. NVML fallback counter unit caveat (monitor/nvlink_monitor.cpp): the
traditional nvmlDeviceGetNvLinkUtilizationCounter fallback returns
counters whose units depend on nvmlDeviceSetNvLinkUtilizationCounter
config and are not guaranteed to be KiB like the Field Values API.
Documented this in a code comment and gated the per-link debug print
behind verbose mode (it previously spammed stdout unconditionally in
continuous mode) with a warning noting the bandwidth estimate may
be inaccurate.
4. Counter reset wording and handling (monitor/bandwidth_calc.cpp): a
negative delta on the 64-bit KiB throughput counters (~8 EiB wrap
range) almost always indicates a driver counter reset, not arithmetic
overflow. Renamed the warning from "overflow" to "counter reset",
added the delta value to the message, and added a comment explaining
that no meaningful rate can be computed across a reset so the sample
is clamped to 0. Renamed the test CounterOverflowClampedToZero to
CounterResetClampedToZero and added a
CounterResetVerboseWarningDoesNotCrash test exercising the verbose
stderr branch.
Tests: 59 pass (was 58; +1 new). make check-format clean. make monitor
builds with -Wall -Wextra and no warnings.
Signed-off-by: staryxchen <staryxchen@tencent.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Four correctness/robustness fixes to
nvlink_monitor, all identified by reading the existing source. No behavioral change tonvlink_bw_test.Fixes
1. Async-signal-safe signal handler (
monitor/nvlink_monitor.{h,cpp})g_running:volatile bool→volatile sig_atomic_t(the standard-mandated type for writes from a signal handler).signal_handler()no longer callsstd::cout— output streams are not async-signal-safe and can deadlock if the signal interrupts the main thread mid-output. The handler now only flips the flag."Received stop signal, exiting..."notice is printed frommain()after the monitoring loop observes the flag, covering both continuous and single modes.2. Timestamp consistency in
runContinuousMonitoring(monitor/nvlink_monitor.cpp)The per-iteration timestamp was recorded before
getNvLinkData(), while the pre-looplastTimewas recorded after its read. This made iteration 1'sactualIntervalexclude the read duration ε while iteration 2+ included ε + the previous iteration's calculate/print time, producing inconsistent bandwidth denominators.Fixed by recording
currentTimeaftergetNvLinkData()returns, so bothlastTimeandcurrentTimemark "moment a counter read completed" and everyactualIntervalexactly equals the observation window between two reads (which is what NVML counters actually reflect).3. NVML fallback counter unit caveat (
monitor/nvlink_monitor.cpp)The
nvmlDeviceGetNvLinkUtilizationCounterfallback returns raw counters whose units depend onnvmlDeviceSetNvLinkUtilizationCounterconfig and are not guaranteed to be KiB likeNVML_FI_DEV_NVLINK_THROUGHPUT_DATA_TX/RX. The code feeds them through the same KiB→GiB conversion regardless.--verbose(it previously spammed stdout unconditionally every sample in continuous mode).4. Counter reset wording & handling (
monitor/bandwidth_calc.cpp)A negative delta on the 64-bit KiB throughput counters (~8 EiB wrap range) almost always indicates a driver counter reset, not arithmetic overflow. The old warning said "overflow", which is misleading.
CounterOverflowClampedToZero→CounterResetClampedToZero.CounterResetVerboseWarningDoesNotCrashto exercise the verbose stderr branch.Verification
make check-format— cleanmake test— 59/59 pass (was 58; +1 new test)make monitor— builds with-Wall -Wextra, no warnings (NVML link verified)Files changed