fix: lock CPU power history swap

[davidberenstein1957]

Description

Fixes a lost-update race on CPU._power_history. The 1 Hz monitor scheduler thread (CPU.monitor_power) appends to it, and the measure_power_secs scheduler thread (CPU.total_power) drains it. The read-then-rebind in total_power was not atomic, so a sample appended between the list comprehension and self._power_history = [] was dropped.

CPU now has a threading.Lock (_power_history_lock), held only for the O(1) list swap. _get_power_from_cpus() still runs outside the lock in both paths, so a slow backend (for example the IntelPowerGadget subprocess) never blocks the monitor thread. The unreachable if not power_history_in_W: branch is removed, since a sample is always appended before the average is taken.

Related Issue

Fixes #1315

Motivation and Context

In cpu_load mode the reported cpu_power is the mean of the buffered samples. The samples lost in the unlocked swap window are the ones taken while a slow measurement is in flight, so the error is small but systematic and silent.

How Has This Been Tested?

Two tests in tests/test_cpu_load.py::TestCPULoad:

• test_cpu_total_power_keeps_samples_added_while_draining: holds cpu._power_history_lock while monitor_power() runs on another thread, checks the monitor waits instead of appending, and that the sample is in the history once the lock is released.
• test_cpu_sample_appended_right_after_the_swap_goes_to_next_window: wraps the real lock so a 100 W monitor_power() sample fires right after the swap, and checks the first total_power() averages the old window (5.5 W) and the second picks up the 100 W sample (50.5 W).

CI is green on this branch.

Screenshots (if appropriate):

N/A

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

AI Usage Disclosure

• 🟥 AI-vibecoded
• 🟠 AI-generated
• ⭐ AI-assisted
• ♻️ No AI used

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the docs/how-to/contributing.md document.
• I have added tests to cover my changes.
• All new and existing tests passed.

[codecov]

Codecov Report

❌ Patch coverage is 93.75000% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.07%. Comparing base (b438616) to head (235b9ea).
⚠️ Report is 5 commits behind head on master.

Files with missing lines	Patch %	Lines
codecarbon/external/hardware.py	91.78%	6 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1335      +/-   ##
==========================================
+ Coverage   92.04%   92.07%   +0.03%     
==========================================
  Files          49       49              
  Lines        5154     5199      +45     
==========================================
+ Hits         4744     4787      +43     
- Misses        410      412       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[benoit-cty]

🤖 This review comment was written and posted by Claude Opus 5.5 (AI assistant), at the request of @benoit-cty. Findings were checked by reading the code and running tests locally (merged with current master where relevant), but please double-check before acting on them.

Verdict: ✅ Approve with nits

The race is real. The monitor scheduler can append to the old list between the list comprehension and the self._power_history = [] rebind, and that sample is lost. The lock is correct:

• It is held only for the O(1) swap and the append. _get_power_from_cpus(), including the Power Gadget subprocess, runs outside it.
• There is no nested locking, so no deadlock risk.
• The removed empty-list branch was indeed unreachable.

56 tests pass on this PR merged with master.

Nits:

1. The test only proves that monitor_power blocks on the lock. It does not show the actual fix: that a sample appended while total_power() is running is kept for the next window. A test that appends from a second thread during the swap and asserts the sample appears in the next total_power() would lock in the behaviour.
2. The PR description describes a different test (a list subclass) from the one in the diff; please update it.
3. The branch is behind master; please update it.

[davidberenstein1957]

Made the changes in 44ef635: merged master, added a test that a post-swap sample lands in the next window.

Not done: updating the description's test paragraph (my PR-edit was blocked; the diff's tests are the lock-blocking one plus the new next-window one, no list subclass).

[benoit-cty]

Review follow-up

The lock fixes the swap/append race on _power_history. While reviewing, I found related gaps around it and pushed fixes for all of them.

1. Sampling now happens under the lock

_get_power_from_cpus() was called outside the lock in total_power() and monitor_power(), but in some modes it mutates shared state:

• tracking_mode="process": it reads and writes _last_measurement_time / _last_cpu_times. If the two threads interleave, one computes its CPU delta against the other's baseline and records a wrong sample.
• intel_power_gadget: both threads could run PowerLog into the same log file and read it back. A truncated read returns {}, so a 0 W sample goes into the average.

Averaging the drained list still happens outside the lock, so a monitor sample is never lost: it lands in the next window.

2. CPU.start() starts a clean window

• It clears _power_history. The monitor scheduler starts before hardware.start() in start_task(), so samples from between tasks, or left over from the previous task, used to end up in the next task's first window.
• In process mode it now takes the CPU-time baseline. Before, the first sample after start() always reported 0 W ("First measurement, no CPU delta available yet").

3. CPU load no longer shares psutil.cpu_percent()'s global baseline

psutil.cpu_percent(interval=None) keeps one baseline per process. The tracker's _monitor_power calls it every second for cpu_utilization_percent, so in machine/cpu_load mode the latest_sample taken by total_power() often covered only a few milliseconds (frequently 0% or 100%). It was still averaged with the same weight as the 1 s samples.

• A new SystemCPULoadMeter computes load from psutil.cpu_times() with its own baseline, using the same busy/total formula as psutil. start() resets it, which also removes the 50 ms blocking priming and the _cpu_load_percent_primed module global (and its hook in hardware_cache.clear_cache()).
• Each history entry is now (power, seconds covered), and total_power() returns a time-weighted average. If no durations are known (e.g. start() was never called), it falls back to the plain mean.

4. Tracker utilization histories are locked

_cpu_utilization_history, _ram_utilization_history, _ram_used_history and _gpu_utilization_history had the same race: the monitor thread appended while the main thread cleared them or computed sum()/len(). _monitor_power now collects its readings first and appends them under _utilization_history_lock. Clearing and averaging go through _clear_utilization_history() / _mean_utilization(), which also removes the duplicated clear blocks in start() and start_task().

Tests

• Replaced the two original tests. test_cpu_total_power_keeps_samples_added_while_draining never called total_power(), so it still passed with an unlocked swap. The other test swapped the lock for a hand-written context manager and chose the mocked wattage by thread name.
• New tests:
  • test_cpu_sample_added_while_draining_goes_to_next_window
  • test_cpu_total_power_and_monitor_power_never_sample_concurrently
  • test_cpu_start_discards_samples_from_before_the_window
  • test_cpu_total_power_weights_samples_by_covered_time
  • test_cpu_load_meter_busy_time_excludes_idle_iowait_and_guest
  • test_cpu_start_sets_process_cpu_time_baseline
  • test_monitor_power_waits_while_utilization_history_is_read
• I checked each one by reverting its fix and confirming it fails.
• uv run task test-package: 674 passed. The only failures are 7 tests in test_config.py / test_electricitymaps_config_backward_compatibility.py, which pick up a local .codecarbon.config and fail the same way without these changes.

Trade-off

monitor_power() now waits while total_power() takes its sample. That is normally milliseconds, but in Intel Power Gadget mode a PowerLog run can make the monitor tick late. That seems acceptable, since the 1 s cadence is best effort and samples are now time-weighted anyway.