Skip to content

Invalidate rx nodes at write time inside a batch - #1229

Merged
philippjfr merged 5 commits into
mainfrom
fix/rx-batch-eager-invalidation
Oct 30, 2026
Merged

philippjfr merged 5 commits into
mainfrom
fix/rx-batch-eager-invalidation

Conversation

@philippjfr

@philippjfr philippjfr commented Oct 29, 2026 •

Copy link
Copy Markdown
Member

Description

An rx node is marked dirty by a weak invalidation watcher registered on its source parameters. Inside param.batch() that watcher was queued like any user callback, which caused two problems.

A read inside a batch answered the value from before the write:

import param

a = param.rx(1)
doubled = a.rx.pipe(lambda v: v * 2)
doubled.rx.value

with param.batch():
    a.rx.value = 5
    print(a.rx.value, doubled.rx.value)  # main: 1 2, this PR: 5 10

A parameter following several cells as references could keep a stale value after the batch. _flush_batch merges the owner's single _sync_refs watcher into one group, created when the walk reaches the first touched cell. Any later cell's invalidator group is created after it, so _sync_refs resolved those references before their cells were marked dirty:

class Follower(param.Parameterized):
    out_a = param.Parameter(allow_refs=True)
    out_b = param.Parameter(allow_refs=True)

a, b = param.rx(None), param.rx(None)
f = Follower(out_a=a, out_b=b)
with param.batch():
    a.rx.value = 1
    b.rx.value = 2
print(f.out_a, f.out_b)  # main: 1 None, this PR: 1 2

Marking a node dirty is bookkeeping rather than a notification, so the invalidators behind _invalidate_current, _invalidate_obj and _invalidate_overrides now run at write time even when the owner is batched. _call_watcher skips the batch for a _WeakInvalidator created with unbatched=True; the class moved from reactive.py to parameterized.py so that check is a plain isinstance with no import. _invalidate_overrides also triggers the override channels, but those are parameter events on a Parameterized, so the batch still defers them. All other watchers are still deferred, including .rx.watch callbacks. The same applies to batch_call_watchers, .param.update() and discard_events: a cached node now follows the parameter it reads even while its notifications are held back or dropped. The batch() docstring now says that only notifications are deferred.

Tests live in a new TestBatch class in tests/testreactive.py. They cover references following every cell written in one batch, a read inside a batch answering the latest write (through pipe, an operand, a parameter and an override), .rx.watch callbacks (including on an overridden node) still being deferred until the batch closes, and a node following its parameter after discard_events. The full suite passes on 3.14.

AI Disclosure

Tool & Model: Claude Code + Claude Opus 5.5
Usage: Investigated the ordering in _flush_batch, implemented the fix and wrote the tests, under my direction and review.

  • I have tested all AI-generated content in my PR.
  • I take responsibility for all AI-generated content in my PR.

Checklist

  • Tests added and are passing

The weak invalidators that mark rx nodes dirty were queued by batch() like any user
watcher. A read inside the batch answered the value from before the write, and a
reference resolved while flushing could run before its cell's invalidator and keep
the stale value. Marking a node dirty is bookkeeping, so these invalidators now run
immediately; every other watcher is still deferred.

Assisted-by: Claude Code:claude-opus-5-5
@codecov

codecov Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.42%. Comparing base (b29607c) to head (3b5b6a3).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1229      +/-   ##
==========================================
+ Coverage   89.40%   89.42%   +0.01%     
==========================================
  Files           9        9              
  Lines        6577     6588      +11     
==========================================
+ Hits         5880     5891      +11     
  Misses        697      697              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@hoxbro hoxbro left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One comments from me and two fixes from an AI review.

AI Review

## #1229: Invalidate `rx` nodes at write time inside a batch

**Ready to merge?** With fixes

**Reasoning:** The change is small and correct. It fixes both reported bugs, and notifications are still deferred until the batch closes. But nothing tests the `_invalidate_current` half, and override refs still return a stale value inside a batch. Making the override invalidator unbatched too is a one-line change, and the suite still passes with it.

### What works

- The fix removes the cause instead of adding more ordering tricks to `_flush_batch`: one line in `_call_watcher` and a flag on `_WeakInvalidator`. Bug 2 happened because `_sync_refs` and the invalidators share precedence -1, so a stable sort kept walk order. Now the invalidators are never queued, so nothing depends on that order.
- Notifications are still deferred. `.rx.watch`, `_sync_refs` and user watchers each fire once at flush. A chain read twice inside a batch delivers `[8]` once, and `when(trigger)` holds its value inside a batch (ran).
- Because of the `and` short-circuit, `_is_batched` is never called for invalidators. An object whose only watchers are invalidators therefore never enters `touched`, which is harmless because it has nothing to flush.
- `_unbatched` is read from `watcher.fn`, which is always the `_WeakInvalidator` instance itself. `functools.partial`, bound methods and `WeakMethod` never come into it, and every other callable defaults to `False`.
- Side benefits:
  - A read inside a batch now raises the correct `error_state` instead of returning the stale value.
  - After `discard_events`, `batch_call_watchers` or `.param.update()`, cached nodes follow their source instead of staying stale until the next change.
- Re-entrancy is fine. A node whose function writes another parameter, read inside the batch, still has that write's watcher deferred to the flush: `[]` inside the batch, `[3]` after (ran).
- Async is fine. An async node read or written inside a batch settles to the right value with a single watch call, including when a task is already in flight. `updating()` gives `[True, False]`, the same as on base (ran).

### Should fix

1. **No test covers the `_invalidate_current` half** (`param/reactive.py:3263-3265`). Every new test goes through `rx(...)` roots, which use `_invalidate_obj`. These cases depend on `_invalidate_current` and go stale if it is reverted (ran):

   ```python
   import param
   from param import rx

   x, y = rx(1), rx(10)
   s = x + y
   s.rx.value
   with param.batch():
       y.rx.value = 20
       print(s.rx.value)
   # b2cdc65:                          21
   # b2cdc65 without unbatched=True:   11
   ```

   `rx(1) * p.param.v` with `p.v = 3` written in the batch behaves the same way: 3 on the PR head, 1 if reverted.
   Fix: add one of these as a test.

2. **Overridden nodes still return a stale value inside a batch** (`param/reactive.py:3481`, `_watch_override`). This is bug 1 again, for override refs, so plain refs and overrides now behave differently (ran):
   ```python
   import param
   from param import rx

   c = rx(1)
   b = rx(10) * rx(0)
   b.rx.overrides[1] = c
   with param.batch():
       c.rx.value = 2
       print(b.rx.value)
   # b2cdc65:                                    10
   # b2cdc65 + unbatched=True at line 3481:      20
   ```
   The PR leaves `_invalidate_overrides` deferred because it "triggers the override channels". But `channel.param.trigger('value')` is a parameter event on a Parameterized, so the batch defers it anyway. The only part that would run eagerly is `_notify_graph_change()`, which updates the weak listeners behind `.rx.updating()`. That is bookkeeping too.
   Fix: pass `unbatched=True` there as well and add the test. If running `_notify_graph_change` eagerly is a concern, separate marking the node dirty from notifying.

### Nice to have

3. **Tests are in the wrong class** (`tests/testreactive.py:6630-6667`). They are in `TestSkipSiblingArguments`, which is about skipped sibling arguments. A batch-specific class in `testreactive.py` would fit better. `TestBatch` in `tests/testwatch.py` has no `rx` usage.
4. **The PR description says `batch_call_watchers`, `.param.update()` and `discard_events` changed, but no test covers them.** For `discard_events`: on the PR head, setting `p.x = 7` inside `discard_events` makes `(p.param.x.rx() * 2).rx.value` read 14 (ran). On base it reads 2 and stays 2 until the next change. In both cases the watch is not called. One test would pin this down.
5. **Docs and changelog.** The `batch()` docstring (`param/parameterized.py` ~595) only talks about deferring notifications. One sentence saying that `rx` values read inside the block include writes already made would help. The `discard_events` change is visible to users, so it deserves a release note.
6. The `_unbatched` comment (`param/reactive.py:2454-2457`) describes what the flag does more than why. Optional.

### Set aside

- Reading through a parameter that follows a ref (`F(v=a)`, then `f.param.v.rx()`) inside a batch still gives the pre-write value. `_sync_refs` is a notification, and the PR deliberately keeps notifications deferred.
- An `.rx.watch` fires `[2]` after a value goes 1→2→1 within one batch. Base behaves the same, so this diff didn't cause it.
- In the probe, async nodes showed `UNDEFINED` before their first read, on both base and head. This comes from how the probe ran the executor, not from the PR.
- Whether `discard_events` should leave `rx` caches stale on purpose. The review assumes that following the source is what users expect, and only asks for a release note (item 5).

Comment thread param/parameterized.py Resolved
… type

Override refs went stale inside a batch just like plain refs did, so
_invalidate_overrides is now unbatched too; the channel trigger it fires is
still deferred by the batch. _WeakInvalidator moves to parameterized.py so
_call_watcher can check isinstance instead of trusting an attribute on an
arbitrary watcher callable. Tests move to a TestBatch class and cover
_invalidate_current, overrides and discard_events.

Assisted-by: Claude Code:claude-opus-5-5
@philippjfr

philippjfr commented Oct 30, 2026 •

Copy link
Copy Markdown
Member Author

Thanks! Addressed in f1b7576:

  • Override refs are now invalidated at write time as well (_invalidate_overrides passes unbatched=True). The override channel trigger is a parameter event, so the batch still defers it, and .rx.watch on an overridden node still fires once after the batch.
  • Added tests for the _invalidate_current path (x + y and rx(1) * p.param.v), for overrides, and for discard_events. Reverting the unbatched flags makes the operand, parameter and override tests fail.
  • Moved the batch tests into a new TestBatch class.
  • Added a sentence to the batch() docstring saying that only notifications are deferred, and tightened the _unbatched comment.

I left the release note for when the release notes are written, since doc/releases.md is only updated at release time.

Only dirtying the node now happens at write time; notifying the override
channels is an ordinary watcher again, so .param.update(),
batch_call_watchers and discard_events no longer let a consumer on another
object see a new override combined with an input not yet flushed.

Assisted-by: Claude Code:claude-opus-5-5
Assisted-by: Claude Code:claude-opus-5-5
@philippjfr
philippjfr merged commit 74d0fe1 into main Oct 30, 2026
19 checks passed
@philippjfr
philippjfr deleted the fix/rx-batch-eager-invalidation branch October 30, 2026 12:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants