Skip to content

Vectorise followed_rule() for mtp = FALSE - #192

Open
peterbruun wants to merge 1 commit into
nt-williams:develfrom
peterbruun:fast-followed-rule
Open

peterbruun wants to merge 1 commit into
nt-williams:develfrom
peterbruun:fast-followed-rule

Conversation

@peterbruun

Copy link
Copy Markdown

Thank you for developing and maintaining lmtp. It is a great and very impressive package, and it has made it possible for us to estimate longitudinal causal effects in a large clinical cohort in a principled way. This PR is a small contribution back.

Summary

With mtp = FALSE, followed_rule() decides for every observation whether the
observed treatment equals the shifted treatment. It does this one observation at a
time with mapply(function(x, y) isTRUE(all.equal(x, y)), ...). Because
followed_rule() is called for every time point and every fold, this element-wise
all.equal() becomes the main cost of the density ratio estimation for large data
with deterministic (dynamic) interventions.

This PR replaces it with a vectorised comparison that has the same semantics as the
element-wise all.equal():

  • numeric: equal if the relative difference is at most sqrt(.Machine$double.eps)
    (an absolute difference when |x| <= tolerance or x is not finite), as in
    all.equal.numeric();
  • factor: same class and levels, then the same value;
  • character: the same value;
  • missing values: NA equals NA, and NA never equals a value.

Other column types (e.g. logical) fall back to the previous element-wise
all.equal(), so their behaviour is unchanged. The row-wise
apply(followed, 1, prod) is replaced by rowSums(), which returns the same 0/1
numeric vector.

Tests

tests/testthat/test-followed_rule.R compares the new implementation with the
previous one (copied into the test file as the reference) using expect_identical()
for:

  • binary, continuous and integer treatments, including values just inside and outside
    the tolerance, values near zero, and NA;
  • Inf, -Inf, NaN and NA;
  • factors with the same and with different levels, and character treatments with NA;
  • logical treatments (the fallback path);
  • several treatment columns;
  • mtp = TRUE;
  • 1,000 random observations.

Benchmark

One call of followed_rule(), binary treatment, 10% of observations not following,
R 4.6.2. Results were identical in every case.

n treatment columns before (s) after (s) speed-up
10,000 1 0.079 0.001 ~80x
100,000 1 0.813 0.003 ~270x
100,000 2 1.569 0.006 ~260x
500,000 1 4.211 0.018 ~230x

followed_rule() is called once per time point and fold, so for long follow-up
these savings add up. For example, 28 time points × 5 folds × several strategies
means hundreds of calls.

We noticed this in a target trial emulation with about 10,000 patients, 28 daily
time points and deterministic threshold strategies (mtp = FALSE). There,
followed_rule() dominated the fitting time. We have used the vectorised version
(patched in with assignInNamespace()) for all analyses, after checking that it
matched the original on the real data for every strategy and time point.

The NEWS entry is under 1.5.5 / General.

This PR was prepared with the help of Claude Code.

followed_rule() compared the natural and shifted treatment one observation
at a time with mapply(all.equal), which dominates the density ratio
estimation time for large data with deterministic (mtp = FALSE)
interventions. The comparison is now vectorised for numeric, factor and
character treatments, with the same semantics as the element-wise
all.equal() (relative tolerance sqrt(.Machine$double.eps), absolute when
|x| <= tolerance; NA equals NA; factors must share class and levels).
Other types fall back to the previous element-wise all.equal().

Adds tests comparing the new implementation with the previous one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant