[Enhancement] Stop looping when runtime is stable (CF-934)

[mohammedahmed18]

PR Type

Enhancement, Tests

---

Description

• Add consistent-loop break based on durations

• Track per-loop test durations via hook

• Introduce dynamic tolerance by runtime scale

• New config: CONSISTENT_LOOP_COUNT

---

Diagram Walkthrough

flowchart LR
  CFG["Add CONSISTENT_LOOP_COUNT const"]
  HOOK["pytest_runtest_logreport collects durations"]
  LOOP["runtestloop aggregates loop durations"]
  TOL["dynamic_tolerance by avg runtime"]
  BREAK["Break when CONSISTENT_LOOP_COUNT consistent"]

  CFG -- "config value" --> LOOP
  HOOK -- "per-test duration" --> LOOP
  LOOP -- "median compare" --> TOL
  TOL -- "consistency check" --> BREAK

Loading

File Walkthrough

	Relevant files
Enhancement	config_consts.py Introduce loop consistency count constant                                codeflash/code_utils/config_consts.py Add CONSISTENT_LOOP_COUNT default to 3. Document intent alongside timing configs. +1/-0      pytest_plugin.py Duration-based consistent loop termination logic                  codeflash/verification/pytest_plugin.py Track passed test durations per loop. Add dynamic_tolerance based on avg runtime. Break loop after N consistent total durations. Use deque with CONSISTENT_LOOP_COUNT. +50/-5
Formatting	env_utils.py Minor formatting cleanup                                                                  codeflash/code_utils/env_utils.py Minor cleanup: remove stray blank line. +0/-1

---

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Hardcoded Path Writing a file to a hardcoded absolute path under the user's home directory can fail on different environments and pollutes user filesystems; consider using a temp dir, pytest's artifacts, or a configurable path and add error handling. Path(f"/home/mohammed/Documents/test-results/break-{session.name}.txt").write_text(str(count)) break Loop Timeout Logic The updated loop no longer refreshes total_time within the loop and relies solely on timed_out/start_time; verify that SHORTEST_AMOUNT_OF_TIME and timed_out still guarantee at least one run and correct termination across configs. while total_time >= SHORTEST_AMOUNT_OF_TIME: count += 1 self.current_loop_durations_in_seconds.clear() for index, item in enumerate(session.items): Zero-Duration Handling Clearing durations when the total duration is zero may mask scenarios with flaky fast tests; consider tracking zeros explicitly or logging a reason to avoid unintended endless loops. if total_duration_in_seconds > 0: durations.append(total_duration_in_seconds) else: durations.clear()

[github-actions]

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
Possible issue	Remove hard-coded file path Avoid writing to a hard-coded, user-specific absolute path which can fail on other environments and introduces side effects during tests. Use a temp directory under the pytest cache dir to persist optional diagnostics, and guard it behind a debug/verbose flag. codeflash/verification/pytest_plugin.py [348] -Path(f"/home/mohammed/Documents/test-results/break-{session.name}.txt").write_text(str(count)) +if session.config.option.verbose > 1: + out_dir = Path(session.config.rootpath, ".pytest_cache", "codeflash") + out_dir.mkdir(parents=True, exist_ok=True) + (out_dir / f"break-{session.name}.txt").write_text(str(count)) Suggestion importance[1-10]: 8 __ Why: Replacing a hard-coded absolute path with a cache-directory path guarded by verbosity prevents environment-specific failures and unintended side effects, and the improved_code correctly reflects this change at the referenced line.	Medium
General	Preserve stability history across loops Clearing durations on a zero-duration loop resets accumulated stability history and can cause non-termination flaps under very fast tests. Instead, append zero and let the consistency check handle zeros; only clear durations when test items were not executed. codeflash/verification/pytest_plugin.py [314-339] self.current_loop_durations_in_seconds.clear() ... total_duration_in_seconds = sum(self.current_loop_durations_in_seconds) -if total_duration_in_seconds > 0: +if session.items: durations.append(total_duration_in_seconds) else: durations.clear() Suggestion importance[1-10]: 6 __ Why: The suggestion reasonably avoids clearing durations on zero totals to reduce flapping and updates logic in the correct region, though it slightly changes semantics and assumes session.items presence as the clearing condition.	Low
	Harden consistency evaluation Guard against division by zero and ensure dynamic_tolerance is only used with positive avg; also handle NaN durations defensively. Short-circuit consistency evaluation if any duration is non-finite. codeflash/verification/pytest_plugin.py [312-353] while total_time >= SHORTEST_AMOUNT_OF_TIME: count += 1 self.current_loop_durations_in_seconds.clear() ... if len(durations) == CONSISTENT_LOOP_COUNT: - avg = statistics.median(durations) - if avg == 0: - consistent = all(d == 0 for d in durations) - else: - consistent = all(abs(d - avg) / avg <= self.dynamic_tolerance(avg) for d in durations) - if consistent: - Path(f"/home/mohammed/Documents/test-results/break-{session.name}.txt").write_text(str(count)) - break + if all((d == 0 or (d > 0 and d != float("inf"))) for d in durations): + avg = statistics.median(durations) + if avg <= 0: + consistent = all(d == 0 for d in durations) + else: + tol = self.dynamic_tolerance(avg) + consistent = all(abs(d - avg) / avg <= tol for d in durations if d >= 0) + if consistent: + if session.config.option.verbose > 1: + out_dir = Path(session.config.rootpath, ".pytest_cache", "codeflash") + out_dir.mkdir(parents=True, exist_ok=True) + (out_dir / f"break-{session.name}.txt").write_text(str(count)) + break - if self._timed_out(session, start_time, count): - break - Suggestion importance[1-10]: 5 __ Why: Adding guards for non-positive and non-finite durations is a defensive improvement and aligns with the surrounding code, but the original code already avoids divide-by-zero when avg == 0, so the impact is moderate.	Low

[aseembits93]

@mohammedahmed18 how did you set these numbers?

[mohammedahmed18]

@aseembits93
there are just experiment numbers, the whole idea is very small runtimes should have high tolerance value because the noise is relatively large