Skip to content

5EEb2pgf7pCFsUDE1QZgnrzrCPL6FYoK1vfoRzzGRMBgTzxi improve harness - #1557

Open
antiscj007 wants to merge 1 commit into
unarbos:mainfrom
antiscj007:main
Open

antiscj007 wants to merge 1 commit into
unarbos:mainfrom
antiscj007:main

Conversation

@antiscj007

Copy link
Copy Markdown

improve harness

@github-actions

Copy link
Copy Markdown

OpenRouter PR Judge

Verdict: PASS
Model: anthropic/claude-opus-4.7
Threshold: 70

Score Value
Overall 72
Real edit 75
Safety 90
Scope 85
Contract 95

Summary

This PR makes several mechanical additions to agent.py: a sed-to-python rewriter in extract_commands, new patch sanitizers (mode-metadata lines, lockfile diffs, build cache directories, .map files), a TypeScript compilation refinement gate (MAX_TS_COMPILE_TURNS) using tsc --noEmit, a thin-patch budget nudge when most time is spent on read-only steps, language-aware directory sibling preloading, hardcoded-secret/credential advisory in the self-check prompt, acceptance-criteria checklist injection, functional-area gap detection (CI/Docker), a second coverage nudge if patch unchanged, large-file skip in preload, and a sed-failure observation hint. Plus disabling git core.fileMode.

Static Checks

  • No static findings.

Judge Reasons

  • Adds real mechanical changes: command rewriting, new refinement gate (tsc), new sanitizers, new budget-pressure nudge, edit-vs-read step tracking
  • Preserves solve() signature and return shape; multishot wrapper unchanged
  • No new third-party imports, no new env vars, no new sampling parameters
  • DANGEROUS_PATTERNS not weakened
  • Inference still routed through validator-supplied api_base/api_key
  • Hardcoded-pattern advisory targets generated patch content, not validator-side scoring
  • MAX_TOTAL_REFINEMENT_TURNS still capped; MAX_COVERAGE_NUDGES raised to 2 with patch-unchanged guard
  • The acceptance-criteria checklist insertion into self-check prompt is prompt-tuning paired with real mechanical changes elsewhere

Risks

  • scope-drift: hardcoded patterns like curDay look benchmark-specific and resemble memorization of expected validator inputs (minor)
  • goodhart: the new _strip_lockfile_diffs and _strip_mode_metadata_lines could be patch-shaping toward Cursor-baseline style rather than fixing the issue, though stripping lockfiles is a defensible cleanup
  • obfuscation: minor — some new helpers use single-letter locals (m, fm, fm2) but readability is overall preserved

Required Changes

  • No required changes returned.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant