Skip to content

[P1] reproduce: cleanup deletes unrelated files in the user-supplied work directory #361

Description

@reacher-z

Problem / impact

clawbench-reproduce --work-dir <existing-directory> recursively deletes the entire supplied directory after comparison unless --keep-cache is supplied. The directory need not have been created by this invocation, so unrelated user files are removed along with downloaded traces.

Source: reproduce.py. The code calls shutil.rmtree(args.work_dir, ignore_errors=True) without tracking ownership. mkdir(..., exist_ok=True) earlier accepts an existing directory.

Verified reproduction (offline, temporary directory only)

  1. Create a temporary user-work/ with unrelated-user-file.txt and downloaded-traces/.
  2. Mock download to return downloaded-traces/ and rescore to return a matching 129-task summary; invoke main with --model deepseek-v4-flash --work-dir user-work.
  3. Actual: exit 0; both the unrelated file and user-work/ no longer exist.

No real data, downloads, or judge API calls were used for verification.

Acceptance criteria

  • Allocate a unique invocation-owned child directory beneath the chosen work directory; remove only that owned child.
  • Preserve pre-existing files and other runs' caches on PASS, FAIL, exceptions and cancellation.
  • Keep --keep-cache behavior explicit, and report the exact owned cache path.
  • Add a regression using a sentinel sibling file plus an existing work directory; verify it survives normal completion and failures.

Suggested priority P1: user data loss. Separate from #296 / #313 (rescore path defaults).

@Perry2004 please review the cleanup ownership contract and prioritize this fix.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions