feat(tier3): rebuild ATIF from OpenCode and Codex agent logs - #119
feat(tier3): rebuild ATIF from OpenCode and Codex agent logs#119mimran-khan wants to merge 15 commits into
Conversation
Parse OpenCode JSON stream events into synthetic trajectories when trajectory.json is empty, and reuse the parser for structured Codex tee logs. Fail closed on non-dict tool inputs. Fixes NVIDIA#118 Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>
Codex tee logs use type=item / agent_message events, not OpenCode tool_use. Add a dedicated parser with stderr-tolerant JSONL scanning, keep OpenCode-shaped JSONL as a fallback, and normalize shell calls to bash for skill checks. Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>
|
Codex parser fixes are pushed from the earlier round. Ready for re-review when you have time. |
rng1995
left a comment
rng1995
left a comment
There was a problem hiding this comment.
The current head still does not reconstruct real Codex exec events, and the new OpenCode fallback is not connected to the collected-results discovery path. Focused converter and refinement tests passed (95 tests), with Ruff and diff checks clean. Requesting changes for the two reproducible integration gaps noted in the review threads.
…ries Merge main and handle Codex exec JSON item.completed events with agent_message text and command_execution items. Include opencode in trajectory discovery so refinement can rebuild ATIF from opencode.txt when trajectory.json is missing.
…an-khan/SkillEvaluator into feat/opencode-atif-fallback
| steps.append(step) | ||
| continue | ||
|
|
||
| if item_type == "command_execution": |
There was a problem hiding this comment.
[P1] Preserve the other completed Codex action items. The Codex 0.153.0 ThreadItem schema also emits file_change and mcp_tool_call, but this parser stops at agent messages and command_execution. A real 0.153.0 file-write run contained a completed file_change while this function produced no write call; a file/MCP write can therefore disappear from downstream file-change evidence, behavior grading, and trace-level security checks. Please map those action items into ATIF calls/observations, preserve status/error and command exit_code, and add captured current-CLI fixtures.
There was a problem hiding this comment.
Added parsing for Codex file_change and mcp_tool_call completed items in synthetic_trajectory_from_codex_json, with tests in test_log_converters.py. This sits alongside the existing command_execution and agent_message handling.
There was a problem hiding this comment.
The new file_change and mcp_tool_call types are mapped, but the requested terminal evidence is still dropped. On this head, a completed command_execution with command=false, exit_code=1, status=failed, and empty aggregated_output becomes a bash call with an empty observation and no exit code or status; file changes likewise omit status/error, and non-object MCP errors are ignored. Downstream grading cannot distinguish these failures from successful empty-output actions. Please preserve terminal status, exit code, and error evidence and add failure fixtures before this thread is resolved.
|
|
||
|
|
||
| def _opencode_output_text(state: dict[str, Any]) -> str: | ||
| output = state.get("output") |
There was a problem hiding this comment.
[P2] Keep failed OpenCode tool outcomes in the observation. OpenCode error events put the failure text in state.error rather than state.output, so this helper returns an empty string and the synthetic ATIF records the attempted call with no indication it failed. In a retained real OpenCode log, all 6 status=error calls became empty observations, leaving downstream recovery and accuracy evidence unable to distinguish failure from a successful call with no output. Please fall back to state.error when output is absent, retain the terminal status, and add an error-state fixture.
There was a problem hiding this comment.
_opencode_output_text now falls back to state.error when output is empty, so failed tool calls keep their failure text in the observation. Covered by test_opencode_tool_error_uses_state_error_observation.
There was a problem hiding this comment.
@chrisknvidia @rng1995 All three Sep 3 items are addressed at 05f7b62 (main merged, including #111). Fork CI needs approval before checks run. Ready for re-review when you have time.
There was a problem hiding this comment.
The error-text fallback works only when output is absent, but terminal status is still not retained. With state.status=error plus partial output and state.error, this head records only the partial output; both the error and failed status disappear. Please preserve the terminal status and error alongside any output, then cover this mixed-output failure case.
Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>
Resolve trial case ids from reward/result metadata before refine lookup. Parse Codex file_change and mcp_tool_call events, and keep OpenCode tool failures in observations via state.error. Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>
Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>
rng1995
left a comment
rng1995
left a comment
There was a problem hiding this comment.
Three earlier integration issues are fixed and their threads are resolved. The focused converter/refinement suite passes (36 tests), with Ruff and diff checks clean. Two correctness gaps remain: Codex terminal status, exit code, and some error forms are discarded, and OpenCode failures that include partial output lose both failed status and error details. These make failed actions indistinguishable from successful empty or partial-output actions in downstream evidence, so I am requesting changes on the two remaining threads.
Preserve exit code, status, and error evidence for failed command runs, file changes, and MCP calls, and keep OpenCode error text alongside partial output for downstream grading. Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>
Keep Harbor trial discovery tests from main and restore OpenCode trajectory fixtures from this branch. Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>
|
Pushed b4644da with main merged in. Codex command_execution now keeps status, exit_code, and errors even when command is false or output is empty; file_change and MCP calls get the same terminal evidence, and OpenCode keeps status plus error text alongside partial output. Harbor trial discovery tests from main are in, plus the OpenCode trajectory fixtures on this branch. Happy to re-test anything specific. |
3 participants
Fixes #118
When Harbor leaves
trajectory.jsonempty, Tier 3 scoring used to fail with "No trajectory or reconstructible agent log" even though OpenCode and Codex tee structured JSON toopencode.txtandcodex.txt.This adds parsers for OpenCode
run --format=jsonevents (text + completedtool_userecords) and wires them intoload_trajectory_with_fallbackafter the existing claude/cursor/cline fallbacks. Codex reuses the same parser when every tee line is JSON. Non-dict tool inputs are ignored so raw strings are not treated as shell commands.Harbor task bundles already copy
log_converters.pythroughadapter.py, so generated verifiers pick this up automatically.Distinct from #110, which is about exec steps inside an existing trajectory.
Test plan
pytest tests/tier3/test_log_converters.pypytest tests/test_harbor_collector_runtime_failures.py