Test/eval framework#6572
Open
shwetapsdet wants to merge 37 commits into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a Python-based evaluation harness for Obot “nanobot” workflows, supporting both live API/MCP/SSE runs against a running Obot instance and static DeepEval runs against JSON fixtures, plus a GitHub Actions workflow to run these evals in CI.
Changes:
- Introduces an
eval_pyPython package implementing a small eval framework, Obot REST/MCP clients, SSE capture/dedup utilities, and DeepEval-based scoring (static + per-turn). - Adds curated static fixtures and a CLI (
python -m eval.run,python -m eval.agent_deepeval,python -m eval.agent_deepeval_generic) for running evals locally and in CI. - Adds a scheduled + manually-invoked GitHub Actions workflow to run static evals daily and optionally run live conversation evals in a container.
Reviewed changes
Copilot reviewed 24 out of 24 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| eval_py/requirements.txt | Adds Python deps for the eval framework (requests + deepeval). |
| eval_py/README.md | Documents setup, env vars, cases, CLI usage, artifacts, and CI workflow behavior. |
| eval_py/.gitignore | Ignores local venv/cache and DeepEval output dirs. |
| eval_py/eval/init.py | Exposes core eval types and helpers as a package API. |
| eval_py/eval/run.py | CLI entrypoint for running live eval cases and writing summaries. |
| eval_py/eval/core/framework.py | Defines Case/Context/Result + runner that wraps API logging lifecycle. |
| eval_py/eval/core/cases.py | Implements live conversation evals, SSE persistence, and a blog-post elicitation SSE assertion. |
| eval_py/eval/core/run_summary.py | Writes JSON/text run summaries and optionally appends GitHub job summaries. |
| eval_py/eval/core/init.py | Core package exports. |
| eval_py/eval/clients/client.py | REST client for Obot endpoints (version/projects/agents/MCP connect URL). |
| eval_py/eval/clients/mcp_client.py | MCP JSON-RPC + SSE event-stream collection client. |
| eval_py/eval/clients/init.py | Client package exports. |
| eval_py/eval/helper/paths.py | Centralizes path resolution for eval/data/. |
| eval_py/eval/helper/event_stream_data.py | Persists SSE output and writes step-eval output files + “distinct” SSE variants. |
| eval_py/eval/helper/api_log.py | Optional API request/response logging to a file. |
| eval_py/eval/helper/init.py | Helper package exports. |
| eval_py/eval/workflow/workflow_prompt.py | Defines conversation prompts/turn criteria for workflow evals. |
| eval_py/eval/workflow/init.py | Workflow module exports. |
| eval_py/eval/agent_deepeval.py | Scenario-specific DeepEval runners for curated static fixtures and latest run. |
| eval_py/eval/agent_deepeval_generic.py | Scenario-agnostic DeepEval runner + per-turn criteria evaluation helper. |
| eval_py/eval/data/python_review_1.json | Static fixture for python_code_review workflow eval. |
| eval_py/eval/data/deep_news_briefing_1.json | Static fixture for deep_news_briefing workflow eval. |
| eval_py/eval/data/antv_charts_1.json | Static fixture for antv_dual_axes_viz workflow eval. |
| .github/workflows/nanobot-python-evals.yml | CI workflow for static scheduled evals + optional live container-based evals and artifacts. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+41
to
+46
| def __init__(self, mcp_url: str, auth_header: str): | ||
| self.mcp_url = mcp_url.rstrip("/") | ||
| self.auth_header = auth_header | ||
| self._session = requests.Session() | ||
| self._session.timeout = MCP_TIMEOUT | ||
|
|
Comment on lines
+63
to
+70
| req_body = json.dumps(body).encode() | ||
| headers = {"Content-Type": "application/json", "Accept": "application/json"} | ||
| if session_id: | ||
| headers["Mcp-Session-Id"] = session_id | ||
| _apply_auth(headers, self.auth_header) | ||
| resp = self._session.post(url, data=req_body, headers=headers) | ||
| out = resp.content | ||
| api_log.log_api_call("POST", url, req_body, resp.status_code, out) |
Comment on lines
+17
to
+26
| def init_api_log(path: str) -> None: | ||
| global _file | ||
| if not path: | ||
| return | ||
| with _lock: | ||
| if _file is not None: | ||
| return | ||
| _file = open(path, "a", encoding="utf-8") | ||
| _file.write("\n=== API log started " + time.strftime("%Y-%m-%dT%H:%M:%S%z") + " ===\n\n") | ||
| _file.flush() |
Comment on lines
+46
to
+73
| if f is None: | ||
| return | ||
| ts = time.strftime("%H:%M:%S.000", time.localtime()) | ||
| f.write("[%s] %s %s\n" % (ts, method, url)) | ||
| if req_body: | ||
| f.write(_truncate(req_body, _MAX_LOG_BODY) + "\n") | ||
| f.write(" -> %d (%d bytes)\n" % (status, len(resp_body))) | ||
| if resp_body: | ||
| f.write(_truncate(resp_body, _MAX_LOG_BODY) + "\n\n") | ||
| else: | ||
| f.write("\n") | ||
| f.flush() | ||
|
|
||
|
|
||
| def log_api_stream_response(url: str, resp_body: bytes) -> None: | ||
| """Append event-stream response for a GET that was already logged (stream kept open).""" | ||
| with _lock: | ||
| f = _file | ||
| if f is None: | ||
| return | ||
| ts = time.strftime("%H:%M:%S.000", time.localtime()) | ||
| f.write("[%s] GET %s [event-stream response]\n" % (ts, url)) | ||
| f.write(" -> 200 (%d bytes)\n" % len(resp_body)) | ||
| if resp_body: | ||
| f.write(_truncate(resp_body, _MAX_LOG_BODY) + "\n\n") | ||
| else: | ||
| f.write("\n") | ||
| f.flush() |
Comment on lines
+1
to
+5
| """Content publishing workflow prompts. | ||
|
|
||
| WordPress MCP must be configured with a user that has Editor or Administrator role | ||
| for create/post/publish to work; Application Passwords inherit the user's capabilities. | ||
| Read-only errors in the workflow usually mean the WordPress user role is too limited. |
Comment on lines
+396
to
+398
| print("response_text: ", response_text) | ||
| print("raw_sse: ", raw_sse) | ||
| print("tools_used: ", tools_used) |
| @@ -0,0 +1,2 @@ | |||
| requests>=2.28.0 | |||
| deepeval | |||
Comment on lines
+239
to
+245
| def _build_deepeval_test_case(trace: AgentTrace) -> LLMTestCase: | ||
| """Create a **generic** LLMTestCase for DeepEval (process-focused).""" | ||
| tool_summary = f"Tools used in order: {trace.tool_calls}" | ||
| search_summary = f"Search queries (from notifications): {trace.search_queries}" | ||
| distinct_tools = sorted(set(trace.tool_calls)) | ||
| tool_counts = {t: trace.tool_calls.count(t) for t in distinct_tools} | ||
| error_summary = f"Tool errors (by name): {trace.tool_error_counts}" |
Comment on lines
+68
to
+74
| last_block_by_id: dict[str, list[str]] = {} | ||
| last_created_by_id: dict[str, str] = {} | ||
| for eid, created, block_lines in blocks: | ||
| if eid is not None: | ||
| if eid not in last_created_by_id or (created and created > last_created_by_id[eid]): | ||
| last_created_by_id[eid] = created or "" | ||
| last_block_by_id[eid] = block_lines |
Comment on lines
+41
to
+46
| def __init__(self, mcp_url: str, auth_header: str): | ||
| self.mcp_url = mcp_url.rstrip("/") | ||
| self.auth_header = auth_header | ||
| self._session = requests.Session() | ||
| self._session.timeout = MCP_TIMEOUT | ||
|
|
| if session_id: | ||
| headers["Mcp-Session-Id"] = session_id | ||
| _apply_auth(headers, self.auth_header) | ||
| resp = self._session.post(url, data=req_body, headers=headers) |
Comment on lines
+68
to
+73
| last_block_by_id: dict[str, list[str]] = {} | ||
| last_created_by_id: dict[str, str] = {} | ||
| for eid, created, block_lines in blocks: | ||
| if eid is not None: | ||
| if eid not in last_created_by_id or (created and created > last_created_by_id[eid]): | ||
| last_created_by_id[eid] = created or "" |
Comment on lines
+396
to
+398
| print("response_text: ", response_text) | ||
| print("raw_sse: ", raw_sse) | ||
| print("tools_used: ", tools_used) |
Comment on lines
+94
to
+96
| - name: Run live conversation evals (one batch + combined summary) | ||
| working-directory: eval_py | ||
| run: python -m eval.run nanobot_python_code_review_conversation_eval |
Comment on lines
+221
to
+245
| # Deduplicate tools in order | ||
| seen = set() | ||
| ordered_tools: List[str] = [] | ||
| for name in tool_calls: | ||
| if name not in seen: | ||
| seen.add(name) | ||
| ordered_tools.append(name) | ||
|
|
||
| return AgentTrace( | ||
| user_prompt=user_prompt, | ||
| final_report=final_report, | ||
| tool_calls=ordered_tools, | ||
| search_queries=search_queries, | ||
| tool_error_counts=tool_error_counts, | ||
| assistant_step_summaries=assistant_step_summaries, | ||
| ) | ||
|
|
||
|
|
||
| def _build_deepeval_test_case(trace: AgentTrace) -> LLMTestCase: | ||
| """Create a **generic** LLMTestCase for DeepEval (process-focused).""" | ||
| tool_summary = f"Tools used in order: {trace.tool_calls}" | ||
| search_summary = f"Search queries (from notifications): {trace.search_queries}" | ||
| distinct_tools = sorted(set(trace.tool_calls)) | ||
| tool_counts = {t: trace.tool_calls.count(t) for t in distinct_tools} | ||
| error_summary = f"Tool errors (by name): {trace.tool_error_counts}" |
| """HTTP client for Obot nanobot APIs (projectsv2, agents, launch, version).""" | ||
| import json | ||
| from typing import Any, Optional | ||
| from urllib.parse import urlparse |
Comment on lines
+6
to
+11
| import json | ||
| import os | ||
| import time | ||
| from dataclasses import dataclass, field | ||
| from typing import TYPE_CHECKING, Callable, Optional | ||
|
|
Comment on lines
+284
to
+326
| response, and finally writes distinct SSE to python_review.txt. | ||
| """ | ||
| before = _event_stream_response_count() | ||
| os.environ["OBOT_EVAL_CONVERSATION_WORKFLOW"] = "python_code_review" | ||
| result = run_workflow_conversation_eval(ctx) | ||
| _update_static_trace_from_latest_sse( | ||
| case_name="python_code_review", | ||
| txt_filename="python_review.txt", | ||
| start_index=before, | ||
| ) | ||
| return result | ||
|
|
||
|
|
||
| def run_deep_news_briefing_conversation_eval(ctx: Context) -> Result: | ||
| """ | ||
| Convenience wrapper to run the deep_news_briefing workflow conversation eval. | ||
| Also writes distinct SSE to news.txt. | ||
| """ | ||
| before = _event_stream_response_count() | ||
| os.environ["OBOT_EVAL_CONVERSATION_WORKFLOW"] = "deep_news_briefing" | ||
| result = run_workflow_conversation_eval(ctx) | ||
| _update_static_trace_from_latest_sse( | ||
| case_name="deep_news_briefing", | ||
| txt_filename="news.txt", | ||
| start_index=before, | ||
| ) | ||
| return result | ||
|
|
||
|
|
||
| def run_antv_dual_axes_conversation_eval(ctx: Context) -> Result: | ||
| """ | ||
| Convenience wrapper to run the antv_dual_axes_viz workflow conversation eval. | ||
| Also writes distinct SSE to antv_charts.txt. | ||
| """ | ||
| before = _event_stream_response_count() | ||
| os.environ["OBOT_EVAL_CONVERSATION_WORKFLOW"] = "antv_dual_axes_viz" | ||
| result = run_workflow_conversation_eval(ctx) | ||
| _update_static_trace_from_latest_sse( | ||
| case_name="antv_dual_axes_viz", | ||
| txt_filename="antv_charts.txt", | ||
| start_index=before, | ||
| ) | ||
| return result |
| __pycache__/ | ||
| *.py[cod] | ||
| .eval/ | ||
| .deepeval/ |
Comment out the static DeepEval job and run all three nanobot conversation workflows in the live job via nanobot_conversation_workflows. Fix AntV multi-turn eval reliability: use per-phase SSE prompt anchors, stop overwriting turn-isolated assistant text in DeepEval, and clarify Phase 2 chart-configuration prompts. Co-authored-by: Cursor <cursoragent@cursor.com>
Run every case even if one throws or fails. Exit non-zero only when all cases fail. Surface overall_pass in run summaries and GitHub job summary. Co-authored-by: Cursor <cursoragent@cursor.com>
…ework Test/eval framework
Use nanobot_conversation_workflows (not a single case) and pull the latest Obot image on each run. Co-authored-by: Cursor <cursoragent@cursor.com>
Add unique [[ANTV_EVAL:P*]] SSE anchors, prefix matching for short prompts, SSE re-extraction per turn, and a chart-configuration heuristic with GEval fallback for Phase 2 when the agent repeats dataset validation. Co-authored-by: Cursor <cursoragent@cursor.com>
Store assistant_response on TurnEvalDetail, emit it in eval_run_summary.json and the GitHub job summary when a turn fails, plus the text summary file. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
JSON always pairs prompt with LLM response per turn. GitHub job summary still shows LLM output only for failed turns; text artifact matches JSON. Co-authored-by: Cursor <cursoragent@cursor.com>
Comment out the static DeepEval job and run all three nanobot conversation workflows in the live job via nanobot_conversation_workflows. Fix AntV multi-turn eval reliability: use per-phase SSE prompt anchors, stop overwriting turn-isolated assistant text in DeepEval, and clarify Phase 2 chart-configuration prompts. Co-authored-by: Cursor <cursoragent@cursor.com>
Run every case even if one throws or fails. Exit non-zero only when all cases fail. Surface overall_pass in run summaries and GitHub job summary. Co-authored-by: Cursor <cursoragent@cursor.com>
Use nanobot_conversation_workflows (not a single case) and pull the latest Obot image on each run. Co-authored-by: Cursor <cursoragent@cursor.com>
Add unique [[ANTV_EVAL:P*]] SSE anchors, prefix matching for short prompts, SSE re-extraction per turn, and a chart-configuration heuristic with GEval fallback for Phase 2 when the agent repeats dataset validation. Co-authored-by: Cursor <cursoragent@cursor.com>
Store assistant_response on TurnEvalDetail, emit it in eval_run_summary.json and the GitHub job summary when a turn fails, plus the text summary file. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
JSON always pairs prompt with LLM response per turn. GitHub job summary still shows LLM output only for failed turns; text artifact matches JSON. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.