Skip to content

refactor: 평가 엔진을 benchmark.py로 통합, measure_recall.py는 CI 래퍼로 축소 - #30

Merged
dnjsgkfka merged 1 commit into
mainfrom
refactor/unify-benchmark-engine
Sep 17, 2026
Merged

dnjsgkfka merged 1 commit into
mainfrom
refactor/unify-benchmark-engine

Conversation

@dnjsgkfka

Copy link
Copy Markdown
Member

요약

벤치마크 평가 엔진을 하나로 통합

  • 이제 benchmark.py가 유일한 평가 엔진이고, measure_recall.py는
    DagsHub/MLflow 로깅만 담당하는 얇은 CI 래퍼

변경 사항

  • benchmark.py: PageHit@1/5/10(모든 arm 동일 기준), evidence_hit@1, TableHit@k(정답 표 자체 일치), ablation 3종, 문서단위 부트스트랩 CI + McNemar, 동일 토큰 예산 LLM 실답변 평가
  • measure_recall.py: benchmark.run()을 호출하는 CI 전용 래퍼로 재작성 (자체 채점 로직 제거)
  • scripts/summarize_results.py: 새 summary 스키마에 맞춰 Actions 요약 로직 갱신
  • .github/workflows/benchmark.yml: 트리거 경로에 benchmark.py 추가
  • 주석 정리

@dagshub

dagshub Bot commented Sep 17, 2026

Copy link
Copy Markdown

@dnjsgkfka dnjsgkfka added bug Something isn't working enhancement New feature or request and removed bug Something isn't working labels Sep 17, 2026
@dnjsgkfka
dnjsgkfka merged commit d50e934 into main Sep 17, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant