Skip to content
BranchesTags

Repository files navigation

COBOLEval: LLM Evaluation for COBOL

CI Python 3.10+ Rust License: MIT

This is a fork. COBOLEval — the benchmark, the 146-task dataset, and the Python-to-COBOL transpiler — is the work of bloop.ai, published at bloopai/COBOLEval and described in their blog post. It is a transpilation of OpenAI's HumanEval into COBOL. The original MIT license and copyright (bloop) are retained unchanged in LICENSE.

This fork, maintained by nitrotap, keeps that dataset and transpiler intact and adds tooling around them: JCL static analysis, a Rust agentic orchestrator, Ollama support, and packaging/CI. See What's new in this fork for exactly what changed.

COBOLEval is a benchmark dataset for evaluating the code generation abilities of Large Language Models on the COBOL programming language.

This repository contains:

  • Python to COBOL transpiler (Rust) — converts HumanEval tasks to COBOL (upstream)
  • Evaluation harness (Python) — tests LLM-generated COBOL code (upstream, extended here)
  • COBOLEval dataset — 146 COBOL programming tasks (upstream, byte-identical to bloop's)
  • JCL tooling (Python) — parser, validator, and linter for mainframe job control (this fork)
  • Agentic orchestrator (Rust) — tool-using LLM agent that writes, compiles, and fixes COBOL (this fork)

What's new in this fork

Everything below is additive. The benchmark data (data/CobolEval.jsonl) is unmodified from upstream.

Area Change
JCL tooling scripts/jcl/ — a JCL parser, validator, and linter with a CLI. Upstream has no JCL code at all.
Agentic orchestrator src/bin/eval_agent.rs + src/eval/ — a Rig-based agent with five tools that iteratively writes, compiles, and repairs COBOL solutions.
Ollama inference OllamaChat / run_ollama in scripts/generate.py, and --provider ollama in the orchestrator, for local models.
GnuCOBOL 3.1.x The Python harness no longer passes -fformat=variable, which 3.1.x does not accept. See the caveat — this covers the Python path only.
Conversation logging The orchestrator writes one JSONL file per task/sample recording every agent prompt, tool call, tool response, and agent reply.
Packaging & CI pyproject.toml, Dockerfile, .pre-commit-config.yaml, a GitHub Actions workflow, fire-based CLIs replacing upstream's edit-the-source-then-run flow, type hints, and a test suite (45 Python tests, 20 Rust tests).
Module split Upstream's single src/main.rs was split into types/parser/transpiler/constants, with clap argument parsing and unit tests.

JCL evaluation tooling

scripts/jcl/ performs static analysis only — it parses and validates JCL text. It does not execute, submit, or simulate mainframe jobs, and needs no z/OS or emulator. It is also independent of the COBOL benchmark: no JCL is involved in the pass@k pipeline.

The parser (parser.py) handles JOB/EXEC/DD/PROC/PEND/SET/IF/INCLUDE/JCLLIB/OUTPUT statements, comments, and continuations, and groups DD statements into steps. The validator (validator.py) emits coded diagnostics — name-length violations (JCL002, JCL011), missing PGM/PROC (JCL012), invalid DISP status (JCL060), unresolved backward references (JCL040), duplicate DD names (JCL031), and others — split into errors, warnings, and lint-level suggestions.

# Summarize a job: steps, programs, DD counts, validation totals
python -m scripts.jcl.cli info data/jcl/sample_batch.jcl

# Dump the parsed statement structure (text or JSON)
python -m scripts.jcl.cli parse data/jcl/sample_batch.jcl --output-format json

# Validate one file (exit 0 valid / 1 errors / 2 warnings under --strict)
python -m scripts.jcl.cli validate data/jcl/sample_with_errors.jcl

# Validation plus best-practice suggestions (REGION, COND, NOTIFY)
python -m scripts.jcl.cli lint data/jcl/sample_batch.jcl

# Validate every *.jcl in a directory
python -m scripts.jcl.cli batch data/jcl

Five sample JCL files ship in data/jcl/: a four-step batch pipeline, a COBOL compile-and-link job, a data load, a report generation job, and sample_with_errors.jcl, which contains deliberate faults for exercising the validator.

Agentic COBOL orchestrator

eval_agent runs an LLM agent over the benchmark using rig-core. Rather than sampling a completion once, the agent is given five tools and left to iterate:

Tool Purpose
read_task Fetch the COBOL skeleton and requirements for a task
submit_solution Submit or revise a candidate solution
compile_solution Compile with GnuCOBOL and return compiler errors
run_tests Compile with the test harness and report per-test expected vs. actual
get_task_status Report attempts used and current progress 
cargo build --release

# Local model via Ollama (the default provider)
./target/release/eval_agent --provider ollama --model llama3.2

# Hosted providers (note: the accepted value for OpenAI is `open-ai`)
./target/release/eval_agent --provider open-ai --model gpt-4o
./target/release/eval_agent --provider anthropic --model claude-3-5-sonnet-latest

Options, from eval_agent --help: --provider {ollama|open-ai|anthropic} (default ollama), --model (default llama3.2), --samples-per-task (1), --max-attempts (3), --temperature (0.0), --output-dir (./preds/eval_agent), --timeout (10s), --data-path (./data/CobolEval.jsonl), and --execute.

Provider credentials are read from the environment by rig (OPENAI_API_KEY, ANTHROPIC_API_KEY).

--execute is off by default. Without it, run_tests compiles the solution together with its test harness and reports whether compilation succeeded, but does not run the binary — so no test can pass and the reported pass@1 will be 0. Pass --execute only inside a sandbox; it runs untrusted model-generated code.

Outputs land under --output-dir:

preds/eval_agent/
├── solutions/            # <entry_point>.cbl per task
├── callers/              # generated test harnesses
├── logs/                 # per-task/sample conversation logs (see below)
└── samples_results.jsonl # one record per sample: completion, per-test pass/compile flags

Pass@1 is computed with the standard unbiased HumanEval estimator (src/eval/metrics.rs) and printed at the end of the run.

Per-task JSONL conversation logging

Each task/sample writes logs/<task_id>__sample_<k>.jsonl, one JSON object per line:

{"timestamp":"...","entry_type":"tool_call","tool_name":"compile_solution","input":"...","output":"..."}

entry_type is one of agent_prompt, tool_call, tool_response, agent_response. Task IDs are filename-sanitized (HumanEval/0 becomes HumanEval_0). This makes a run auditable after the fact — you can see exactly which compiler errors the model saw and how it responded.

Quick Start

Using Docker

docker build -t coboleval .

docker run -e OPENAI_API_KEY=$OPENAI_API_KEY coboleval \
    python -m scripts.generate openai --model-name gpt-4

The image installs GnuCOBOL and the Python package and copies in the cobol-eval-transpiler binary. It does not include the eval_agent binary; build that with cargo on the host.

Manual Installation

Prerequisites

  1. GnuCOBOL — required for COBOL compilation. The Python harness works with 3.1.x and later (see the compatibility caveat); the Rust orchestrator still requires 3.2.0+.

    # Ubuntu/Debian
sudo apt-get install gnucobol

# macOS
brew install gnucobol

cobc -v

  2. Python 3.10+

  3. Rust — needed for the transpiler and the eval_agent orchestrator.

    curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Installation

git clone https://github.com/nitrotap/COBOLEval.git
cd COBOLEval

python -m venv .venv
source .venv/bin/activate  # or `.venv\Scripts\activate` on Windows

pip install -e ".[dev]"

# Optional extras
pip install ollama                 # required for `scripts.generate ollama`
pip install -e ".[huggingface]"    # torch/transformers for local HF models

cp .env.example .env
# Edit .env with your API keys

ollama is not a declared dependency — install it only if you use the Python Ollama generator. The Rust orchestrator's Ollama support talks to the daemon over HTTP and does not need it.

Usage

Security warning. This program executes untrusted model-generated code. Run it in a sandbox.

This fork differs from upstream here. Upstream ships with the execution call in scripts/evaluation.py commented out, so that you must read the warning before running anything. In this fork that call is uncommented and execution is enabled by default in the Python harness. The Rust orchestrator keeps execution behind an explicit --execute flag.

Generate Completions

OpenAI Models

python -m scripts.generate openai --model-name gpt-4 --samples-per-task 1

python -m scripts.generate openai --model-name gpt-4 --temperature 0.7 --samples-per-task 10

Ollama (local models)

# Defaults to granite3.1-dense:8b against a local Ollama daemon
python -m scripts.generate ollama --model-name qwen2.5-coder --samples-per-task 1

# Point at a remote daemon
python -m scripts.generate ollama --model-name llama3.2 --host http://192.168.1.10:11434

Huggingface Models

# Completion mode
python -m scripts.generate huggingface --model-name codellama/CodeLlama-7b-hf

# Infill mode (for models with fill-in-the-middle capability)
python -m scripts.generate huggingface-infill \
    --model-name bigcode/starcoder \
    --prefix-token "<fim_prefix>" \
    --suffix-token "<fim_suffix>" \
    --middle-token "<fim_middle>"

Huggingface generation loads the model with device_map="cuda" and requires a CUDA GPU.

From Pre-generated JSONL

python -m scripts.generate json --jsonl-path ./completions.jsonl

Each generator writes solutions to preds/<model-name>/solutions/ plus a samples.jsonl.

Evaluate Results

# Evaluate a single model
python -m scripts.evaluate_functional_correctness evaluate \
    --pred-path preds/gpt-4 \
    --k "1,10,100"

# Batch evaluate multiple models
python -m scripts.evaluate_functional_correctness batch \
    --pred-paths '["preds/gpt-4", "preds/gpt-3.5-turbo"]'

Run the Transpiler

To regenerate the COBOLEval dataset from HumanEval:

cargo build --release

./target/release/cobol-eval-transpiler

./target/release/cobol-eval-transpiler \
    --input ./data/HumanEval.jsonl \
    --output ./data/CobolEval.jsonl \
    --verbose

146 of HumanEval's 164 tasks transpile. The current run reports the other 18 as 11 parse errors, 4 with no valid tests, 2 conversion errors, and 1 skipped for defining multiple functions.

GnuCOBOL compatibility caveat

The "GnuCOBOL 3.1.x compatibility" change applies to the Python path only. scripts/evaluation.py and scripts/generate.py now invoke cobc -w -x ... and cobc -m ... without -fformat=variable, so they work on 3.1.x.

src/eval/compiler.rs — used by the Rust orchestrator — still passes -fformat=variable and so still requires GnuCOBOL 3.2.0+. This is a known inconsistency, not a claim of repo-wide 3.1.x support.

Project Structure

COBOLEval/
├── src/                          # Rust
│   ├── main.rs                   # Transpiler CLI entry point
│   ├── types.rs / parser.rs      # Python AST parsing and type mapping
│   ├── transpiler.rs             # COBOL code generation
│   ├── constants.rs              # Configuration constants
│   ├── bin/eval_agent.rs         # Agentic orchestrator CLI
│   └── eval/                     # Agent internals
│       ├── tools.rs              # Rig tool definitions
│       ├── compiler.rs           # cobc invocation
│       ├── runner.rs             # Test harness execution
│       ├── checker.rs            # Output parsing and comparison
│       ├── metrics.rs            # pass@k estimator
│       └── types.rs              # Task/agent state, log entries
├── scripts/                      # Python
│   ├── generate.py               # OpenAI / Ollama / HF / JSONL generators
│   ├── evaluation.py             # Correctness checking
│   ├── evaluate_functional_correctness.py  # pass@k CLI
│   ├── data.py / utils.py        # Data loading and shared helpers
│   └── jcl/                      # JCL parser, validator, CLI
├── data/
│   ├── HumanEval.jsonl           # Original Python benchmark (164 tasks)
│   ├── CobolEval.jsonl           # Transpiled COBOL benchmark (146 tasks)
│   └── jcl/                      # Sample JCL files
├── tests/                        # Python test suite
├── .github/workflows/ci.yml
├── pyproject.toml
├── Cargo.toml
└── Dockerfile

Development

Running Tests

pytest tests/ -v          # 45 tests
cargo test                # 20 tests
pytest tests/ --cov=scripts --cov-report=html

Code Quality

pip install pre-commit
pre-commit install

ruff check scripts/
cargo clippy

ruff format scripts/
cargo fmt

CI status

The workflow at .github/workflows/ci.yml runs on this fork. As of the most recent run on main, the Python Tests, Rust Tests, Rust Build, and Integration Test jobs pass, and the two lint jobs fail: ruff check scripts/ reports roughly 100 style findings (mostly UP007 Optional[X] → X | None and import ordering), and cargo fmt --check / cargo clippy -- -D warnings are not clean. The badge above therefore shows red. These are formatting and lint-rule failures, not test failures.

Configuration

Environment variables (set in .env or export):

Variable Description Default
OPENAI_API_KEY OpenAI API key Required for OpenAI models
ANTHROPIC_API_KEY Anthropic API key Required for eval_agent --provider anthropic
COBOLEVAL_TIMEOUT Python harness command timeout (seconds) 30
HF_TOKEN Huggingface token (for gated models) Optional
RUST_LOG Rust logging level info

Benchmark Results

Model Pass@1 Source
GPT-4 10.27% bloop.ai, Evaluating LLMs on COBOL

That figure is bloop's, reproduced from the original repository. No benchmark numbers from this fork are published here — neither the Ollama generator nor the agentic orchestrator has a recorded full-benchmark run in this repository. Any results you produce are your own.

Citation

If you use COBOLEval in your research, please cite the original work:

@software{coboleval2024,
  author = {bloop.ai},
  title = {COBOLEval: LLM Evaluation for COBOL},
  year = {2024},
  url = {https://github.com/bloopai/COBOLEval}
}

License

MIT License, copyright bloop — see LICENSE. The license is unchanged from upstream and covers this fork's additions.

Contributing

  1. Fork the repository
  2. Create a feature branch (git checkout -b feature/amazing-feature)
  3. Run tests (pytest tests/ && cargo test)
  4. Commit changes (git commit -m 'Add amazing feature')
  5. Push to branch (git push origin feature/amazing-feature)
  6. Open a Pull Request

Acknowledgments

  • COBOLEval — the benchmark, dataset, and transpiler — by bloop.ai (bloopai/COBOLEval), initial implementation by Gabriel Gordon-Hall, with Huggingface model support contributed upstream by R Müller
  • Based on HumanEval by OpenAI
  • Uses GnuCOBOL for COBOL compilation
  • Agentic orchestrator built on rig
  • Fork additions (JCL tooling, agentic orchestrator, Ollama support, packaging and CI) by nitrotap; much of that work was AI-assisted, as the commit history shows

About

Evaluate LLM-generated COBOL

Resources

Readme
License
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages