Make a COTTAS query killable, and make a timeout report as one

[ecrum19]

What was wrong

06_equivalence hung twice on the same query — 41 hours, then 10 hours with --validation-query-timeout 1800 and --validation-time-budget 14400 on the command line. Two independent causes, either of which alone would have been enough:

1. The query was not interruptible. CottasEngine queried in-process through rdflib.Graph(store=pycottas.COTTASStore(...)). The call sits inside pycottas, so no timeout can reach it, and the time budget is checked between queries so it never gets a turn either.

2. Nothing ever produced exit 124. That is the code the query loop keys on (executions[query_id].get("exitCode") == 124). Both HTTP engines caught every exception and returned 1, making a timeout indistinguishable from a crash — so --stop-after-query-timeout was unreachable in any configuration, and timeouts were reported as EXECUTION_FAILED. Meanwhile run_query_process, which kills the whole process group and returns 124, was defined in the module with no callers at all.

The measurement that chose the design

I was about to build a long-lived worker subprocess with a query protocol, to preserve the load-once property the suite deliberately has. Measuring first, on a 39 MB artifact:

open store      0.015 s        LIMIT 1     12.719 s
re-open store   0.003 s        COUNT(*)    did not finish in 120 s

Opening is free. For COTTAS the artifact is the index — there is no load to amortise, so load-once does not apply to this engine and a process per query costs ~3 ms. The worker design would have been complexity bought with an assumption the measurement contradicts.

The same numbers say the hang is not a q05 bug: rdflib evaluates SPARQL over a custom Store by pulling triples through the Store API in Python. This fix bounds COTTAS; it does not make it fast.

Changes

• CottasEngine.execute runs each query via run_query_process — its own process group, killed on timeout, exit 124. start() keeps the dependency check; stop() has nothing to close.
• is_timeout_error distinguishes a genuine timeout (direct or wrapped in URLError) from a fault, and both HTTP engines now return 124 for one.

Verification

382 unit tests pass. The new ones cover the mechanism, not the report: that run_query_process returns 124 and that a forked grandchild does not outlive the group kill (the orphan case its docstring exists for), that CottasEngine no longer queries in-process, that timeouts classify correctly, and that something still produces the 124 the loop depends on.

End to end on the benchmark VM, against the exact artifact and query shape that hung:

elapsed   : 45.4 s   (bound was 45 s)
exitCode  : 124
error     : query exceeded 45s
VERDICT   : TIMEOUT ENFORCED

Note for the benchmark

With this, 06_equivalence completes and records q05 as a timeout for cottas while the other engines answer it. That is the honest cross-engine result rather than a hung run — but it means the COTTAS column will contain timeouts at this scale, which belongs in the limitations rather than being read as disagreement.

🤖 Generated with Claude Code

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 90.58824% with 8 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/validation/validation_runner.py	74.19%	8 Missing ⚠️

📢 Thoughts on this report? Let us know!