extensive improvement of validation suite and general cleaning

Author: ecrum19

.github/workflows/validation-mutation.yml

@@ -0,0 +1,70 @@
+name: validation-mutation
+
+# The mutation score is the measured coverage of the semantic validation suite.
+# The host job runs on every push because it needs no Docker; the container job
+# is the authority, replaying the same catalogue under the engines that ship in
+# the image.
+
+on:
+  push:
+    branches:
+      - "**"
+  pull_request:
+  workflow_dispatch:
+
+jobs:
+  mutation-score:
+    name: mutation score (rdflib)
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Setup Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      - name: Install test dependencies
+        run: python -m pip install --upgrade pip rdflib
+
+      - name: Run the mutation harness
+        env:
+          VCF_RDFIZER_MUTATION_REPORT: mutation-score.json
+        run: python -m unittest test.test_validation_mutation_unit -v
+
+      - name: Report the score
+        run: |
+          python - <<'PY'
+          import json
+          report = json.load(open("mutation-score.json"))
+          print(f"mutation score: {report['detected']}/{report['total']} "
+                f"({report['score']:.0%})")
+          for item in report["mutations"]:
+              if not item["detected"]:
+                  print(f"  undetected: {item['id']} ({item['representation']}) "
+                        f"- {item['knownUndetected']}")
+          PY
+
+      - name: Upload the score
+        uses: actions/upload-artifact@v4
+        with:
+          name: mutation-score
+          path: mutation-score.json
+
+  engine-agreement:
+    name: cross-engine query agreement
+    runs-on: ubuntu-latest
+    # The image build is slow, so this gates merges rather than every push.
+    if: github.event_name == 'pull_request' || github.event_name == 'workflow_dispatch'
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Build the image
+        run: docker build -t vcf-rdfizer:mutation-ci .
+
+      - name: Every validation query must agree across Comunica and QLever
+        run: |
+          docker run --rm -v "$PWD:/repo:ro" vcf-rdfizer:mutation-ci \
+            /opt/pycottas-venv/bin/python /repo/test/cross_engine_agreement.py

ACKNOWLEDGEMENTS.md

@@ -15,7 +15,7 @@ adapted for the Acknowledgements section:
 > This work was carried out at KNoWS, IDLab, Ghent University - imec, as part
 > of the doctoral research of Elias Crum. The authors thank the maintainers of
 > the open-source components on which VCF-RDFizer builds - RMLStreamer,
-> hdt-cpp, hdtc, pycottas, Comunica, bcftools, and cyvcf2.
+> hdt-cpp, hdtc, pycottas, Comunica, QLever, pySHACL, bcftools, and cyvcf2.
 
 If the work is co-funded by a specific project or grant, add that project name
 and grant number to the sentence above and record it in the table at the top of

Dockerfile

@@ -1,6 +1,11 @@
 ARG RMLSTREAMER_VERSION=2.5.0
 ARG HDTC_VERSION=1.1.0
 ARG COMUNICA_VERSION=5.3.0
+# QLever is an optional second SPARQL engine for validation. Its binaries are
+# copied from the upstream published image rather than built here: compiling
+# QLever needs a large C++ toolchain and would dominate this image's build.
+# Pin a digest or release tag here to make validation runs reproducible.
+ARG QLEVER_IMAGE=adfreiburg/qlever:latest
 
 FROM eclipse-temurin:11-jre AS build-hdt-cpp
 
@@ -59,6 +64,10 @@ RUN cargo build --locked --release \
   && cp LICENSE /opt/third_party_licenses/HDTC.LICENSE
 
 
+# Named stage so the runtime image can COPY QLever's binaries out of it.
+FROM ${QLEVER_IMAGE} AS qlever
+
+
 FROM eclipse-temurin:11-jre
 
 ARG RMLSTREAMER_VERSION
@@ -94,7 +103,8 @@ RUN python3 -m venv /opt/pycottas-venv \
     duckdb==1.5.5 \
     pyarrow==22.0.0 \
     numpy==2.4.6 \
-    cyvcf2==0.34.0
+    cyvcf2==0.34.0 \
+    pyshacl==0.30.1
 
 RUN npm install --global "@comunica/query-sparql-file@${COMUNICA_VERSION}"
 
@@ -110,6 +120,33 @@ COPY --from=build-hdt-cpp /usr/local/lib/libhdt* /usr/local/lib/
 COPY --from=build-hdt-cpp /opt/third_party_licenses/ /usr/share/licenses/vcf-rdfizer/
 COPY --from=build-hdtc /opt/hdtc/target/release/hdtc /usr/local/bin/hdtc
 COPY --from=build-hdtc /opt/third_party_licenses/ /usr/share/licenses/vcf-rdfizer/
+
+# Optional QLever SPARQL engine (--validation-engine qlever). Comunica remains
+# the default, so an image whose QLever binaries turn out to be unusable is
+# still fully functional; the validator reports the reason instead of failing
+# obscurely. Pin a different tag or digest with
+# --build-arg QLEVER_IMAGE=adfreiburg/qlever:<tag>.
+#
+# QLever's image is built on a different Ubuntu release than this one, so the
+# binaries alone are not enough: their Boost, ICU, jemalloc and io_uring
+# sonames are release-specific and absent here. Those libraries travel with the
+# binaries into a private directory that only QLever's own processes are
+# pointed at, so they cannot shadow anything the rest of the image links
+# against. (glibc itself is not copied - it is backward compatible, and this
+# base is newer than QLever's.)
+COPY --from=qlever /qlever/qlever-index /opt/qlever/bin/qlever-index
+COPY --from=qlever /qlever/qlever-server /opt/qlever/bin/qlever-server
+COPY --from=qlever \
+  /lib/x86_64-linux-gnu/libboost_iostreams.so.1.83.0 \
+  /lib/x86_64-linux-gnu/libboost_program_options.so.1.83.0 \
+  /lib/x86_64-linux-gnu/libboost_url.so.1.83.0 \
+  /lib/x86_64-linux-gnu/libgomp.so.1 \
+  /lib/x86_64-linux-gnu/libicudata.so.74 \
+  /lib/x86_64-linux-gnu/libicui18n.so.74 \
+  /lib/x86_64-linux-gnu/libicuuc.so.74 \
+  /lib/x86_64-linux-gnu/libjemalloc.so.2 \
+  /lib/x86_64-linux-gnu/liburing.so.2 \
+  /opt/qlever/lib/
 COPY THIRD_PARTY_NOTICES.md /usr/share/licenses/vcf-rdfizer/THIRD_PARTY_NOTICES.md
 COPY src/*.sh /opt/vcf-rdfizer/
 COPY src/*.py /opt/vcf-rdfizer/
@@ -121,7 +158,22 @@ COPY vcf_rdfizer_gzip.py /opt/vcf-rdfizer/
 RUN chmod +x /opt/vcf-rdfizer/*.sh \
   && chmod +x /usr/local/bin/rdf2hdt \
   && chmod +x /usr/local/bin/hdt2rdf \
-  && chmod +x /usr/local/bin/hdtc
+  && chmod +x /usr/local/bin/hdtc \
+  && chmod +x /opt/qlever/bin/qlever-index /opt/qlever/bin/qlever-server
+
+# QLever's binaries come from a different base image, so record at build time
+# whether they actually link here. The validator reads this marker to explain
+# an unavailable engine instead of surfacing a bare "not found".
+RUN set -eu; \
+  status="ok"; \
+  for binary in qlever-index qlever-server; do \
+    missing="$(LD_LIBRARY_PATH=/opt/qlever/lib ldd "/opt/qlever/bin/$binary" 2>&1 | grep 'not found' || true)"; \
+    if [ -n "$missing" ]; then \
+      status="QLever binary $binary has unresolved shared libraries in this base image: $missing"; \
+      echo "WARNING: $status" >&2; \
+    fi; \
+  done; \
+  printf '%s\n' "$status" > /opt/vcf-rdfizer/qlever-status.txt
 
 ENV RMLSTREAMER_JAR=/opt/rmlstreamer/RMLStreamer-v${RMLSTREAMER_VERSION}-standalone.jar
 ENV JAR=/opt/rmlstreamer/RMLStreamer-v${RMLSTREAMER_VERSION}-standalone.jar
@@ -132,6 +184,8 @@ ENV COTTAS_MERGE_BATCH_ROWS=2048
 ENV RDF2HDT_BIN=/usr/local/bin/rdf2hdt
 ENV HDT2RDF_BIN=/usr/local/bin/hdt2rdf
 ENV COTTAS_PYTHON_BIN=/opt/pycottas-venv/bin/python
+ENV QLEVER_INDEX_BUILDER_BIN=/opt/qlever/bin/qlever-index
+ENV QLEVER_SERVER_BIN=/opt/qlever/bin/qlever-server
 ENV LD_LIBRARY_PATH=/usr/local/lib
 
 # COTTAS creates a temporary DuckDB database in the container working

README.md

@@ -19,6 +19,12 @@ VCF-RDFizer is a Docker-first CLI wrapper for:
 
 The VCF-RDFizer vocabulary is available at [https://w3id.org/vcf-rdfizer/vocab#](https://w3id.org/vcf-rdfizer/vocab#).
 
+This README is the task-oriented reference. For how the tool works, why it is
+built that way, and where it stops working, see the documentation set in
+**[`docs/`](docs/README.md)** - starting with
+[Architecture](docs/architecture.md) and, before you rely on the output,
+[Limitations](docs/limitations.md).
+
 ## Requirements
 
 - Python 3.10+
@@ -134,6 +140,59 @@ stage summary at `stages/validation/<dataset-id>.json`. See [Semantic VCF/RDF
 validation](docs/validation.md) for query definitions, preflight checks, result
 statuses, and cleanup evidence.
 
+
+### Artifacts and engines
+
+`--rdf` accepts any artifact the pipeline produces - `.nt`, `.nt.gz`, `.nt.br`,
+`.hdt`, `.cottas`, `.cottas.gz`, `.cottas.br`. A compressed or indexed artifact
+is decoded back to N-Triples **inside the container** and then put through the
+full semantic suite, which proves it decodes to a graph that still reproduces
+every VCF summary - stronger than the triple-count round-trip that runs during
+compression.
+
+In full mode, `--validate-artifacts {aggregate,hdt,cottas,all}` chooses which
+produced artifacts to check; each is validated independently with its own
+report:
+
+```bash
+vcf-rdfizer --mode full -i ./cohort.vcf.gz \
+  --rdf-storage-mode space-optimized --representations hdt,cottas \
+  --validate --validate-artifacts all -o ./results
+```
+
+`--validation-engine {comunica,qlever}` selects the SPARQL backend. Comunica
+(default) queries the file in memory; [QLever](https://github.com/ad-freiburg/qlever)
+builds an on-disk index inside the container and serves it, which is what makes
+cohort-scale graphs queryable. Both answer identical queries, so the choice is
+never semantic, and every report records which engine ran.
+
+```bash
+vcf-rdfizer --mode validation -i ./cohort.vcf.gz --rdf ./results/cohort/cohort.hdt \
+  --validation-engine qlever --qlever-memory-gb 32 -o ./validation-results
+```
+
+Tuning: `--qlever-memory-gb`, `--qlever-port`, `--qlever-startup-timeout`,
+`--validation-query-timeout`, and repeatable `--qlever-index-arg` /
+`--qlever-server-arg` escape hatches.
+
+Validation runs three independent layers: exact aggregate comparison against
+the VCF, a predicate/class census plus per-record and per-value identity
+digests, and — with `--shacl-shapes` — SHACL conformance against the
+vocabulary's published shapes.
+
+```bash
+vcf-rdfizer --mode validation -i ./cohort.vcf.gz --rdf ./results/cohort/cohort.nt.gz \
+  --shacl-shapes ./vocabulary/shacl/vcf-rdfizer-vocabulary.shacl.ttl \
+  --strict-conformance -o ./validation-results
+```
+
+> **What a PASS means.** Coverage is measured, not asserted: a mutation harness
+> corrupts a correct graph in 36 named ways and records which are detected
+> (currently **64/66**). See [`docs/vcf-coverage.md`](docs/vcf-coverage.md) for
+> the element-by-element matrix and the remaining gaps, and
+> [`docs/validation-methodology.md`](docs/validation-methodology.md) for how the
+> number is produced.
+
 ## Compression Plan
 
 Compression is configured as three independent decisions:
@@ -204,6 +263,23 @@ host filesystem.
 - `--remove-rdf-storage-output` explicitly remove the aggregate `.nt`/`.nt.gz` after successful compression
 - `-e, --estimate-size` preflight size estimate
 
+## VCF Coverage
+
+Full-mode conversion covers every VCF column. Three options control how much
+structure is emitted; all default to the richer form.
+
+| Option | Effect |
+| --- | --- |
+| `--sample-representation {expanded,condensed}` | Genotype shape (see below) |
+| `--info-representation {structured,raw}` | `structured` adds one `vcfr:InfoFieldValue` per record and key, with typed values, alongside `vcfr:infoRaw` |
+| `--header-representation {structured,basic}` | `structured` types each `##` line with its vocabulary subclass and lifts FILTER/ALT/contig attributes into their own properties |
+
+QUAL is always emitted, typed `xsd:decimal` or `vcfr:Null`, because the
+published SHACL shape requires a datatype that depends on the value.
+
+[`docs/vcf-coverage.md`](docs/vcf-coverage.md) maps every VCF element to its RDF
+terms and to the validation check that covers it.
+
 ## Sample Representation Modes
 
 Full mode has exactly two explicit sample workflows. There is no automatic
@@ -853,10 +929,25 @@ in `vcf_rdfizer.py` append it directly to the aggregate, because the equivalent
 RML maps would first have to materialize variants x samples (x FORMAT keys)
 helper TSV rows. See [`docs/sample-representation-guide.md`](docs/sample-representation-guide.md).
 
-Further reading:
+Further reading: **[`docs/`](docs/README.md) is the in-depth documentation set** -
+how each part of the tool works, why, and where it stops working.
+
+| Document | Covers |
+| --- | --- |
+| [Architecture](docs/architecture.md) | Host/container split, failure policy, pinned toolchain |
+| [Conversion](docs/conversion.md) | VCF -> TSV -> RDF, stage by stage |
+| [Representations](docs/representations.md) | Compression, HDT/COTTAS, chunking, round-trip checks |
+| [Output and metrics](docs/output-and-metrics.md) | Run layout, reports, progress, exit codes |
+| [Custom RML mappings](docs/rml-mappings.md) | The `--rules` contract in full |
+| [Sample representations](docs/sample-representation-guide.md) | Expanded vs condensed genotype shapes |
+| [Validation](docs/validation.md) | The semantic suite, and what it does not test |
+| [Validation methodology](docs/validation-methodology.md) | How coverage is measured, not asserted |
+| [VCF coverage matrix](docs/vcf-coverage.md) | Element by element, with the mutation that proves each row |
+| [CLI reference](docs/cli-reference.md) | Every flag, with constraints and interactions |
+| [Limitations](docs/limitations.md) | Everything the tool cannot do, in one place |
+| [Roadmap](docs/roadmap.md) | Planned work, known defects, and rejected options |
+| [Data linking design](docs/datalinking-design.md) | Proposal: a plug-in system for external links |
 
-- [`docs/validation.md`](docs/validation.md) - semantic validation design and query set
-- [`docs/sample-representation-guide.md`](docs/sample-representation-guide.md) - emitted genotype shapes
 - [`changelog.md`](changelog.md) - dated change history
 - [`ACKNOWLEDGEMENTS.md`](ACKNOWLEDGEMENTS.md) - funding and attribution