v2.0.0 - Compression and Optimization Refactoring

[ecrum19]

VCF-RDFizer v2.0.0

Highlights

VCF-RDFizer 2.0.0 introduces a redesigned RDF compression workflow focused on scalable HDT and COTTAS generation, lower temporary disk usage, and stronger output validation.

• Added partitioned HDT generation with HDTCat merging for large RDF graphs.
• Added COTTAS as a queryable RDF representation alongside HDT.
• Added record-safe, configurable RDF chunking shared by HDT and COTTAS compression.
• Added plain and space-optimized RDF storage modes, including streamed .nt.gz aggregation for lower disk usage.
• Added validation after HDT/COTTAS compression to verify artifact validity and confirm decoded triple counts match the RDF source.
• Added automatic HDT index initialization using HDT Java’s versioned index sidecar format.
• Improved output organization, basename handling, and preflight collision checks to prevent accidental overwrites.
• Added explicit control of raw RDF retention with --keep-rmlstreamer-rdf-output and --remove-rdf-storage-output.
• Reworked compression options into separate RDF staging, representation, and artifact-packaging decisions.
• Corrected compression timing metrics so aggregate and partitioned wall times reflect the full operation.
• Improved Docker support with multi-architecture amd64 and arm64 images, versioned image tags, and latest.
• Hardened PyPI and conda-forge release automation, package metadata, packaged rule files, and release validation.

Breaking Changes

• Full pipeline runs now require an explicit --rdf-storage-mode selection.
• Compression configuration should use:
  • --rdf-compression
  • --representations
  • --artifact-compression
• The former --keep-rdf option has been replaced by --keep-rmlstreamer-rdf-output.
• Output files are now grouped under a basename-derived directory, such as sample/sample.hdt, rather than directories retaining .nt suffixes.

Upgrade Notes

Review existing pipeline commands before upgrading, particularly compression-related options and expected output paths. For large inputs, use --rdf-storage-mode space-optimized with --representations hdt,cottas and adjust --chunk-target-bytes only when the default chunk size is unsuitable for available memory or storage.

[codecov-commenter]

Welcome to Codecov 🎉

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

ℹ️ You can also turn on project coverage checks and project coverage reporting on Pull Request comment

Thanks for integrating Codecov - We've got you covered ☂️