Add PyLucene integration and CPU/GPU end-to-end tests

[nvzm123]

Summary

This ports the cuVS-Lucene production changes from NVIDIA/cuvs-lucene#174 into java/cuvs-lucene after the project moved into the cuVS monorepo.

The port:

• targets the Lucene 10.2 runtime ABI while retaining the established Lucene101AcceleratedHNSWCodec SPI name;
• uses Lucene 10.2's Lucene101Codec as the accelerated codecs' explicit delegate;
• exposes the correctly typed scalar and binary provider APIs while retaining legacy JVM descriptors with actionable failures;
• initializes scalar- and binary-quantized vector formats lazily so SPI discovery cannot be poisoned by an unavailable optional Lucene format;
• preserves the concrete throwable type when reflective format construction fails;
• keeps PR Fix silent brute force fallback at aligned vector dimensions #2483's padded-dataset behavior for aligned and unaligned CAGRA serialization;
• corrects GPU query-path documentation for multi-partition algorithm selection and global-document-ID result partitioning; and
• verifies the exact contents of the published thin JAR, including SPI providers and exclusion of Lucene, base cuvs-java, and test-only payloads.

The PyLucene pytest suite and its test-only Java adapters are owned by the cuVS Bench backend in NVIDIA/cuvs#2385, not by the production cuvs-lucene artifact. The standard thin cuvs-lucene JAR remains the artifact under test; Lucene and the base cuvs-java JAR remain external classpath dependencies.

Apache does not publish PyLucene 10.2.0, so end-to-end validation uses a custom PyLucene wrapper generated against the same Lucene 10.2.0 sources as this project.

Test coverage

The in-tree Java coverage adds or strengthens:

• fresh-JVM codec and vector-format SPI discovery;
• lazy scalar and binary format construction with an empty provider cache;
• Lucene 10.2 binary-format capability and constructor-failure classification;
• legacy provider-descriptor compatibility; and
• exact thin-JAR SPI and ownership checks during Maven verify.

The dependent pytest matrix in #2385 proves these execution paths:

• CPU HNSW build and HNSW search;
• GPU CAGRA build followed by one-layer or three-layer HNSW search; and
• GPU CAGRA build and CAGRA search.

GPU-required cases assert the concrete accelerated writer, reader, and query implementations and fail on unavailable cuVS or CPU fallback. CPU cases explicitly verify a stock Lucene HNSW path.

The matrix covers a single live document, one and ten segments, 10-to-1 and 100-to-10 force merges, CAGRA searchWidth values 1, 16, and 32, deletions, selective filters, persisted HNSW graph degree/layers, exact filtered search, and deterministic brute-force recall. Assertions check rank-one self matches where applicable, exact hit counts, duplicate exclusion, inactive/filter-rejected document exclusion, and configurable recall floors.

Warning-free CAGRA cases use graphDegree=32, intermediateGraphDegree=64, and enough vectors to avoid graph-parameter clamping, including 24,832 vectors for the three-layer case.

Validation

Final producer revision: d6fcab0946837d7d3997cec4ed18189d3faa12e6.

Validated on an NVIDIA A10G with JDK 22, matching cuVS Java/native 26.10 artifacts, and Lucene/PyLucene 10.2:

• fetched pull/2475/head from GitHub and detached at the exact revision above: passed;
• ./build.sh lucene from that detached checkout: passed;
• mvn clean verify from that detached checkout: 332 tests, 0 failures, 0 errors, 30 skipped;
• ThinJarContentsIT: 1 test, 0 failures, 0 errors, 0 skipped;
• focused TestBackCompat,TestLucene102FormatConstruction: 20 passed;
• Add PyLucene backend for cuVS-Lucene vector search in cuvs-bench #2385 non-live PyLucene tests: 293 passed, 43 deselected;
• Add PyLucene backend for cuVS-Lucene vector search in cuvs-bench #2385 focused CAGRA searchWidth 16/32 cases: 2 passed, 18 deselected, recall 1.0;
• Add PyLucene backend for cuVS-Lucene vector search in cuvs-bench #2385 full GPU PyLucene suite against the exact detached JAR: 43 passed, 293 deselected in 47.26 seconds; and
• Add PyLucene backend for cuVS-Lucene vector search in cuvs-bench #2385 full cuVS Bench suite: 487 passed, 71 skipped in 72.68 seconds.

The Maven suite retains existing warnings from randomized small-dataset graph clamping, JVM vector/native access, intentional fallback or invalid-configuration cases, and the Javadoc plugin's existing unknown includePackageNames parameter. The PyLucene suite emitted no cuVS graph-clamping or CPU-fallback warnings; it emitted the expected JVM notice for the incubating vector module.

A clean rebuild of all current-main native dependencies was not used for the final GPU rerun: after adding the container's missing matching libcufile-dev headers, the build exposed 5,420 native targets and was stopped as impractical for this Java/Python-only change. The final producer JAR itself was built from the exact revision above and tested against a compatible native runtime reporting cuVS 26.10.

Known randomized-test failure

Seed CC0EA94328BAB3E5 exposes an existing failure in TestCuVSVectorsFormat.testRandomWithUpdatesAndGraph: IllegalStateException: Index not found for field:field at CuVS2510GPUVectorsReader.java:425.

The same seed reproduces identically on this branch and on the untouched base commit 0fa5ebe5, including the secondary seed and stack trace. It is therefore not introduced by this port. Reproduction command from java/cuvs-lucene:

mvn -q -Dtest=TestCuVSVectorsFormat#testRandomWithUpdatesAndGraph \
  -Dtests.seed=CC0EA94328BAB3E5 \
  -Dtests.locale=ti-Ethi-ET \
  -Dtests.timezone=Greenwich test

The full validation listed above passes.

Follow-up multithreaded concurrency coverage is tracked in NVIDIA/cuvs#2407.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[cjnolet]

We shouldn't be hosting python files in java directories. Please move to cuvs-bench.

[cjnolet]

Again, this should be hosted in cuvs-bench, not in cuvs-lucene.

[cjnolet]

I mentioned this in the prior PR in cuvs-lucene- you should not need to list special instructions in the docs for running the pylucene tests. They should be part of the cuvs-bench python test suite. Please remove all this stuff from the docs. You also should not need to explicitly state how to run the java tests for cuvs-lucene- running cuvs-lucene tests should be enough to run all the java tests in cuvs-lucene.

[cjnolet]

This still needs some work.