Add PyLucene integration and CPU/GPU end-to-end tests

.gitignore

@@ -12,3 +12,5 @@ target/
 # IDE - Eclipse
 .project
 cuvs-workdir
+__pycache__/
+.pytest_cache/

README.md

@@ -12,23 +12,23 @@ This is a project for using [cuVS](https://github.com/rapidsai/cuvs), NVIDIA's G
 
 ## What is cuvs-lucene?
 
-`cuvs-lucene` provides a pluggable [KnnVectorsFormat](https://lucene.apache.org/core/10_2_0/core/org/apache/lucene/codecs/KnnVectorsFormat.html) that uses cuVS to offload vector index build — and optionally search — to NVIDIA GPUs. Because it plugs in through a standard Lucene codec, existing Lucene applications can take advantage of GPU acceleration with minimal code changes and gracefully fall back to the default CPU codec when no GPU is present.
+`cuvs-lucene` provides a pluggable [KnnVectorsFormat](https://lucene.apache.org/core/10_2_0/core/org/apache/lucene/codecs/KnnVectorsFormat.html) that uses cuVS to offload vector index build — and optionally search — to NVIDIA GPUs. The accelerated-HNSW codecs can fall back to Lucene's CPU HNSW writer when cuVS is unavailable; the GPU-search codec requires cuVS. This development line is compiled and tested against the Lucene 10.2.0 runtime ABI.
 
 Four codecs are currently provided:
 
-- `Lucene101AcceleratedHNSWCodec` — GPU-accelerated HNSW build with CPU HNSW search. The on-disk format is standard Lucene HNSW, so indexes built on the GPU can be read by any stock Lucene 10.x reader.
+- `Lucene101AcceleratedHNSWCodec` — GPU-accelerated HNSW build with CPU HNSW search. Its vector data uses Lucene's standard HNSW format and stock HNSW reader; applications still need a compatible `cuvs-lucene` codec provider to resolve the segment codec.
   - `LuceneAcceleratedHNSWScalarQuantizedCodec` — scalar-quantized vectors for a smaller index footprint.
   - `LuceneAcceleratedHNSWBinaryQuantizedCodec` — binary-quantized vectors for an even smaller index footprint.
-- `CuVS2510GPUSearchCodec` — GPU-accelerated HNSW build and GPU search
+- `CuVS2510GPUSearchCodec` — GPU CAGRA build and GPU CAGRA search
 
 ## Installing cuvs-lucene
 
 ### Prerequisites
 
-- [CUDA 12.0+](https://developer.nvidia.com/cuda-toolkit-archive)
+- [CUDA 12.2+](https://developer.nvidia.com/cuda-toolkit-archive)
 - [JDK 22](https://jdk.java.net/archive/)
 - [Maven 3.9.6+](https://maven.apache.org/download.cgi)
-- A compatible cuVS installation (26.04 - 26.06). For Maven usage, install the cuVS tarball and add it to your system library load path. See the cuVS [tarball install instructions](https://docs.rapids.ai/api/cuvs/stable/build/#download-extract).
+- For the published `cuvs-lucene` 26.08.0 release, a matching cuVS 26.08 installation. Install the cuVS tarball and add it to your system library load path; see the cuVS [tarball install instructions](https://docs.nvidia.com/cuvs/installation/c#tarball).
 
 ### Maven
 
@@ -38,19 +38,24 @@ To pull `cuvs-lucene` into a Maven project, add the following dependency to your
 <dependency>
   <groupId>com.nvidia.cuvs.lucene</groupId>
   <artifactId>cuvs-lucene</artifactId>
-  <version>26.06.0</version>
+  <version>26.08.0</version>
 </dependency>
 ```
 
 ### Building from source
 
+This development checkout currently targets `cuvs-lucene` and `cuvs-java`
+26.10.0. Build it against matching cuVS 26.10 Java and native artifacts; the
+published Maven release above remains 26.08.0 until the 26.10 release is
+available.
+
 ```sh
 git clone https://github.com/rapidsai/cuvs-lucene.git
 cd cuvs-lucene
 mvn clean compile package
 ```
 
-The resulting artifacts are written to `target/`. To run the tests, first install cuVS and add it to your system library load path, as described in the cuVS [tarball install instructions](https://docs.rapids.ai/api/cuvs/stable/build/#download-extract), then run:
+The resulting artifacts are written to `target/`. To run the tests, first install cuVS and add it to your system library load path, as described in the cuVS [tarball install instructions](https://docs.nvidia.com/cuvs/installation/c#tarball), then run:
 
 ```sh
 mvn clean test
@@ -60,7 +65,7 @@ mvn clean test
 
 The example below plugs the GPU-accelerated HNSW codec into a standard Lucene `IndexWriter`. Once the codec is set on the `IndexWriterConfig`, indexing proceeds exactly as it would with the default Lucene codec, and search uses the stock `KnnFloatVectorQuery`.
 
-Before running it, make sure cuVS is installed and available on your system library load path. The cuVS [tarball install instructions](https://docs.rapids.ai/api/cuvs/stable/build/#download-extract) show how to set this up.
+Before running it, make sure cuVS is installed and available on your system library load path. The cuVS [tarball install instructions](https://docs.nvidia.com/cuvs/installation/c#tarball) show how to set this up.
 
 ### RMM async allocation for GPU search
 
@@ -117,15 +122,71 @@ public class HelloCuvsLucene {
 }
 ```
 
-The artifacts would be built and available in the target / folder.
+The artifacts are built in the `target/` directory.
 
-### Running Tests
+Run the example with:
 
 ```sh
 mvn -q compile org.codehaus.mojo:exec-maven-plugin:3.5.1:java \
   -Dexec.mainClass=com.nvidia.cuvs.lucene.examples.HelloCuvsLucene
 ```
 
+### Using with PyLucene
+
+The complete codec set requires PyLucene generated against Lucene 10.2.0. The
+official PyLucene 10.0.0 distribution is not a compatible full-feature runtime:
+in particular, GPU search uses Lucene 10.2 APIs and binary quantization uses
+Lucene102 vector formats.
+
+Apache does not publish a PyLucene 10.2.0 release. Linux development and
+testing therefore require a custom PyLucene wrapper build generated against
+the Lucene 10.2.0 sources. Prepare and activate that matching external
+environment before using PyLucene or running pytest; Maven can build the jar
+independently and does not produce the PyLucene runtime.
+
+Build the standard thin `cuvs-lucene` jar:
+
+```sh
+mvn clean package -DskipTests
+```
+
+Add that jar and the matching base `cuvs-java` jar to the classpath passed to
+`lucene.initVM(...)`. PyLucene can then load the codec through Lucene's service
+provider lookup:
+
+```python
+from org.apache.lucene.codecs import Codec
+
+codec = Codec.forName("Lucene101AcceleratedHNSWCodec")
+```
+
+Use the returned codec with `IndexWriterConfig.setCodec(codec)`. Initialize the
+JVM only after the custom 10.2 environment and every application jar are on its
+classpath; PyLucene cannot replace that classpath after `lucene.initVM(...)`.
+
+### Running Tests
+
+Run the Java tests with `mvn clean test`. Once the custom PyLucene 10.2
+environment is activated and the cuVS classpath and native-library environment
+are available, run the full parametrized CPU/GPU end-to-end suite directly
+with pytest:
+
+```sh
+python3 -m pytest -q -s src/test/python/test_pylucene_end_to_end.py
+```
+
+The cases live in `src/test/python/test_pylucene_end_to_end.py`; reusable
+runtime helpers are in `pylucene_test_support.py`, and the Java test bridge is
+compiled to `target/test-classes`. Set `CUVS_LUCENE_JAR`,
+`CUVS_LUCENE_CUVS_JAVA_JAR`, or `CUVS_LUCENE_PYLUCENE_TEST_CLASSES` only when
+their standard Maven locations are not appropriate. The helper verifies that
+PyLucene's Lucene version exactly matches the `lucene-core` version in this
+checkout's POM before starting the JVM, so a mismatched wrapper fails with an
+actionable error instead of a later linkage failure.
+
+The pytest IDs identify CPU HNSW, CAGRA-built HNSW, and CAGRA-search cases;
+use pytest's `-k` option for a focused run.
+
 For more examples, including one that indexes and searches entirely on the GPU using `CuVS2510GPUSearchCodec`, please refer to the [`examples/`](examples) directory.
 
 ## Contributing

src/main/java/com/nvidia/cuvs/lucene/CuVS2510GPUSearchCodec.java

@@ -44,11 +44,7 @@ public CuVS2510GPUSearchCodec() throws Exception {
    * @param delegate the delegate codec
    */
   public CuVS2510GPUSearchCodec(String name, Codec delegate) {
-    this(
-        name,
-        delegate,
-        new GPUSearchParams.Builder().build(),
-        FilterBitsetCacheConfig.DEFAULT);
+    this(name, delegate, new GPUSearchParams.Builder().build(), FilterBitsetCacheConfig.DEFAULT);
   }
 
   /**
@@ -69,8 +65,8 @@ public CuVS2510GPUSearchCodec(GPUSearchParams params) throws Exception {
    * @param filterCacheConfig filter-bitset-cache configuration
    * @throws Exception Exception raised when initializing the codec
    */
-  public CuVS2510GPUSearchCodec(
-      GPUSearchParams params, FilterBitsetCacheConfig filterCacheConfig) throws Exception {
+  public CuVS2510GPUSearchCodec(GPUSearchParams params, FilterBitsetCacheConfig filterCacheConfig)
+      throws Exception {
     this(NAME, LuceneProvider.getCodec("101"), params, filterCacheConfig);
   }
 
@@ -97,8 +93,7 @@ public CuVS2510GPUSearchCodec(
    * @param params an instance of {@link GPUSearchParams}
    * @param filterCacheConfig filter-bitset-cache configuration
    */
-  private void initializeFormat(
-      GPUSearchParams params, FilterBitsetCacheConfig filterCacheConfig) {
+  private void initializeFormat(GPUSearchParams params, FilterBitsetCacheConfig filterCacheConfig) {
     try {
       format = new CuVS2510GPUVectorsFormat(params, filterCacheConfig);
       setKnnFormat(format);

src/main/java/com/nvidia/cuvs/lucene/CuVS2510GPUVectorsReader.java

@@ -447,8 +447,8 @@ public void search(String field, float[] target, KnnCollector knnCollector, Bits
       }
       topK = Math.min(knnCollector.k() + 10, mask[0].cardinality());
       // numDocs must be the total vector count so cuVS sizes the prefilter to cover every ordinal.
-      // BitSet.length() is (highest set bit + 1), which under a selective filter is smaller than the
-      // vector count, leaving the trailing ordinals outside the filter and thus default-accepted.
+      // A selective BitSet may end before the vector count, which would leave trailing ordinals
+      // outside the filter and therefore accepted by default.
       maskLength = acceptedOrds.length();
     }
 

src/main/java/com/nvidia/cuvs/lucene/CuVS2510GPUVectorsWriter.java

@@ -249,7 +249,7 @@ private void writeCagraIndex(OutputStream os, CuVSMatrix dataset) throws Throwab
                 .withIndexParams(params)
                 .build();
         var deviceVectors = dataset.toDevice(getCuVSResourcesInstance());
-        var indexDataset = index.makePaddedDataset(deviceVectors)) {
+        var indexDataset = index.makePaddedDatasetView(deviceVectors)) {
       index.updateDataset(indexDataset);
       index.serialize(os);
     }

src/main/java/com/nvidia/cuvs/lucene/GPUKnnFloatVectorQuery.java

@@ -52,9 +52,9 @@
  * single multi-partition search to cuVS, passing one Lucene segment per cuVS partition. cuVS
  * runs the per-partition CAGRA searches, applies distance post-processing, and performs the
  * cross-partition top-k merge internally; the returned arrays are mapped to Lucene doc IDs on
- * the host. The effective CAGRA algorithm (SINGLE_CTA or MULTI_KERNEL) is selected by cuVS
- * based on {@code searchAlgo} and {@code itopk_size}, with MULTI_KERNEL handling k beyond
- * SINGLE_CTA's per-partition cap.
+ * the host. For a multi-partition search, cuVS resolves {@code AUTO} to {@code SINGLE_CTA} or
+ * {@code MULTI_CTA} from the search parameters and query/partition topology; {@code MULTI_KERNEL}
+ * is not supported by the multi-partition API.
  *
  * <p>If the query has an explicit {@code filter}, or if any segment carries live-document deletes,
  * the acceptance mask (filter ∩ liveDocs) is packed into one {@link FilterBitsetHandle} per segment
@@ -161,10 +161,9 @@ public Query rewrite(IndexSearcher indexSearcher) throws IOException {
       gpuReaders.add(gpuReader);
     }
 
-    // Build one filter handle per segment encoding (filter ∩ that segment's liveDocs) whenever any
-    // filtering is required — either an explicit Lucene filter, or live-document deletes in at least
-    // one segment. Each segment's handle becomes that partition's filter; a segment with neither an
-    // explicit filter nor deletes gets a null entry (unfiltered for that partition).
+    // Build one filter handle per segment whenever an explicit filter or any segment deletion
+    // requires filtering. Each handle encodes (filter ∩ segment liveDocs); an unaffected segment
+    // gets a null entry and remains unfiltered for that partition.
     boolean hasExplicitFilter = (filter != null);
     boolean hasDeletes = false;
     for (LeafReaderContext ctx : leaves) {
@@ -386,8 +385,7 @@ private List<FilterBitsetHandle> buildPerSegmentFilterHandles(
   private FilterBitsetHandle buildSegmentFilterHandle(
       Weight filterWeight, LeafReaderContext ctx, FloatVectorValues fvv) throws IOException {
     Bits liveDocs = ctx.reader().getLiveDocs();
-    // When filterWeight is null, accept all live documents (acceptDocs == liveDocs, which may itself
-    // be null to mean "all docs accepted" in this segment).
+    // Without an explicit filter, accept liveDocs directly; null means every document is live.
     Bits acceptDocs = (filterWeight != null) ? evalFilter(filterWeight, ctx, liveDocs) : liveDocs;
     Bits acceptedOrds = fvv.getAcceptOrds(acceptDocs);
     int numOrds = fvv.size();
@@ -479,10 +477,9 @@ private static CuVS2510GPUVectorsReader unwrapGpuReader(LeafReaderContext ctx, S
   /**
    * Builds a {@link Query} that matches exactly the given pre-scored documents.
    *
-   * <p>Partitions {@code scoreDocs} by segment (using {@link ScoreDoc#shardIndex} as the segment
-   * offset relative to {@link LeafReaderContext#docBase}), then returns a {@link Scorer} per
-   * segment that iterates those docs in ascending doc-ID order and replays their pre-computed
-   * scores.
+   * <p>Partitions {@code scoreDocs} by each global doc ID's membership in a leaf's {@link
+   * LeafReaderContext#docBase} range, then returns a {@link Scorer} per segment that iterates those
+   * docs in ascending doc-ID order and replays their pre-computed scores.
    */
   private static Query docAndScoreQuery(ScoreDoc[] scoreDocs) {
     return new Query() {

src/main/java/com/nvidia/cuvs/lucene/Lucene101AcceleratedHNSWCodec.java

@@ -1,5 +1,5 @@
 /*
- * SPDX-FileCopyrightText: Copyright (c) 2025-2026, NVIDIA CORPORATION.
+ * SPDX-FileCopyrightText: Copyright (c) 2025-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
  * SPDX-License-Identifier: Apache-2.0
  */
 package com.nvidia.cuvs.lucene;
@@ -52,7 +52,12 @@ public Lucene101AcceleratedHNSWCodec(String name, Codec delegate) {
    */
   public Lucene101AcceleratedHNSWCodec(AcceleratedHNSWParams acceleratedHNSWParams)
       throws Exception {
-    this(NAME, LuceneProvider.getCodec("101"));
+    this(NAME, LuceneProvider.getCodec("101"), acceleratedHNSWParams);
+  }
+
+  private Lucene101AcceleratedHNSWCodec(
+      String name, Codec delegate, AcceleratedHNSWParams acceleratedHNSWParams) {
+    super(name, delegate);
     initializeFormat(acceleratedHNSWParams);
   }
 

src/main/java/com/nvidia/cuvs/lucene/Lucene99AcceleratedHNSWVectorsFormat.java

@@ -1,5 +1,5 @@
 /*
- * SPDX-FileCopyrightText: Copyright (c) 2025-2026, NVIDIA CORPORATION.
+ * SPDX-FileCopyrightText: Copyright (c) 2025-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
  * SPDX-License-Identifier: Apache-2.0
  */
 package com.nvidia.cuvs.lucene;
@@ -81,7 +81,8 @@ public KnnVectorsWriter fieldsWriter(SegmentWriteState state) throws IOException
     } else {
       log.log(
           Level.WARNING,
-          "GPU based indexing not supported, falling back to using the Lucene99HnswVectorsWriter");
+          "GPU based indexing not supported, falling back to using the"
+              + " Lucene99HnswVectorsWriter");
       try {
         return LUCENE_PROVIDER.getLuceneHnswVectorsWriterInstance(
             state,

src/main/java/com/nvidia/cuvs/lucene/LuceneAcceleratedHNSWBinaryQuantizedCodec.java

@@ -1,5 +1,5 @@
 /*
- * SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION.
+ * SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
  * SPDX-License-Identifier: Apache-2.0
  */
 package com.nvidia.cuvs.lucene;
@@ -35,7 +35,7 @@ public LuceneAcceleratedHNSWBinaryQuantizedCodec(String name, Codec delegate) {
 
   public LuceneAcceleratedHNSWBinaryQuantizedCodec(AcceleratedHNSWParams acceleratedHNSWParams)
       throws Exception {
-    this(NAME, LuceneProvider.getCodec("101"));
+    super(NAME, LuceneProvider.getCodec("101"));
     initializeFormat(acceleratedHNSWParams);
   }