docs: update the user guide for the 1.1.0 release (#6168)

* docs: update the user guide for the 1.1.0 release

Sweep of the user guide against everything merged since 1.0.0 branched
(af534e0), per the release preparation step in the release process.

Correct statements that no longer match the code: the CI coverage table
(only Spark 4.1 is gated before merge since the non-default profiles
moved to the nightly run), the Spark 3.x Parquet limitations Comet has
long since closed, cast/JSON/regex/array notes that say "falls back"
where Comet now routes through the codegen dispatcher, S3 region
auto-detection, Iceberg NOT IN pushdown, tokio thread defaults, shuffle
and scan metric names, and a Kubernetes example that pointed at a
Java 11 / Spark 3.5 image and left Comet disabled for lack of off-heap
memory.

Remove known-divergence entries that 1.1.0 fixed (boolean to decimal
cast, explicit positive timestamp years, the ABFS object store cache
key, most of the ANSI error-message items), and document new 1.1.0
behaviour: the experimental in-memory cache, the native Iceberg writer
and the WRITE credential access mode, nested hash shuffle keys, BINARY
sort-merge join keys, aggregate spill and memory metrics, native TIME
functions, Variant-bearing tables whose Variant columns are not read,
and the missing expression and plan-node rows.

Add an "Upgrading to Comet 1.1.0" section to the upgrade guide, and drop
the release-process notes about tag pushes publishing a Docker image,
since that workflow was removed in #4241.

* docs: address review feedback on the 1.1.0 user guide

Document the Kryo registrator under "Additional Configuration" in the
installation guide. Native broadcast needs it with
spark.kryo.registrationRequired=true whether or not the in-memory cache
is enabled, so link to it from the cache docs rather than describing it
as a cache-only setting. Point the operator compatibility cache section
and the operators table at the in-memory cache page that landed on main,
instead of repeating it, which also drops a stale storage-format claim.

List every source Comet checks for the Iceberg S3 region, and link the
ANSI integral overflow divergence to #6217.

Author: andygrove

docs/source/contributor-guide/release_process.md

@@ -287,9 +287,6 @@ git tag 0.13.0-rc1
 git push apache 0.13.0-rc1
 ```
 
-Note that pushing a release candidate tag will trigger a GitHub workflow that will build a Docker image and publish
-it to GitHub Container Registry at https://github.com/apache/datafusion-comet/pkgs/container/datafusion-comet
-
 ### Publishing Documentation
 
 In `docs` directory:
@@ -468,9 +465,6 @@ git tag 0.13.0
 git push apache 0.13.0
 ```
 
-Note that pushing a release tag will trigger a GitHub workflow that will build a Docker image and publish
-it to GitHub Container Registry at https://github.com/apache/datafusion-comet/pkgs/container/datafusion-comet
-
 Reply to the vote thread to close the vote and announce the release. The announcement email should include:
 
 - The release version

docs/source/user-guide/latest/compatibility/expressions/_category_template/cast.md

@@ -22,17 +22,23 @@ under the License.
 Cast operations in Comet fall into three levels of support:
 
 - **C (Compatible)**: The results match Apache Spark
-- **I (Incompatible)**: The results may match Apache Spark for some inputs, but there are known issues where some inputs
-  will result in incorrect results or exceptions. The query stage will fall back to Spark by default. Setting
-  `spark.comet.expression.Cast.allowIncompatible=true` will allow all incompatible casts to run natively in Comet, but this is not
-  recommended for production use.
-- **U (Unsupported)**: Comet does not provide a native version of this cast expression and the query stage will fall back to
-  Spark.
+- **I (Incompatible)**: Comet's native cast may produce different results from Apache Spark for some inputs. By
+  default, Comet instead runs Spark's own code-generated cast inside the Comet pipeline, so the results match Spark.
+  Setting `spark.comet.expression.Cast.allowIncompatible=true` selects the native cast, but this is not recommended
+  for production use.
+- **U (Unsupported)**: Comet does not provide a native version of this cast, so it runs Spark's own code-generated
+  cast inside the Comet pipeline.
 - **N/A**: Spark does not support this cast.
 
-## ANSI Mode Fallback
+Incompatible and unsupported casts fall back to Spark only when Comet's codegen dispatcher cannot handle them (for
+example, casts involving `VariantType`) or when `spark.comet.exec.scalaUDF.codegen.enabled=false`.
 
-Cast will fall back to Spark in some cases when ANSI mode is enabled. This can be enabled by setting `spark.comet.expression.Cast.allowIncompatible=true`. See the [Comet Supported Expressions Guide](../../../expressions.md) for more information on this configuration setting.
+## ANSI Mode Support
+
+Enabling ANSI mode does not by itself make a cast fall back to Spark or require
+`spark.comet.expression.Cast.allowIncompatible=true`: every cast that Spark permits in ANSI mode has the same support
+level as in legacy mode. See the [ANSI Mode](#ansi-mode) table below for per-pair support, and the
+[Comet Supported Expressions Guide](../../../expressions.md) for more information on the `allowIncompatible` setting.
 
 ## Whitespace Trimming in Casts from String
 
@@ -143,20 +149,12 @@ Comet's native `CAST(date AS STRING)` is compatible with Spark. Years below 1000
 zero-padded to four digits (e.g. year 999 renders as `0999-01-01`). Years above 9999 are
 rendered without truncation. The cast is timezone-independent.
 
-## String to TimestampNTZ
-
-Comet's native `CAST(string AS TIMESTAMP_NTZ)` implementation matches Apache Spark's behavior.
-Unlike `CAST(string AS TIMESTAMP)`, this cast is timezone-independent: any timezone offset in
-the input string (e.g. `+08:00`, `Z`, `UTC`) is silently discarded, and the local date-time
-components are preserved as-is. Time-only strings (e.g. `T12:34:56`, `12:34`) produce `NULL`.
-The result is always a wall-clock timestamp with no timezone conversion or DST adjustment.
-
 ## Decimal with Negative Scale to String
 
 Casting a `DecimalType` with a negative scale to `StringType` is marked as incompatible when
 `spark.sql.legacy.allowNegativeScaleOfDecimal` is `false` (the default). When that config is
-disabled, Spark cannot create negative-scale decimals, so Comet falls back to avoid running
-native execution on unexpected inputs.
+disabled, Spark cannot create negative-scale decimals, so by default Comet runs Spark's
+code-generated cast instead of the native one.
 
 When `spark.sql.legacy.allowNegativeScaleOfDecimal=true`, the cast is compatible. Comet matches
 Spark's behavior of using Java `BigDecimal.toString()` semantics, which produces scientific

docs/source/user-guide/latest/compatibility/expressions/_category_template/map.md

@@ -26,7 +26,7 @@ keys, in `try_element_at`, and in other contexts where map ordering must be dete
 runs `MapSort` natively, so map shuffle and group-by-on-map stay on Comet under Spark 4.0.
 
 When `spark.comet.exec.strictFloatingPoint=true`, `MapSort` falls back to Spark for maps whose
-keys contain `Float` or `Double` (consistent with `SortOrder` and `SortArray`). Arrow's sort uses
+keys contain `Float` or `Double`. Arrow's sort uses
 IEEE total ordering for floating-point, which differs from Spark's `Double.compare` semantics for
 `NaN` and `-0.0`.
 

docs/source/user-guide/latest/compatibility/expressions/index.md

@@ -32,6 +32,9 @@ Spark 4.0 <spark-4.0/index>
 Spark 4.1 <spark-4.1/index>
 ```
 
+Spark 3.4 support is deprecated and will be removed in a future release; see
+[Spark Version Compatibility](../spark-versions.md#spark-34).
+
 Expressions that are not 100% Spark-compatible fall back to Spark by default, except those
 with a JVM codegen-dispatch path, which stay in Comet's native pipeline and match Spark
 exactly. Set `spark.comet.expression.EXPRNAME.allowIncompatible=true`, where `EXPRNAME` is

docs/source/user-guide/latest/compatibility/expressions/spark-3.4/index.md

@@ -27,6 +27,9 @@ expression class name, to run Comet's native implementation despite its differen
 from Spark. See the [Comet Supported Expressions Guide](../../../expressions.md)
 for more information on this configuration setting.
 
+Spark 3.4 support is deprecated and will be removed in a future release; see
+[Spark Version Compatibility](../../spark-versions.md#spark-34).
+
 ```{toctree}
 :maxdepth: 1
 

docs/source/user-guide/latest/compatibility/index.md

@@ -68,7 +68,8 @@ which document their per-expression configs and the specific differences to expe
 
 This is distinct from expressions that have **no** codegen-dispatch path: there, the
 incompatible cases fall back to Spark by default, and `allowIncompatible=true` runs the native
-(incompatible) path instead. `cast` is the main example; see the
+(incompatible) path instead. Aggregate functions such as `mode` are the main example, because the
+codegen dispatcher covers only scalar expressions; see the
 [expression reference](../expressions.md) for which expressions have incompatible cases.
 
 ## Strings with non-UTF-8 bytes
@@ -111,12 +112,10 @@ Code that catches `SparkException` and only asserts on message substrings is una
 inspects the exception class, `getCondition()`, or the parameterised error class will observe
 divergence:
 
-- Byte / Short `Add`, `Subtract`, and `Multiply` overflow raises `ARITHMETIC_OVERFLOW` where Spark
-  4.1 raises `BINARY_ARITHMETIC_OVERFLOW`, and Long overflow surfaces as `"integer overflow"`
-  rather than `"long overflow"`. `Abs` uses Rust type names (`Int8`, `Int64`, ...) in the message
-  instead of Spark's SQL type names, and the scalar path of `UnaryMinus` on Byte / Short emits a
-  malformed message. The `try_` suggestion is omitted from all of these
-  ([#5071](https://github.com/apache/datafusion-comet/issues/5071)).
+- Byte / Short `Add`, `Subtract`, and `Multiply` overflow raises `ARITHMETIC_OVERFLOW` (for
+  example `byte overflow`) where Spark raises `BINARY_ARITHMETIC_OVERFLOW`, and integral
+  `ARITHMETIC_OVERFLOW` messages omit Spark's `try_` suggestion
+  ([#6217](https://github.com/apache/datafusion-comet/issues/6217)).
 - Wide-decimal arithmetic overflow, decimal divide-by-zero, and decimal-to-decimal cast overflow
   raise raw Arrow errors that bypass `SparkErrorConverter` and surface as `CometNativeException`
   rather than `SparkArithmeticException` with the proper error class and query context
@@ -130,18 +129,11 @@ The following native paths silently return values that differ from Spark for edg
 Most also have entries in the per-category expression pages linked above; they are collected here
 so users hunting an unexpected value have a single place to check:
 
-- `CAST(boolean AS DECIMAL(p, s))` where `10^s` exceeds the target precision (e.g.
-  `DECIMAL(1, 1)`) throws `NUMERIC_VALUE_OUT_OF_RANGE` regardless of the eval mode. Spark returns
-  `NULL` under legacy and try mode, and only throws under ANSI
-  ([#5068](https://github.com/apache/datafusion-comet/issues/5068)).
 - `CAST(string AS timestamp)` and `CAST(string AS timestamp_ntz)` trim Unicode whitespace.
   Spark trims only the bytes `0x00`-`0x20` and `0x7F`, so a value padded with an ASCII control byte
   parses in Spark and returns `NULL` in Comet, while a value padded with non-ASCII whitespace such
   as `U+3000` returns `NULL` in Spark and parses in Comet
   ([#5149](https://github.com/apache/datafusion-comet/issues/5149)).
-- **Explicit positive timestamp years:** Spark accepts strings such as `+7528` as the start
-  of that year, while Comet's native string-to-timestamp cast returns NULL in non-ANSI mode
-  ([#5716](https://github.com/apache/datafusion-comet/issues/5716)).
 - Native `RANGE` window frames with an explicit `PRECEDING` / `FOLLOWING` offset diverge from
   Spark when the boundary arithmetic overflows for `DATE` or `DECIMAL` `ORDER BY` columns
   ([#5022](https://github.com/apache/datafusion-comet/issues/5022)).
@@ -156,13 +148,3 @@ so users hunting an unexpected value have a single place to check:
   `spark.sql.codegen.fallback` or when the generated code exceeds
   `spark.sql.codegen.hugeMethodLimit`, where an intermediate overflow that later cancels out
   returns `NULL` (or raises under ANSI) in Spark but the recovered value in Comet.
-
-## Object store cache
-
-When Comet's native scan reads Parquet files, it caches one object store instance per
-`(scheme + host + port, hadoop-config-hash)` key. For `abfss://container@account.dfs.core.windows.net/...`
-URLs, the container lives in the URL userinfo, not the host, so two containers in one storage
-account currently collide on the same cache entry. Within a single executor process, reading from
-a second container after a first can be served by the first container's store instance and return
-its data. S3, GCS, and HDFS are unaffected because their bucket / host lives in the URL host
-component. Tracked by [#4993](https://github.com/apache/datafusion-comet/issues/4993).

docs/source/user-guide/latest/compatibility/operators.md

@@ -32,6 +32,18 @@ operator restrictions and aggregate buffer compatibility checks still apply.
 Parquet writes whose input plans contain an empty relation use Spark's writer to preserve
 readable empty output files and their schema metadata.
 
+## In-Memory Cache
+
+Comet can store cached relations (`df.cache()`, `CACHE TABLE`) in Arrow format and scan them
+natively. This is experimental and disabled by default; see [In-Memory Cache](../in-memory-cache.md)
+for how to enable it. Comet does not replace a `spark.sql.cache.serializer` that the application
+has already set. Relations whose schema Comet's Arrow writer does not support are cached in
+Spark's default format, and their scans fall back to Spark. Reads that feed Spark operators rather
+than Comet operators can be slower than Spark's cache.
+
+With Kryo and `spark.kryo.registrationRequired=true`, Comet needs its Kryo registrator whether or
+not the cache is enabled; see [Kryo serialization](../installation.md#kryo-serialization).
+
 ## Sampling
 
 Comet runs `SampleExec` natively when sampling is performed without replacement, which covers
@@ -95,8 +107,11 @@ runs natively; it is controlled by `spark.comet.exec.windowGroupLimit.enabled` (
 
 **Known incompatibilities:**
 
-- Signed-zero ordering (`-0.0` vs `+0.0`) diverges from Spark's `RankLimitIterator`; see
-  [floating-point ordering](./floating-point.md#ordering-signed-zero-00-vs-00).
+- Floating-point values nested in array or struct `ORDER BY` keys are compared with Arrow's raw
+  total ordering, so ranks can differ from Spark when the data mixes `-0.0` and `+0.0` or more
+  than one NaN representation ([#5507](https://github.com/apache/datafusion-comet/issues/5507)).
+  Scalar `FLOAT` and `DOUBLE` keys are normalized and match Spark; see
+  [floating-point ordering](./floating-point.md).
 
 ## Round-Robin Partitioning
 

docs/source/user-guide/latest/compatibility/scans.md

@@ -48,6 +48,8 @@ The following features are not supported and cause Comet to fall back to Spark:
   does not replicate. By default Comet falls back to Spark in this case. Set
   `spark.comet.scan.allowDisabledParquetVectorizedReader=true` to opt in to running the
   Comet Parquet scan regardless.
+- A read schema that repeats a Parquet field id, at the top level or within a struct, when
+  `spark.sql.parquet.fieldId.read.enabled=true`.
 
 The following limitation may produce incorrect results without falling back to Spark:
 

docs/source/user-guide/latest/compatibility/spark-versions.md

@@ -33,40 +33,32 @@ Spark 3.4.3 is supported with Java 17 and Scala 2.12/2.13.
 ```{warning}
 Spark 3.4 support is deprecated as of the 1.0.0 release and will be removed in a future release.
 Comet continues to build and publish Spark 3.4 binaries in the meantime, but Apache Spark's own
-SQL test suite is no longer run against Spark 3.4 on every change: it runs only when a contributor
+SQL test suite no longer runs against Spark 3.4 automatically: it runs only when a contributor
 opts a pull request into it. Regressions specific to Spark 3.4 are therefore more likely to reach a
 release than on the other supported versions. We recommend moving to Spark 3.5 or later.
 ```
 
 ### Known Limitations
 
-- **Reading `TimestampLTZ` as `TimestampNTZ`**: Spark 3.4 raises an error for this operation
-  (SPARK-36182), but Comet's Parquet scan silently returns the raw UTC value instead.
-  See [Parquet Compatibility](scans.md#parquet-scan-limitations) for details.
-
-- **Unsupported Parquet type conversions**: Spark 3.4 raises schema incompatibility errors for
-  certain type mismatches (e.g., reading INT32 as BIGINT, decimal precision changes), but Comet's
-  Comet's Parquet scan may not detect these and could return unexpected values.
-  See [Parquet Compatibility](scans.md#parquet-scan-limitations) for details.
+- **Extra `SparkException` layer in Parquet schema mismatch errors**: when a Parquet read is
+  rejected because a file's type cannot be converted to the requested type, the error's cause chain
+  has one more `SparkException` layer than Spark's own reader produces.
+  See [Parquet Compatibility](scans.md#schema-mismatch-handling) for details.
 
 ## Spark 3.5
 
 Spark 3.5.9 is supported with Java 17 and Scala 2.12/2.13.
 
 ### Known Limitations
 
-- **Reading `TimestampLTZ` as `TimestampNTZ`**: Spark 3.5 raises an error for this operation
-  (SPARK-36182), but Comet's Parquet scan silently returns the raw UTC value instead.
-  See [Parquet Compatibility](scans.md#parquet-scan-limitations) for details.
-
-- **Unsupported Parquet type conversions**: Spark 3.5 raises schema incompatibility errors for
-  certain type mismatches (e.g., reading INT32 as BIGINT, decimal precision changes), but Comet's
-  Comet's Parquet scan may not detect these and could return unexpected values.
-  See [Parquet Compatibility](scans.md#parquet-scan-limitations) for details.
+- **Extra `SparkException` layer in Parquet schema mismatch errors**: when a Parquet read is
+  rejected because a file's type cannot be converted to the requested type, the error's cause chain
+  has one more `SparkException` layer than Spark's own reader produces.
+  See [Parquet Compatibility](scans.md#schema-mismatch-handling) for details.
 
 ## Spark 4.0
 
-Spark 4.0.4 is supported with Java 17 and Scala 2.13.
+Spark 4.0.4 is supported with Java 17/21 and Scala 2.13.
 
 ### Known Limitations
 
@@ -93,6 +85,7 @@ Spark 4.1.3 is supported with Java 17/21 and Scala 2.13.
 Spark 4.2.0 is provided as experimental support with Java 17 and Scala 2.13.
 
 ```{warning}
-Spark 4.2 support is experimental. Comet tests run in CI for this version, but the Spark SQL tests
-do not yet. It is intended for early evaluation only and should not be used in production.
+Spark 4.2 support is experimental. Comet's own test suites run nightly for this version, but
+Apache Spark's SQL test suite does not run against it yet. It is intended for early evaluation only
+and should not be used in production.
 ```

docs/source/user-guide/latest/datasources.md

@@ -29,15 +29,17 @@ Arrow format, allowing the Comet pipeline to take over after that, but the proce
 
 ### Apache Iceberg
 
-Comet accelerates Iceberg scans of Parquet files. See the [Iceberg Guide] for more information.
+Comet accelerates Iceberg scans of Parquet files and has an experimental, opt-in native Iceberg writer.
+See the [Iceberg Guide] and [Iceberg Writes](iceberg-writes.md) for more information.
 
 [Iceberg Guide]: iceberg.md
 
 ### CSV
 
 Comet provides experimental Rust-based CSV scan support. When `spark.comet.scan.csv.v2.enabled` is enabled, CSV files
 are read in Rust for improved performance. This feature is experimental and performance benefits are
-workload-dependent.
+workload-dependent. Only Spark's DataSource V2 CSV scan is accelerated, and Spark reads CSV through the V1 API by
+default, so also remove `csv` from `spark.sql.sources.useV1SourceList`.
 
 Alternatively, when `spark.comet.convert.csv.enabled` is enabled, data from Spark's CSV reader is immediately
 converted into Arrow format, allowing the Comet pipeline to take over after that.
@@ -308,6 +310,10 @@ credential providers and options documented above also apply to alias-scheme URL
 translation feeds the native Iceberg scan; see
 [Object store configuration (S3)](iceberg.md#object-store-configuration-s3) in the Iceberg guide.
 
+A native Parquet scan whose alias-scheme paths span more than one bucket falls back to Spark. Alias
+schemes apply to native scans only: a native Iceberg write to an alias-scheme location falls back to
+iceberg-java.
+
 ### Examples
 
 The following examples demonstrate how to configure S3 access using different authentication methods.
@@ -344,7 +350,7 @@ Comet's S3 support has the following limitations:
 
 1. **Partial Hadoop S3A configuration support**: Not all Hadoop S3A configurations are currently supported. Only the configurations listed in the tables above are translated and applied to the underlying `object_store` crate.
 
-2. **Custom credential providers**: Custom implementations of AWS credential providers are not supported. The implementation only supports the standard credential providers listed in the table above. We are planning to add support for custom credential providers through a JNI-based adapter that will allow calling Java credential providers from Rust code. See [#1829](https://github.com/apache/datafusion-comet/issues/1829) for more details.
+2. **Custom credential providers**: Custom credential provider classes named in `fs.s3a.aws.credentials.provider` are not supported; only the standard providers listed in the table above are. To route credential requests through your own Java code, implement Comet's `CometS3CredentialProvider` SPI; see [S3 Credential Providers](s3-credential-providers.md). Broader Hadoop S3A integration is tracked in [#1829](https://github.com/apache/datafusion-comet/issues/1829).
 
 ## Azure