Skip to content

[VL] Spark 4.1: Support NullType Parquet read/write (SPARK-54220) #11921

@baibaichen

Description

@baibaichen

Backend

VL (Velox)

Gluten version: main branch

Description

Spark 4.1 added support for reading and writing NullType columns in Parquet (SPARK-54220). Gluten/Velox does not handle NullType in Parquet format.

Spark 4.1 only.

Parent issue: #11910 ([VL] Spark 4.x: Tracking new feature support)

Impact

Suite Exclude spark40 spark41
GlutenParquetIOSuite SPARK-54220 🟢 🔴

Note: GlutenParquetIOSuite also has struct field excludes tracked under a separate #11550 sub-issue.

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type
    No fields configured for issues without a type.

    Projects

    Status

    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions