Python write support

[Fokko]

Feature Request / Improvement

This is a placeholder ticket for implementing write support for PyIceberg.

Since we don't want PyIceberg to write the actual data, and only handle the metadata part of the Iceberg table format, we need to get an overview of the frameworks we most likely want to integrate with (PyArrow, Dask (fastparquet?), etc).

I would suggest the following first steps to keep it simple: Write using PyArrow (since that's the most commonly used FileIO) and start with unpartitioned tables.

What we need:

• Avro write support: Python: Avro write support #7255
• Write files and extract statistics: Python: Write Parquet file using PyArrow #7256
• Ability to alter the Manifest JSON: Python: Ability to the Metadata JSON #7257
• Proper integration tests between Java and Python: Python: Integration tests #6398

Query engine

None

[nazq]

Keen to see this one progress, especially w pyarrow. Cc @asheeshgarg. Happy to help out

[Hayder-Aziz-cardano]

it would be really nice to get it working with polars (which utilises pyarrow / connectorx) as it provides the possibility of running execution services without large monolithic services like hive or spark.

[marsupialtail]

A great intermediate step would be to simply allow an engine like Polars/Pandas/Quokka to write out the actual parquet files in a given location with provided names that don't conflict with existing files, and then "register" those Parquet files in the iceberg metadata file.

[asheeshgarg]

One thing we can use is directly leverage Java APIs using PY4J or JYPE to call the underlying API. This will reduce the maintenance at two places.

[corleyma]

One thing we can use is directly leverage Java APIs using PY4J or JYPE to call the underlying API. This will reduce the maintenance at two places.

From my perspective at least, any solution that makes the JVM a dependency for users of the Python API is a poor solution.

[Matthieusalor]

Regarding statistics of written files, the write_to_dataset and write_dataset functions of pyarrow are providing a file_visitor argument that allows to retrieve the path and metadata of each written file.
https://arrow.apache.org/docs/python/generated/pyarrow.parquet.write_to_dataset.html

The metadata object is https://arrow.apache.org/docs/python/generated/pyarrow.parquet.FileMetaData.html
It allows to retrieve each row group metadata https://arrow.apache.org/docs/python/generated/pyarrow.parquet.RowGroupMetaData.html#pyarrow.parquet.RowGroupMetaData

Allowing to retrieve each column chunk metadata (including statistics) https://arrow.apache.org/docs/python/generated/pyarrow.parquet.ColumnChunkMetaData.html#pyarrow.parquet.ColumnChunkMetaData

[Fokko]

@Matthieusalor Thanks for the suggestion. I was looking into the metadata_collector: https://arrow.apache.org/docs/python/generated/pyarrow.parquet.write_metadata.html It looks like we can leverage this to collect the metadata during the write.

[luancaarvalho]

Hello everyone!
I was wondering if there have been any updates recently. Additionally, is there a roadmap available that shows if any contributors are currently working on the feature I am interested in?
Thank you!

[corleyma]

I dunno about roadmap, but there are some outstanding PRs in different stages of development that are directly relevant to this feature and are linked/referenced by the task list in the issue body.

Specifically, these PR seems worth following up on if you're looking to contribute:
#7831

Also, this repo organizes issues into milestones; there you can see milestones for the next two planned releases of pyiceberg and the issues that are currently assigned to each.

[Fokko]

Migrated to the new repository: apache/iceberg-python#23