feat: Add array data type support by charlesdong1991 · Pull Request #433 · apache/fluss-rust

charlesdong1991 · 2026-03-07T16:02:37Z

Purpose

Linked issue: close #386

Brief change log

Add end-to-end ARRAY column support in fluss-rust

Tests

Yes all tests are passed locally

charlesdong1991 · 2026-03-07T16:06:56Z

crates/fluss/src/row/compacted/compacted_row_reader.rs

                        let (val, next) = reader.read_long(cursor);
-                        let decimal = Decimal::from_unscaled_long(val, precision, scale)
-                            .expect("Failed to create decimal from unscaled long");
+                        let decimal =


i think we should convert all expect to typed errors so malformed array payload handling does not leave panic backdoor

charlesdong1991 · 2026-03-07T16:10:02Z

crates/fluss/src/row/compacted/compacted_key_writer.rs

+        // silently widen key semantics.
+        if matches!(
+            field_type,
+            DataType::Array(_) | DataType::Map(_) | DataType::Row(_)


this applies to all array/map/row, so i understand this PR only adds Array, but i think better to add all of them.
if there is objection, i can remove

luoyuxia · 2026-03-08T02:31:21Z

@charlesdong1991 Thanks for the great pr. I'll find some time to review

Copilot

Pull request overview

Adds end-to-end ARRAY column support to the Fluss Rust client, including a Java-compatible binary array representation and integration across row encoders/decoders, Arrow interop, and public APIs.

Changes:

Introduces FlussArray / FlussArrayWriter (BinaryArray layout) and wires it into Datum, InternalRow, and runtime field/value writers.
Adds compacted row read/write support for arrays (length-prefixed BinaryArray bytes) and rejects ARRAY as a key column type.
Extends Arrow conversion logic and updates user/docs + tests for primitive, nullable, empty, and nested arrays.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
website/docs/user-guide/rust/data-types.md	Documents `ARRAY<T>` type mapping and how to construct/read arrays.
website/docs/user-guide/rust/api-reference.md	Adds `InternalRow::get_array` and documents `FlussArray` API.
crates/fluss/src/row/mod.rs	Exposes `binary_array` module and re-exports `FlussArray`; adds `InternalRow::get_array`.
crates/fluss/src/row/field_getter.rs	Adds `Array` field getter + tests.
crates/fluss/src/row/encode/compacted_key_encoder.rs	Adds test ensuring array types are rejected in key encoding.
crates/fluss/src/row/datum.rs	Adds `Datum::Array` and Arrow ListBuilder append/conversion logic.
crates/fluss/src/row/compacted/compacted_row_writer.rs	Adds `write_array` (delegates to length-prefixed bytes).
crates/fluss/src/row/compacted/compacted_row_reader.rs	Makes deserialization fallible and adds array decoding via `FlussArray::from_bytes`.
crates/fluss/src/row/compacted/compacted_row.rs	Propagates fallible deserialization and adds `InternalRow::get_array` + array tests.
crates/fluss/src/row/compacted/compacted_key_writer.rs	Explicitly rejects complex key types and adds `write_array` to the delegated writer surface.
crates/fluss/src/row/column.rs	Implements `ColumnarRow::get_array` by converting Arrow `ListArray` → `FlussArray`; adds tests.
crates/fluss/src/row/binary_array.rs	New Java-compatible binary array implementation + writer + tests.
crates/fluss/src/row/binary/binary_writer.rs	Adds `BinaryWriter::write_array` and `InnerValueWriter::Array`.
crates/fluss/src/record/arrow.rs	Adds Arrow `List` builder support and `from_arrow_type` mapping for list element conversion.
bindings/cpp/src/types.rs	Propagates `Datum::Array` handling in type resolution and row ownership conversion.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

crates/fluss/src/row/binary_array.rs

crates/fluss/src/row/datum.rs

crates/fluss/src/row/compacted/compacted_row_reader.rs

crates/fluss/src/row/compacted/compacted_key_writer.rs

crates/fluss/src/row/column.rs

crates/fluss/src/row/binary_array.rs

fresh-borzoni

@charlesdong1991 LGTM overall, left minor comments
If you fixed smth in the meantime - good, sorry for the noise, as I started reviewing before the new code :)

crates/fluss/src/row/datum.rs

crates/fluss/src/row/column.rs

crates/fluss/src/record/arrow.rs

crates/fluss/src/row/binary_array.rs

bindings/cpp/src/types.rs

crates/fluss/src/row/datum.rs

charlesdong1991 · 2026-03-08T14:48:21Z

Hi @fresh-borzoni thanks a lot!
It would be great if you can give another round of reviews since i made some changes during your review ^^

leekeiabstraction

Thank you for the PR! Left some comments. PTAL

leekeiabstraction · 2026-03-08T20:36:12Z

crates/fluss/src/record/arrow.rs

+                let inner_builder = Self::create_builder(field.data_type(), capacity)?;
+                Ok(Box::new(ListBuilder::with_capacity(inner_builder, capacity)))


This reads like we are over provisioning here with capacity^2?

it's capacity + capacity, bc it's offset buffer and values buffer, it's fine, though we use lower bound of each list - 1 element, which is fine for now I guess

leekeiabstraction · 2026-03-08T20:40:21Z

crates/fluss/src/record/arrow.rs


+/// Converts an Arrow data type back to a Fluss `DataType`.
+/// Used for reading array elements from Arrow ListArray back into Fluss types.
+pub fn from_arrow_type(arrow_type: &ArrowDataType) -> Result<DataType> {


Should this be pub (crate)?

Also, have you check if there is an already existing function that does something similar? (Reads familiar but I do not have access to codebase right now)

+1 pub (crate), but I don't think we have similar function

leekeiabstraction · 2026-03-08T20:56:57Z

crates/fluss/src/row/encode/compacted_key_encoder.rs

            // TIMESTAMP_LTZ(1): 1698235273182
            0xDE, 0x9F, 0xD7, 0xB5, 0xB6, 0x31,
            // TIMESTAMP_LTZ(5): 1698235273182
            0xDE, 0x9F, 0xD7, 0xB5, 0xB6, 0x31, 0x00,


I vaguely remember that Java side does encode array.

See this PR which was created to give us a way to check that compacted key encoding is consistent across both Java and Rust

https://github.com/apache/fluss/pull/2312/files

+ 1 it seems that server rejects arrays as primary key, but encoding is allowed

fresh-borzoni

@charlesdong1991 TY, looked through again, left comments
PTAL

fresh-borzoni · 2026-03-08T23:47:32Z

crates/fluss/src/row/binary_array.rs

+        data.truncate(self.cursor);
+        FlussArray::from_bytes(&data)
+    }
+


we already own data Vec, so from_bytes would unnecessary copy

fresh-borzoni · 2026-03-09T01:00:29Z

crates/fluss/src/row/binary_array.rs

+                ),
+            });
+        }
+        let raw_size = i32::from_ne_bytes(data[0..4].try_into().unwrap());


I think Java uses LE

fresh-borzoni · 2026-03-09T01:00:49Z

crates/fluss/src/row/binary_array.rs

+
+    pub fn get_short(&self, pos: usize) -> i16 {
+        let offset = self.element_offset(pos, 2);
+        i16::from_ne_bytes(self.data[offset..offset + 2].try_into().unwrap())


ditto and all other cases in this file

fresh-borzoni · 2026-03-09T01:04:08Z

crates/fluss/src/row/encode/compacted_key_encoder.rs

            // TIMESTAMP_LTZ(1): 1698235273182
            0xDE, 0x9F, 0xD7, 0xB5, 0xB6, 0x31,
            // TIMESTAMP_LTZ(5): 1698235273182
            0xDE, 0x9F, 0xD7, 0xB5, 0xB6, 0x31, 0x00,


+ 1 it seems that server rejects arrays as primary key, but encoding is allowed

fresh-borzoni · 2026-03-09T01:18:42Z

crates/fluss/src/row/column.rs

+                    writer.set_null_at(i);
+                } else {
+                    let nested_values = list_arr.value(i);
+                    let nested_element_type = from_arrow_type(nested_values.data_type())?;


we allocate DataType in a loop, it's wasteful. Shall we hoist it?

fresh-borzoni · 2026-03-09T01:23:07Z

crates/fluss/src/row/binary_array.rs

+        (offset, size)
+    }
+
+    pub fn get_string(&self, pos: usize) -> Result<&str> {


why do we return Result in half of the getters and half of the methods go unchecked?
I doubt that Java is the same half-way, pls, check

fresh-borzoni · 2026-03-09T01:27:19Z

crates/fluss/src/record/arrow.rs


+/// Converts an Arrow data type back to a Fluss `DataType`.
+/// Used for reading array elements from Arrow ListArray back into Fluss types.
+pub fn from_arrow_type(arrow_type: &ArrowDataType) -> Result<DataType> {


+1 pub (crate), but I don't think we have similar function

fresh-borzoni · 2026-03-09T01:35:25Z

crates/fluss/src/row/mod.rs

+
+    fn get_array(&self, pos: usize) -> Result<FlussArray> {
+        match self.get_value(pos)? {
+            Datum::Array(a) => Ok(a.clone()),


it clones on everY call, mb we wish to use Bytes to zero-copy?

fresh-borzoni · 2026-03-09T01:42:02Z

crates/fluss/src/record/arrow.rs

+                let inner_builder = Self::create_builder(field.data_type(), capacity)?;
+                Ok(Box::new(ListBuilder::with_capacity(inner_builder, capacity)))


it's capacity + capacity, bc it's offset buffer and values buffer, it's fine, though we use lower bound of each list - 1 element, which is fine for now I guess

charlesdong1991 added 2 commits March 7, 2026 16:59

add array data type support

5a8ca2e

add docs

e35adce

charlesdong1991 commented Mar 7, 2026

View reviewed changes

luoyuxia requested a review from Copilot March 8, 2026 02:29

Copilot started reviewing on behalf of luoyuxia March 8, 2026 02:29 View session

Copilot AI reviewed Mar 8, 2026

View reviewed changes

charlesdong1991 added 3 commits March 8, 2026 11:45

resolve conflicts

a8fdac0

address comments

8cacf63

avoid intermediate record batch in get-array

8208023

fresh-borzoni reviewed Mar 8, 2026

View reviewed changes

Address comments

c29c5ad

leekeiabstraction reviewed Mar 8, 2026

View reviewed changes

fresh-borzoni reviewed Mar 9, 2026

View reviewed changes

		let inner_builder = Self::create_builder(field.data_type(), capacity)?;
		Ok(Box::new(ListBuilder::with_capacity(inner_builder, capacity)))

Conversation

charlesdong1991 commented Mar 7, 2026

Purpose

Brief change log

Tests

Uh oh!

charlesdong1991 Mar 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

luoyuxia commented Mar 8, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fresh-borzoni left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

charlesdong1991 commented Mar 8, 2026

Uh oh!

leekeiabstraction left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fresh-borzoni left a comment

Choose a reason for hiding this comment

Uh oh!

fresh-borzoni Mar 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fresh-borzoni Mar 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

charlesdong1991 Mar 7, 2026 •

edited

Loading

fresh-borzoni left a comment •

edited

Loading

fresh-borzoni Mar 8, 2026 •

edited

Loading

fresh-borzoni Mar 9, 2026 •

edited

Loading