You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Destructure proto hooks for the Arrow and Avro sources #24622
Both nodes carry only base_conf on the wire. Every remaining field (table_schema, projection, batch_size, metrics) is either carried by the base config, set at execution time, or runtime state -- so this group should be a clean, purely mechanical conversion. It is a good first issue for the EPIC.
Why
Serde hooks that read state through getters or self.field make an added field invisible to serialization: nothing breaks at compile time, the field simply stops round-tripping, and Debug-comparing round-trip tests do not notice. HashJoinExec::fetch was lost exactly this way (#24165), and #24609 is a second live instance found by applying the convention to one file.
#24164 established the fix -- exhaustive destructuring in both directions -- and applied it to the join plans. The physical-plan plan nodes are done. physical-expr and the datasource* crates were never converted.
What to do
For each hook in scope:
In try_to_proto, start with an exhaustive let Self {{ .. }} -- no .. rest pattern. Fields that are genuinely not serialized bind to _ with a short comment saying why (derived at construction, runtime state, recomputed on decode, carried by a parent message).
In try_from_proto, destructure the prost-generated node struct the same way, so adding a field to datafusion.proto is a compile error in every decoder.
If the destructure turns up a field that should round-trip but has no wire representation, add it to the message and cover it with a test that fails without the fix.
Definition of done
Every field of the plan/expression struct is either serialized or bound to _ with a reason.
Every field of the prost node is destructured in the decoder.
Any newly serialized field has a round-trip test, verified to fail before the fix.
Activity
datasourcecrates #24612buraksenn commentedon Aug 24, 2026
take
refactor(proto): destructure Arrow and Avro source serde hooks (apach…