ORC/Spark: forward-port default-value reads via idToConstant (LI #76)

[cbb330]

Summary

Forward-port of the Spark ORC default-value read path from LI #76 onto openhouse-1.2.0, adapted to initial-default and stacked on id-bound ORC structs (#265).

Absent defaulted fields are omitted from the per-file ORC projection and injected into idToConstant. The Spark row reader fills them. Omit is opt-in (supportsInitialDefaults) because ORCSchemaUtil is shared with Generic.

Vectorized ORC default-fill is out of scope. SparkBatchScan disables columnar ORC when the projection includes an initial-default. The vectorized-dir leftovers from #76 were dropped from this PR.

Mechanism (same as #76):

1. Omit absent defaulted fields from the per-file ORC projection
2. Inject their defaults into idToConstant
3. Existing ConstantReader materializes them

Inline notes mark each site as as-is or changed vs #76 (review).

Stack #270: this PR → #268 → #269.

---

Reviewer guide

As-is

Behavior	Where	Note
Overridable visitRecord	OrcSchemaWithTypeVisitor	as-is
Iceberg/ORC id alignment	containsInOrder	as-is
Row reader uses that visitor	SparkOrcReader	as-is
Nested convertConstant STRUCT/LIST/MAP	kept, unused	convertConstant

Changed

Divergence	This PR	Why	Note
Default API + omit	initialDefault(); omit gated	#250; ORCSchemaUtil is shared with Generic	omit · inject · gate
Who opts in	Spark row reader only	Generic does not fill yet; vectorized fill is out of scope	row · batch
Inject null-guard	Put only when initialDefault() != null	Avoid stuffing null constants	inject
Struct reader ctor	Id-bound struct(record, ...)	Required by #265	struct
Nested-typed defaults	Machinery kept; cannot declare yet	castDefault rejects non-null nested defaults (apache/iceberg#14611). Nested scalar defaults (e.g. loc.country) are covered	kept unused
Filter-on-defaulted-column tests	In #269	Needs file-level filter pushdown disabled for omitted columns	—

Follow-ups

• #268: omit only for complete embedded field IDs
• #269: skip file-level filters / empty-projection filters
• Later: Generic reader; lift castDefault + nested-typed defaults

Replaces #263/#264 for the Spark fill path.

---

Testing Done

export JAVA_HOME=$(/usr/libexec/java_home -v 11)
./gradlew -DsparkVersions=3.1 -DscalaVersion=2.12 -DhiveVersions= -DflinkVersions= \
  :iceberg-spark:iceberg-spark-3.1_2.12:test \
  --tests 'org.apache.iceberg.spark.data.TestSparkOrcReaderForFieldsWithDefaultValue' \
  --tests 'org.apache.iceberg.spark.source.TestSparkBatchScanInitialDefaults' \
  -x generateGitProperties

• testOrcScalarDefaultValues — pass
• testOrcNestedScalarDefaultValues — pass
• TestSparkBatchScanInitialDefaults — pass

[cbb330]

Inline annotations for the reviewer guide: forward-ported as-is vs forward-ported with changes. See the updated tables in the PR description for links back to each comment.

[cbb330]

Reviewer guide: as-is vs changed vs LI #76. Tables in the PR description link here.

[cbb330]

@mkuchenbecker carrying over your notes from #264 (and the Generic fill site on #263). Those PRs are closed; this is where the answers live.

Vectorized: no fill PR. Defaulted projections are forced onto the row reader (SparkBatchScan.hasNoInitialDefaults). BatchDataReader does not call supportsInitialDefaults(). Out of scope for this stack.

Spark 3.1 vs 3.5: this stack is openhouse-1.2.0 / Spark 3.1 (production OpenHouse). 3.5 is the 1.5.x line (id-binding is #257, separate).

convertConstant moved out of BaseDataReader: it was not. Option A introduced a new SparkValueConverters path. Here inject uses the existing BaseDataReader.convertConstant (same helper as partition constants).

Silent failure / is null an error: if we cannot prove the column was never written, we do not fill — we keep null (status quo). We do not throw and we do not invent a default. That is intentional. The case you wanted to avoid (silently overlaying a default on real data) is handled by refusing to omit, not by failing the read.

All data types: current coverage is int/string and a nested scalar (loc.country). castDefault still rejects nested-typed defaults (list/map/struct values). Broader scalar coverage (bool, decimal, timestamp, …) is a fair gap — not in this stack yet. Same for an OpenHouse Trino/Spark integration test; still outstanding.

Generic OrcValueReaders fill: not in this stack. Generic does not opt in. Fill is Spark row idToConstant → ConstantReader only.

[mkuchenbecker]

why is the vectorized dir needed?

[cbb330]

why is the vectorized dir needed?

@mkuchenbecker good catch, i removed them. they are dead code in this PR since I disabled vectorized.

these came forward from the original work from RZ from the forward port. but we aren't using it any time soon

[abhisheknath2011]

Approving to unblock the merge. @mkuchenbecker already approved.