Skip to content

Bump linkedin/iceberg to 1.2.0.23 / 1.5.2.20 (ORC initial-defaults) - #701

Open
cbb330 wants to merge 1 commit into
mainfrom
chbush/bump-iceberg-orc-defaults
Open

Bump linkedin/iceberg to 1.2.0.23 / 1.5.2.20 (ORC initial-defaults)#701
cbb330 wants to merge 1 commit into
mainfrom
chbush/bump-iceberg-orc-defaults

Conversation

@cbb330

@cbb330 cbb330 commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator

Summary

Bumps both OpenHouse Iceberg pins so Spark 3.1 and Spark 3.5 pick up the ORC/Spark initial-default read path (idToConstant) and the complete-embedded-field-id omit rule.

  • iceberg_1_2_version: 1.2.0.201.2.0.23 (published as v1.2.0.23)
  • iceberg_1_5_version: 1.5.2.171.5.2.20 (next two Shipkit tags after current v1.5.2.18, once linkedin/iceberg#273 then linkedin/iceberg#274 merge)

CI is expected to fail resolving 1.5.2.20 until those Iceberg PRs merge and Shipkit publishes the tags. Rerun CI after v1.5.2.20 exists. Does not include linkedin/iceberg#269.

Changes

  • Client-facing API Changes
  • Internal API Changes
  • Bug Fixes
  • New Features
  • Performance Improvements
  • Code Style
  • Refactoring
  • Documentation
  • Tests

Spark/ORC readers fill missing columns from Iceberg initial defaults via idToConstant, and skip that fill only when the file already embeds a complete field-id set.

1.2 line: linkedin/iceberg#267 (v1.2.0.22) and linkedin/iceberg#268 (v1.2.0.23), plus CI-only linkedin/iceberg#266 (v1.2.0.21).

1.5 line: linkedin/iceberg#257 is already v1.5.2.18; #273 / #274 are the 1.5 twins of #267 / #268 and become v1.5.2.19 / v1.5.2.20 if they are the next two merges on openhouse-1.5.2.

Testing Done

  • Manually Tested on local docker setup. Please include commands ran, and their output.
  • Added new tests for the changes made.
  • Updated existing tests to reflect the changes made.
  • No tests added or updated. Please explain why. If unsure, please feel free to ask for help.
  • Some other form of testing like staging or soak time in production. Please explain.

Version pin only. Iceberg coverage lives in linkedin/iceberg #267/#268 and #273/#274.

Additional Information

  • Breaking Changes
  • Deprecations
  • Large PR broken into smaller PRs, and PR plan linked in the description.

Depends on merge order 273 then 274 on linkedin/iceberg (openhouse-1.5.2). Retarget 274 onto openhouse-1.5.2 after 273 lands. If another openhouse-1.5.2 PR merges first, the 1.5 pin needs to move to the actual tag that contains #274.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant