`write-default` null representation issue

[Tishj]

Apache Iceberg version

None

Query engine

None

Please describe the bug 🐞

TL;DR:
It's not always possible to set the DEFAULT value for newly written rows to NULL, without also altering what the DEFAULT value for existing rows should be.

Because of the backwards-compatibility of write-default, it will default to initial-default if it's not present.
To represent NULL, the default has to be omitted.

null seems to not be an accepted state for either initial-default or write-default.
The spec doesn't say they're nullable, and the iceberg-rest-fixture Catalog will return 500 if it reads a initial-default or write-default that is null

That means it's impossible to set the write-default to NULL, unless initial-default is also NULL (omitted)

Table to illustrate the problem:

┌──────────────────┬────────────────┬────────────────────┐
│ initial-default  │ write-default  │ result (new rows)  │
├──────────────────┼────────────────┼────────────────────┤
│ 5                │ -              │ 5                  │
│ -                │ 21             │ 21                 │
│ 5                │ 21             │ 21                 │
│ -                │ -              │ NULL               │
└──────────────────┴────────────────┴────────────────────┘

write-default is omitted (-), but because initial-default is non-null, the result won't be NULL

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time