feat(datafusion): fill omitted INSERT columns with Iceberg write-default values

[moomindani]

Which issue does this PR close?

• Closes DataFusion INSERT fills omitted columns with NULL instead of the column's write-default #2803. Part of EPIC: v3 Support Tracking #2411 (Default Values).

What changes are included in this PR?

Per the spec, writers must use a column's write-default for columns they do not supply. DataFusion's insert planner consults TableProvider::get_column_default for columns omitted from an INSERT and falls back to NULL; IcebergTableProvider did not implement it, so tables with write-default values silently got NULLs.

• IcebergTableProvider caches the schema's top-level write-default values as DataFusion expressions at construction and serves them via get_column_default.
• Adds a literal_to_scalar_value conversion covering the primitive types (boolean, int, long, float, double, string, date, time, timestamp/timestamptz in µs and ns, decimal, binary, fixed, uuid); defaults with no scalar representation are skipped. The planner casts the expression to the target arrow type, so representation differences are reconciled downstream.
• This mirrors iceberg-java's approach of delegating write-default application to the engine: its Spark integration exposes write-default as the engine column default (TypeToSparkType), and Spark materializes it at INSERT planning.
• The static provider is read-only and keeps the trait default (no column defaults).

Are these changes tested?

Yes — an end-to-end test (INSERT omitting defaulted columns, then scanning to assert the defaults land and explicitly provided values win) plus unit tests for the literal conversion. The end-to-end test fails without the get_column_default implementation. cargo test -p iceberg-datafusion --lib (90 tests) and cargo clippy -p iceberg-datafusion --lib --tests pass locally.

This pull request and its description were written by Claude Fable 5.

[moomindani]

Gentle ping — open for ~2.5 weeks, CI green, no review yet.

Per the spec, writers must use a column's write-default for columns they don't supply. DataFusion's insert planner consults TableProvider::get_column_default for columns omitted from an INSERT and falls back to NULL when it's not implemented — which IcebergTableProvider doesn't, so omitted columns silently get NULL instead of the declared write-default. Two files.

@CTTY you've reviewed most of the recent datafusion integration work — would you mind taking a look? Closes #2803, part of #2411.

[xanderbailey]

literal_to_scalar_value re-derives the (PrimitiveType, PrimitiveLiteral) → arrow-type mapping that already exists in iceberg-core in a couple of places: get_arrow_datum (crates/iceberg/src/arrow/schema.rs) and create_primitive_array_single_element (crates/iceberg/src/arrow/value.rs). This is effectively a fourth copy of that knowledge, and it can drift e.g. this match already handles Time, which get_arrow_datum currently doesn't.

One option that reuses the tested mappings and lets DataFusion do the array→scalar step:

let arrow_type = type_to_arrow_type(field_type)?;              // already pub
let array = create_primitive_array_single_element(&arrow_type, &Some(lit))?;
ScalarValue::try_from_array(&array, 0).ok()

Bonus: the resulting scalar's arrow type already matches the column (LargeBinary, FixedSizeBinary(len), UTC tz, …), which removes the reliance on the downstream cast for reconciliation.

Tradeoff: create_primitive_array_single_element is pub(crate), so this needs promoting it to pub in iceberg-core. If you'd rather not expand the core crate's public surface, keeping this match is reasonable, it's self-contained. Flagging mainly so the duplication is a conscious choice. @CTTY or @blackmwk might have stronger opinions.

[moomindani]

Thanks for flagging this. I looked into the reuse path before deciding, and the composition would regress today.

type_to_arrow_type maps Time → Time64(Microsecond), Uuid → FixedSizeBinary(16), Fixed(len) → FixedSizeBinary(len) and Binary → LargeBinary. But create_primitive_array_single_element (crates/iceberg/src/arrow/value.rs:627) only has arms for Boolean, Int32, Date32, Int64, Timestamp(us|ns, tz), Float32, Float64, Utf8, Binary, Decimal128 and Struct(None), and falls through to Err("Unsupported constant type combination"). There is no Time64 arm, no FixedSizeBinary arm and no LargeBinary arm. Since column_defaults_from_schema uses filter_map, those four types would silently lose their write-default and fall back to NULL.

So the drift runs in both directions: this match handles Time precisely because core does not, and the same holds for Uuid, Fixed and Binary.

I kept the local match and made the duplication explicit in the doc comment. What I did take from your comment is the arrow-type alignment — see the two replies below: Binary now produces LargeBinary, and Fixed(len) is sized from the declared width, so the scalars carry exactly the types type_to_arrow_type assigns and the downstream cast has nothing left to reconcile. A new unit test asserts scalar.data_type() == type_to_arrow_type(field_type) for all 16 supported combinations, so a future divergence fails the build.

Happy to do the core side as a follow-up if you would like the single source of truth: add the Time64 / FixedSizeBinary / LargeBinary arms to create_primitive_array_single_element, promote it to pub, then reduce this function to your three lines. That expands iceberg-core's public surface, so it seemed better as its own PR than folded in here — but I am happy either way, and would defer to @CTTY / @blackmwk on whether core should export it.

[xanderbailey]

Minor: this sizes the FixedSizeBinary from the default value's length (v.len()) rather than the column's declared Fixed(len). It works today because the planner casts to the target type, but a value whose length didn't match the declared width would produce a differently-typed scalar. Reusing the shared mapping (see comment above) would make this moot, otherwise a one-line note might be worth it.

[moomindani]

Fixed. It now sizes the scalar from the declared Fixed(len) and skips the default when the value's length contradicts the declaration:

let width = i32::try_from(*len).ok()?;
if v.len() != usize::try_from(*len).ok()? {
    return None;
}
ScalarValue::FixedSizeBinary(width, Some(v.clone()))

Covered by test_literal_to_scalar_value_skips_fixed_default_of_wrong_width.

[xanderbailey]

Minor: Binary maps to ScalarValue::Binary here, whereas iceberg-core's type_to_arrow_type maps Binary → LargeBinary (crates/iceberg/src/arrow/schema.rs). The downstream cast reconciles it, so not a correctness issue but this is another spot where reusing the shared mapping would keep things aligned.

[moomindani]

Fixed — it emits ScalarValue::LargeBinary now, matching type_to_arrow_type. The new test_literal_to_scalar_value_matches_column_arrow_type asserts the scalar's arrow type equals type_to_arrow_type(field_type) for every supported primitive, so this cannot drift back unnoticed.

[moomindani]

Rebased onto current main; this had fallen 205 commits behind and was conflicting. Mergeable again, and cargo test -p iceberg-datafusion passes (93 + 9 + 1), with clippy, cargo fmt and cargo public-api all clean. I squashed the six commits into one before rebasing, so your review comments will show as outdated — the code they refer to is unchanged apart from the rebase.

@xanderbailey thanks again for the review; all three points are addressed and answered above. If the resolutions look right to you, saying so would help this find a committer.

@CTTY @kevinjqliu this one has a completed review round. An INSERT that omits a column currently leaves it NULL even when the Iceberg schema declares a write-default; the table provider now surfaces the default so the planner fills it in.

[CTTY]

Sorry I still haven't got a chance to review this. We are planning to migrate datafusion integrations to https://github.com/apache/datafusion-iceberg very soon and don't want to add more code to the current integration. You are welcome to raise the PR there once the repo is up

[moomindani]

Sounds good — thanks for the heads-up, no objection to the close.

I'll re-raise this in apache/datafusion-iceberg once that repo is ready to take contributions. In the meantime I'll leave #2803 open here as the record of the gap — let me know if you'd rather it be tracked in the new repo instead.