Skip to content

Sync main with apache/iceberg-rust main (c9d52179) - #35

Open
alexanderbianchi wants to merge 891 commits into
mainfrom
alex.bianchi/sync-apache-main-58af4edf
Open

alexanderbianchi wants to merge 891 commits into
mainfrom
alex.bianchi/sync-apache-main-58af4edf

Conversation

@alexanderbianchi

@alexanderbianchi alexanderbianchi commented Nov 16, 2026 •

Copy link
Copy Markdown

Fast-forwards the fork's main to apache/iceberg-rust@c9d52179bb (upstream main on 2026-11-16). The fork's main has no commits of its own: it is dc349284, an older upstream commit, so this adds the 891 upstream commits since then and nothing else.

It includes, among others:

Please merge with a merge commit, not squash, so the upstream SHAs stay reachable from main and later syncs remain fast-forwards.

Note: upstream main is on Arrow 60 and reqwest 0.13.

JosephLenton and others added 30 commits August 26, 2026 17:37
* docs: fix references in public docs

* chore: make private items that are publicly referenced public

* feat: make TransactionAction public and correct more issues with docs

* docs: change Schema method docs to be worded more naturally

* docs: fix incorrect links when rendering docs

* feat: make actions public again

* chore: remove default from transactions

* Remove default implementation on actions, remove public visibility of TransactionAction

---------

Co-authored-by: Daniel Carl Jones <djonesoa@amazon.com>
…leteFile (apache#2868)

* feat(scan): carry deletion-vector coordinates on FileScanTaskDeleteFile

Add referenced_data_file, content_offset, and content_size_in_bytes to
FileScanTaskDeleteFile, populated from the delete file's manifest entry.
These locate a deletion-vector blob and scope it to its data file, which
the delete loader needs to read and apply V3 deletion vectors.

Refs apache#2792.

* fix after merging in main

* fix public-api.txt

* fix test name

* update

* Address PR feedback
…r-row clones (apache#3080)

* perf(arrow): build repeated constant string/binary columns without per-row clones

create_primitive_array_repeated built a throwaway `vec![value.clone(); num_rows]`
for the Utf8/Binary/LargeBinary/FixedSizeBinary arms before handing it to the array
constructor. For strings that clones the value into num_rows separate heap
allocations per batch; the binary arms allocate a throwaway intermediate Vec.

Stream the single value straight into the Arrow buffer via `from_iter_values` with
`std::iter::repeat_n` instead.

Benchmarks (release, throwaway harness, old vs new toggled on this file only):
- leaf create_primitive_array_repeated (Utf8, 8192-row batch): 23.0 -> 4.1 ns/row (5.6x)
- full scan select(x, _partition) over a real manifest + 262k-row parquet file,
  string-partitioned: 20.9 -> 5.7 ns/row (3.7x)

Note that the gain is allocation-counts and scales with how much of the
projection is the string partition column.

Fixes apache#3079

* Feedback
…ield (apache#2826)

* chore(repo): Update Apache Iceberg Rust version options

Updated the version options for Apache Iceberg Rust in the bug report template.

* Replace optional version list with required freeform text
* fix(sql): propagate catalog commit errors

* test(sql): clarify transaction commit coverage

* test(sql): exercise commit errors through catalog API

* test(sql): keep commit regression catalog-focused
* Refs apache#2879

Read-side synthesis of the v3 _row_id metadata column, similar to the _last_updated_sequence_number materialization. We do a per-row coalesce of the physically-stored column against a per-row positional fallback of first_row_id plus the row position, gated on first_row_id (null first_row_id nulls the whole column). The positional fallback consumes the RowNumber virtual column, which is now added whenever _row_id is synthesized.

Added tests

* Fix CI failure

* Fix typo flagged by lint

* Synthesize _row_id in a reader-level stream step

Reject a name-only physical _row_id column unconditionally (it never
threads), fixing a false reject of user columns named _row_id under
positional fallback. Add tests for global position across row groups,
survival through positional deletes, and name collision under fallback.

* Fix pr feedback

* Fix merge collision
Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.12.28 to 0.13.4.
- [Release notes](https://github.com/seanmonstar/reqwest/releases)
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md)
- [Commits](seanmonstar/reqwest@v0.12.28...v0.13.4)

---
updated-dependencies:
- dependency-name: reqwest
  dependency-version: 0.13.4
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…e#3115)

Bumps [taiki-e/install-action](https://github.com/taiki-e/install-action) from 2.86.1 to 2.86.6.
- [Release notes](https://github.com/taiki-e/install-action/releases)
- [Changelog](https://github.com/taiki-e/install-action/blob/main/CHANGELOG.md)
- [Commits](taiki-e/install-action@288e746...6cd1350)

---
updated-dependencies:
- dependency-name: taiki-e/install-action
  dependency-version: 2.86.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps the codeql-action group with 2 updates: [github/codeql-action/init](https://github.com/github/codeql-action) and [github/codeql-action/analyze](https://github.com/github/codeql-action).


Updates `github/codeql-action/init` from 4.37.7 to 4.37.8
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](github/codeql-action@ff2f1c6...db488dd)

Updates `github/codeql-action/analyze` from 4.37.7 to 4.37.8
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](github/codeql-action@ff2f1c6...db488dd)

---
updated-dependencies:
- dependency-name: github/codeql-action/init
  dependency-version: 4.37.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: codeql-action
- dependency-name: github/codeql-action/analyze
  dependency-version: 4.37.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: codeql-action
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [uuid](https://github.com/uuid-rs/uuid) from 1.24.0 to 1.25.0.
- [Release notes](https://github.com/uuid-rs/uuid/releases)
- [Commits](uuid-rs/uuid@v1.24.0...1.25.0)

---
updated-dependencies:
- dependency-name: uuid
  dependency-version: 1.25.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…ngs/python (apache#3112)

chore(deps-dev): Bump huggingface-hub in /bindings/python

Bumps [huggingface-hub](https://github.com/huggingface/huggingface_hub) from 1.27.0 to 1.28.0.
- [Release notes](https://github.com/huggingface/huggingface_hub/releases)
- [Commits](huggingface/huggingface_hub@v1.27.0...v1.28.0)

---
updated-dependencies:
- dependency-name: huggingface-hub
  dependency-version: 1.28.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Prune data columns for metadata-only projections

A projection of only metadata columns (e.g. SELECT _row_id) reads no data
columns for their values, since they are synthesized or constant. Without an
independent row-count source, the projection mask fell back to reading every
data column just to recover the row count. Install the RowNumber virtual
column (derived from row-group metadata, no column I/O) as that source for any
metadata-only projection, so the read is pruned to zero data columns. COUNT(*)
(an empty projection) is left unchanged.

Applies to all metadata-only projections: _row_id, _last_updated_sequence_number,
_file, _spec_id, and _partition.

Measured on a 58 MB, 200k-row, 30-column Parquet file, projecting only
_last_updated_sequence_number:

  this change (pruned):        524 KB read,    6 ms
  before (read all columns):  58.6 MB read, 1289 ms

* Incorporate PR feedback
…e#3101)

* refactor(memory): derive catalog configuration from properties

Replace manual catalog property extraction with the shared derive while preserving warehouse validation and forwarding unmodeled properties.

Tracks [apache#3095](apache#3095).

Generated-by: Codex

* refactor(memory): retain typed catalog properties

Remove the redundant config wrapper and catch-all property field, expose the typed properties, and retain the catalog name and typed properties on MemoryCatalog.

Addresses review feedback on [apache#3101](apache#3101).

Generated-by: Codex

* refactor(memory): retain raw catalog properties

* fix(memory): defer missing warehouse validation

* refactor(memory): move raw properties into file io

* fix(memory): restore catalog validation contract

* refactor(memory): retain public catalog properties

* refactor(memory): forward raw catalog properties
* chore: Add LICENSE and NOTICE files for crates missing them in 0.10.0 (apache#2830)

## Which issue does this PR close?

For release: apache#2527

## What changes are included in this PR?

These changes are cherry-picks of apache#2823 and apache#2828. The crates were
missing the LICENSE and NOTICE files required for the distributed source
code.

## Are these changes tested?

Tested using the following script provided by @kevinjqliu. Introducing
this in CI is left as a follow-up
(apache#2827).

```bash
set -Eeuo pipefail

failed=0

while IFS= read -r crate; do
    echo "Checking ${crate}..."
    files="$(cargo package \
    --package "${crate}" \
    --list \
    --locked)"

    for required in LICENSE NOTICE; do
    if ! grep -Fxq "${required}" <<<"${files}"; then
        echo "::error::${crate} package is missing top-level ${required}"
        failed=1
    fi
    done
done < <(
    cargo metadata --no-deps --format-version 1 |
    jq -r '.packages[] | select(.publish != []) | .name'
)

exit "${failed}"
```

---------

Co-authored-by: Andrew Kane <andrew@ankane.org>

* chore: Update changelog for 0.10.0 RC4 (apache#2831)

## Which issue does this PR close?

Follow up for 0.10.0 RC4.

## What changes are included in this PR?

Change log entries for missing changes in 0.10.0 RC4. The changelog is
worded for the original commits to `main`, and not as the squashed
cherry pick in apache#2830.

## Are these changes tested?

N/A

* chore: Update crossbeam-epoch to 0.9.20, pin CI ruff version to 0.15.22 (apache#2911)

## Which issue does this PR close?

Small changes needed for 0.10.1.

## What changes are included in this PR?

First, address advisory in crossbeam-epoch with patch version update.

Second, to address CI, pin Ruff linter for Python to version available
when iceberg-rust 0.10.0 was released.

## Are these changes tested?

Already covered by CI.

* chore: Bump version to 0.10.1, update changelog (apache#2909)

## Which issue does this PR close?

This bumps the crate and pyiceberg-core version to 0.10.1, for the
0.10.1 release.

Additionally, it pins ruff to 0.15.22 which was the version used for
iceberg-rust 0.10.0.

## What changes are included in this PR?

Version bump, updated dependency TSV manifests, updated changelog.

Pins CI version of ruff Python linter.

## Are these changes tested?

N/A

---------

Co-authored-by: Andrew Kane <andrew@ankane.org>
…ICE (apache#3084)

* chore(ci): Fail CI when a publishable crate is missing LICENSE or NOTICE

Missing LICENSE and NOTICE files for newly introduced crates (apache#2823) were
only discovered manually during release preparation, because nothing
asserted their presence.

Add dev/check_license_notice.sh, which lists the packaged files of every
publishable crate with `cargo package --list` and fails when a top-level
LICENSE or NOTICE is absent. Checking the packaged file list rather than
the crate directory also catches files present on disk but excluded from
the published crate. Because `cargo package --list` reports an entry even
when a symlinked target is gone, the file contents are checked separately
so a removed or emptied root copy fails too. Each crate's copy must also
resolve to the repository root file itself, so the license text cannot
drift from one crate to the next.

Run it from the CI lint job so pull requests fail when a crate is missing
either file, and stop excluding the root LICENSE and NOTICE from the
pull_request path filter, since every crate symlinks to those copies and a
PR removing them must still run the job. Also wire it into `make check` so
it can be run locally the same way as the other lint checks.

The check does not pass at this commit. It reports iceberg-property-macro,
which is published but ships neither file, and iceberg-catalog-s3tables,
whose LICENSE and NOTICE are real copies rather than the repository root
files. Both gaps are fixed in the two commits that follow.

Closes apache#2827

* chore: Add missing LICENSE and NOTICE to iceberg-property-macro

iceberg-property-macro is published to crates.io but shipped neither a
LICENSE nor a NOTICE, so the crate was distributed without the files an
Apache release requires. It was missed because the crate was added after
apache#1601, which introduced the files for the crates publishable at the time.

Add the same relative symlinks to the repository root copies that every
other publishable crate uses, which is what the new check in the preceding
commit reports.

* chore: Symlink s3tables LICENSE and NOTICE to the repository root

Every publishable crate references the root LICENSE and NOTICE through a
relative symlink. That convention was established in apache#1601, which added the
files to the crates that were publishable at the time. `iceberg-catalog-s3tables`
was `publish = false` back then so it was skipped, and apache#1916 later added real
copies when the crate was made publishable. Every crate added since has followed
the symlink convention, leaving s3tables as the only outlier.

Replace the copies with symlinks so the root files remain the single source of
truth and per-crate copies cannot drift from them. This is also required for the
check added earlier in this branch to pass, which rejects a per-crate copy that
is not the root file itself: the s3tables copies are byte-identical to the root
files today, but nothing kept them that way.

`cargo package` dereferences the symlinks, so the published crate is unchanged:
LICENSE and NOTICE are still regular files with byte-identical content.

* Update based on PR feedback
* refactor(scan): extract scan planning utilities

* comments

* comments

* remove allocation
* refactor(scan): Validate FileScanTask construction

Make FileScanTask fields private, expose read-only getters, and validate builder output so non-empty partition data always has an associated partition spec.

Generated-by: Codex (GPT-5)

* refactor(scan): Simplify FileScanTask validation

Use typed-builder output conversion for validation instead of coupling the custom build method to generated typestate arguments.

Generated-by: Codex (GPT-5)

* fix(scan): Validate FileScanTask partition data

Validate partition arity and transformed value types while preserving support for historical partition specs whose source columns were dropped. Port the partition coverage from PR apache#3091 to builder regression tests.

Generated-by: Codex (GPT-5)

* fix(scan): Refine partition validation

Enforce symmetric partition/spec presence and tuple arity while avoiding per-value transform validation during task construction. Update the spec-id scan fixture to provide its partition tuple.

Generated-by: Codex (GPT-5)

* fix(scan): Validate partition specs against schema

Preserve historical specs with dropped source columns while validating resolvable partition specs against the task schema. Add regression coverage for an incompatible evolved schema.

Generated-by: Codex (GPT-5)

* fix(scan): Always validate partition specs

Remove the dropped-source shortcut and rely directly on PartitionSpec::partition_type for schema validation. Update dropped-source coverage to assert the resulting planning error.

Generated-by: Codex (GPT-5)
…ache#3022)

* fix(partition): Align DateTime Transform to Java's Implementation

* Add Doc Tests and Update Match Branching

* Adding Spark Equivalence Tests

* Update comments and return early

* Removes requested integration tests

---------

Co-authored-by: Kurtis Wright <kurwrigh@amazon.com>
* feat(scan): carry deletion-vector coordinates on FileScanTaskDeleteFile

Add referenced_data_file, content_offset, and content_size_in_bytes to
FileScanTaskDeleteFile, populated from the delete file's manifest entry.
These locate a deletion-vector blob and scope it to its data file, which
the delete loader needs to read and apply V3 deletion vectors.

Refs apache#2792.

* fix after merging in main

* fix public-api.txt

* fix test name

* update

* apply V3 deletion vectors during scan

* identify deletion vectors by content type and file format

* Fix after merging upstream/main

* Update comment wording.
* bump df to 55.0 (head on main) and arrow to 59.1

* upmerge main after opendal bump

* fix some CI failures

* python version bumps

* bump to branch-55 and rc2 commit. bump to arrow and parquet 59.2

* bump to branch-55 and rc2 commit. bump to arrow and parquet 59.2

* bump to final crates, fix API changes

* update public API

* gate test
…che#3143)

* refactor(python): remove DataFusion integration

Remove the DataFusion table provider export, its tests, and its Rust and Python dependency trees from pyiceberg-core.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* chore(python): leave dependency inventory unchanged

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* docs(python): clarify PyArrow test dependency

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* docs(python): explain PyIceberg core conflict

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…action (apache#3121)

Unify the reserved/metadata column handling behind a single set of transform
actions, per review feedback on apache#3058.

Behavior is unchanged.
JeonDaehong and others added 29 commits November 6, 2026 07:45
…#3435)

* fix(arrow): release a positional delete file whose load fails

try_start_pos_del_load marks a positional delete file Loading, and only
the success path marked it Loaded. If opening or parsing the file
failed, or the load was dropped because another delete file failed,
the entry stayed Loading: a retry on the same loader, a clone of the
same ArrowReader, or a concurrent load of the same file waited for a
notification that never came.

The Load action now carries a PosDelLoadGuard. finish() marks the file
loaded; dropping the guard without finishing removes the Loading marker
and wakes the waiters. A woken waiter checks the state again and loads
the file itself if it isn't loaded, instead of assuming it is.

Closes apache#3434.

* refactor(arrow): assert the Loading invariant when abandoning a load

Only try_start_pos_del_load inserts into positional_deletes, and only the
guard that it returns changes the entry afterwards, so the entry is always
the guard's Loading marker when it is abandoned.

* test(arrow): cover waiters that load after an abandoned positional load

Hold the Load guard, poll load_file_for_task for the same file until it
waits, then drop the guard: the waiter loads the file itself, or returns
the open error when the file is gone. Also load a positional delete file
that opens but fails to parse, then rewrite it and load it again on the
same loader, which hangs without the guard.

* fix(arrow): return a failed positional delete load's error to its waiters

Waiters on a failed load used to retry it one after another, so a missing
file shared by many tasks was opened once per task. The load now records
its error and the tasks waiting on that attempt return it. A load dropped
without an error, when a sibling's error stops load_deletes, still lets
its waiters retry, and later calls start a new attempt either way.

Abandoning a load also tolerates a poisoned lock, since it runs from Drop.

* test(arrow): keep the sibling's load in flight when another delete file fails

On the current-thread test runtime the local filesystem never yields, so the
valid file was only loaded on the second call and its guard was never dropped
mid-load. A test storage now yields before reads and holds one file until the
test opens a gate.

* fix(arrow): pass the whole load error to waiters
)

* feat(arrow): read manifest lists into Arrow with arrow-avro

* refactor(spec): define manifest field IDs next to their fields.

* fix(arrow): reject unknown manifest list format versions
* docs: ask agents to consider public API maintenance

* docs: lead public API guidance with justification
* add bypass capability for PR limits and document, add dannycjones and xanderbailey

* Update CONTRIBUTING.md

Co-authored-by: Kevin Liu <kevinjqliu@users.noreply.github.com>

* Update .asf.yaml

Co-authored-by: Kevin Liu <kevinjqliu@users.noreply.github.com>

---------

Co-authored-by: Kevin Liu <kevinjqliu@users.noreply.github.com>
…ache#3582)

* feat(catalog): support write.metadata.delete-after-commit.enabled

Add the `write.metadata.delete-after-commit.enabled` table property
(default false). When it is set, catalogs that write metadata files
themselves delete the previous metadata files that fall out of the
metadata log (bounded by `write.metadata.previous-versions-max`) after
a successful commit, matching Java's
`CatalogUtil.deleteRemovedMetadataFiles`.

The shared `delete_removed_metadata_files` helper diffs the base and
committed metadata logs and deletes best-effort, logging failures
without failing the commit. It is wired into the memory, SQL and Glue
catalogs. REST and S3 Tables are left out because the server or
service owns metadata file cleanup, and HMS does not support commits
yet.

Co-authored-by: Isaac <no-reply@databricks.com>

* test(catalog): cover enabling delete-after-commit in the same commit

Address review: test that cleanup reads the property from the new
metadata, and say the property must be true in the helper's doc.

Co-authored-by: Isaac <no-reply@databricks.com>

---------

Co-authored-by: Isaac <no-reply@databricks.com>
…and s3.retry.* (apache#3605)

* feat(storage-opendal): read the s3.retry.* properties

The S3 operator was always wrapped in `RetryLayer::new()`: 3 retries, a 1 s
first delay, no jitter. A throttled bucket (S3 `SlowDown`) answers a burst of
requests faster than that policy gives up, and the error reaches the caller
after about 7 s.

Define Iceberg Java's `s3.retry.num-retries`, `s3.retry.min-wait-ms` and
`s3.retry.max-wait-ms` in `iceberg::io` so every S3 backend can read the same
keys, and apply them to the opendal S3 operator. Setting any of them also turns
on jitter; with none set the layer stays `RetryLayer::new()`.

Generated-by: Claude Code (Sonnet 5.5)

* feat(storage-opendal): add backend-independent opendal.retry.* properties

`opendal.retry.num-retries`, `opendal.retry.min-wait-ms` and `opendal.retry.max-wait-ms`
configure the `RetryLayer` of every backend, in the place where the `TimeoutLayer` is
applied. For S3 each `s3.retry.*` key overrides the matching `opendal.retry.*` key,
resolved per key; unset keeps OpenDAL's default. Any set key turns jitter on.

Generated-by: Claude Code (Sonnet 5.5)

* refactor(storage-opendal): keep the reviewed retry tests, add only the new cases

Restore the four tests of a767ae1 as they were (their S3-only semantics still hold) and
add only the generic-keys and per-key precedence cases; `s3_retry_layer` stays as a thin
wrapper of `retry_layer(true)`.

Generated-by: Claude Code (Sonnet 5.5)

* refactor(storage-opendal): resolve s3.retry.* overrides when building the S3 storage

Parse the s3.retry.* properties next to the other s3.* properties and
overlay them, per key, onto the opendal.retry.* settings of the S3
variant's client config. OpenDalClientConfig no longer knows about S3, so
retry_layer takes no backend flag and other backends neither parse nor
serialize s3.retry.*.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FhHg5U2mpkFVSMgVtmhThk

---------

Co-authored-by: Claude <noreply@anthropic.com>
Document the dev-only SQLite usage and remove the exception once sqlx supports a patched libsqlite3-sys release.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…3653)

* build(python): build pyiceberg-core wheel with opt-level = 3

* update docs
…pache#3643)

* fix(spec): write manifests and manifest lists with the deflate codec

They were written without an avro.codec entry (uncompressed). Readers that assume a default for a
missing entry refuse them: PyIceberg 0.12 reads it as "gzip" and raises 'Unsupported codec: gzip',
so a table with one snapshot from iceberg-rust cannot be scanned from PyIceberg. Deflate is what
other writers use for manifests, and it makes them smaller.

* fix(avro): centralize deflate writer defaults

Use a shared internal factory for Avro container writers and verify codec metadata in existing round-trip tests.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Kevin Liu <kevin.jq.liu@gmail.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Bump to Arrow 60

* fix(arrow): borrow page indexes and cover missing offsets

---------

Co-authored-by: Kevin Liu <kevinjqliu@users.noreply.github.com>
…3668)

The REST catalog spec recommends a standard User-Agent format whose
library token for Rust is iceberg-rust (apache/iceberg#17727), so send
iceberg-rust/<version> instead of iceberg-rs/<version>. A header.user-agent
property still overrides the default.
* fix(encryption): reject encryption.key-id on pre-v3 tables

EncryptionManager::from_table_metadata returned Ok(None) for v1/v2 tables
before reading encryption.key-id, so commits to such tables silently wrote
plaintext manifest lists and manifests. Return DataInvalid instead, matching
iceberg-python.

* fix(encryption): reject pre-v3 encryption properties on metadata updates

Reject encryption.key-id and encryption.data-key-length when creating a
table or setting properties below v3, before anything is persisted.
Pre-v3 tables that already carry them still load, so they can be
repaired by removing the properties or upgrading to v3, and snapshot
production on them fails before any manifest is written.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(encryption): reject pre-v3 encryption properties only on table creation

Match Java, which runs EncryptionUtil.checkCompatibility only when
creating new table metadata: check in TableMetadataBuilder::new and
leave property updates, loading, and snapshot writes unchanged.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* refactor(spec): inline pre-v3 encryption property check

Only TableMetadataBuilder::new uses it, so check inline like the
reserved-property check instead of adding a TableProperties method.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* refactor(spec): move pre-v3 encryption check into a helper

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* test(spec): cover pre-v3 encryption check at the builder level

Test v1, v2, and v3 with each encryption property and both together in
TableMetadataBuilder, where the check lives, instead of through the
memory catalog.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(spec): reject pre-v3 encryption properties in set_properties

Check in TableMetadataBuilder::set_properties so both table creation and
property updates reject encryption.key-id and
encryption.data-key-length below v3, as iceberg-python does. Only the
properties being set are checked, so existing pre-v3 tables that already
carry them can still be loaded and updated.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* refactor(spec): rename property format-version check helper

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(spec): check property format compatibility on every metadata build

Run check_properties_format_compatibility in TableMetadataBuilder::build
against the final properties and format version, so every metadata
mutation is checked rather than only set_properties. A pre-v3 table that
already has encryption properties must remove them or upgrade to v3 in
its next commit; loading it is unaffected.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* refactor(spec): shorten format compatibility helper name

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(spec): reject encryption keys on pre-v3 table metadata

v1 and v2 metadata do not serialize encryption-keys, so a key added
below v3 was silently dropped on write. Check it in
check_format_compatibility so every metadata build rejects it.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* test(spec): consolidate pre-v3 encryption tests

Merge the property and encryption-key tests into one table-driven test,
and keep only the unrelated-update and removal cases for pre-v3 tables
that already have encryption.key-id.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* test(spec): tidy pre-v3 encryption tests

---------

Co-authored-by: Kevin Liu <kevin.jq.liu@gmail.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
fix(arrow): rewrite FixedSizeList file hints to List

parquet-rs picks its list reader from the embedded Arrow schema. Iceberg
has no fixed-size list type, so coerce FixedSizeList hints to List before
name mapping and fallback IDs. The 3-level LIST reader then handles null
rows without PromotePlan conversion or pad filtering.
…ing (apache#3438)

* fix(arrow): keep struct presence when its requested children are missing

When none of a struct's requested children are in a Parquet file and
another column is projected, the projection mask selected no leaf under
the struct, so it read as NULL on every row. Read one leaf under such a
struct for its definition level, like Java's PresenceColumnSelector:
inner structs first, then the leaf with the lowest repetition level,
then the first. A leaf under a map reads the whole map, and a candidate
must pass the projection's type check.

Closes apache#3378

* fix(arrow): count list views in the presence walk, group projected field ids

* fix(arrow): record no presence leaf under a list view
* feat(arrow): read unshredded variant columns

Rebased onto main.

* fix(arrow): address review on unshredded variant read

- RecordBatchTransformer matches a variant's metadata/value by name, casts
  binary storage to the table type and sets arrow.parquet.variant on output
- projection validates projected variants and tags only projected ones
- files without field ids or name mapping reject variant projection
- shredded variants fail with FeatureUnsupported linking apache#3606
- tests assert the extension end to end, cover a VARIANT-annotated file,
  dictionary and LargeBinary storage, name mapping and every malformed shape

* fix(arrow): share variant storage check and cover null reads

- projection and RecordBatchTransformer validate variant storage with one
  check, so both report shredded storage as FeatureUnsupported and
  malformed storage as DataInvalid naming the column and field id
- an optional variant metadata field is DataInvalid
- tests read null variant rows and a file written before the variant
  column was added

* test(arrow): read a nested variant under a null struct row
…ds columns (apache#3665)

* feat(transaction): update the name mapping when UpdateSchemaAction adds columns

When a table has schema.name-mapping.default, UpdateSchemaAction now adds
entries for added columns to it in the same commit, as Java's
MappingUtil.update does: new entries go under the parent's entry, an
existing entry in the same struct loses a name an added column takes, and
deleted columns keep their entries. A mapping that fails to parse is
logged and left unchanged.

Co-authored-by: Isaac <no-reply@databricks.com>

* fix(transaction): skip name mapping entries for additions under deleted ancestors

An addition whose parent is deleted in the same update is dropped from the
new schema, and its ID can be assigned again later. Giving it a name
mapping entry would let id-less files resolve that name to an unrelated
column. Also look up added names through a map instead of scanning the
additions for every alias.

Co-authored-by: Isaac <no-reply@databricks.com>

---------

Co-authored-by: Isaac <no-reply@databricks.com>
* fix: support fast append across partition specs

* fix: address multi-spec append review feedback
…gBuilderFactory (apache#3407)

The factories in the registry return a Box<dyn BoxedCatalogBuilder>, not a
CatalogBuilder, so name the alias after the trait it produces. Suggested in
review of apache#3286.
…he#3424)

* feat(scan): optionally retain column statistics on FileScanTask

Add TableScanBuilder::with_column_stats, mirroring Java's
Scan.includeColumnStats. The named columns resolve to field ids at build
time with the scan's case sensitivity, and each planned FileScanTask
carries a FileScanTaskColumnStats holding the data file's value, null and
NaN counts and lower and upper bounds for only those field ids.

Retention is opt-in: by default column_stats is None, nothing is copied
from the manifest entry, and the field is omitted when a task is
serialized, so serialized tasks are unchanged.

* fix(scan): serialize column bounds as spec binary values

Bounds went through Datum's JSON encoding, which writes non-finite floats
as null and rejects many finite float values when reading them back, so a
task with such a bound serialized but failed to deserialize. Serialize
each bound as its type and the spec's single-value binary encoding, as the
DataFile JSON serde already does, so every bound round-trips exactly.

* refactor(scan): address review of column statistics retention

Retain column_sizes for the requested ids too, matching Java's
BaseFile(copy, requestedColumnIds). Share the column-name lookup and its
error with column selection, keep the requested names as an Option like
column_names so an empty list yields empty statistics, and pass the ids as
Arc<[i32]>. Skip work for an empty map when filtering, trim the struct
docs, and cover nested field names.

* fix(scan): reject column names that cannot have statistics

Struct, list and map fields have no statistics of their own, and metadata
columns are not stored in data files, so requesting stats for them
silently returned empty maps or failed with a misleading "not found"
error. Fail build() with an error that says why. Also cover bounds that
keep the manifest's type after a type promotion.

* fix(scan): promote column stats bounds to the scan schema type

A manifest written before a type promotion keeps the narrower type, so one
scan could return an int bound for one file and a long bound for another,
and Datum cannot order values of different types. Promote bounds to the
column's type in the scan schema (int to long, float to double, decimal to
a wider precision) and omit a bound that cannot be promoted.

Box the retained statistics so tasks of scans that never request them do
not grow, and add a doctest for with_column_stats.

* refactor(scan): resolve column stats types once and trim tests and docs

Resolve each requested column's type when the scan is built, so planning
no longer looks the column up in the schema for every bound of every
manifest entry. Rebuild the column stats tests on the builder, one
behavior per test, and cover list names. Drop private field docs that
repeated the getter docs, and state the decimal and date-to-timestamp
promotion rules in the struct docs.

* docs(scan): drop the date to timestamp claim from the column stats docs

A date bound under a timestamp column is stored in 4 bytes, which
Datum::try_from_bytes cannot decode as a timestamp, so the manifest read
fails before planning sees the bound. The docs described a case that
planning never handles.
build(deps): update quinn-udp for RUSTSEC-2026-0538
* fix(arrow): skip data files with a known zero record count

* test(arrow): simplify empty-file reader regressions
…#3514)

* feat(scan): expose portable reader options for planned tasks

* refactor(arrow): separate reader options from scan planning

* feat(arrow): include Parquet tuning in shared reader options

* refactor(arrow): refine reader options API after review

* refactor(arrow): address reader configuration review feedback

* fix: preserve scan concurrency defaults for Arrow reads

* refactor(arrow): clarify options API and focus reader tests

* test(arrow): migrate upstream scheduling tests to reader options

* refactor(arrow): retain reader options and address review feedback

* fix(arrow): serialize Parquet preload settings

* fix(arrow): expose preload settings on Parquet options builder

* fix(arrow): expose Parquet preload getters

* refactor(arrow): resolve reader concurrency independently of scans

* refactor(arrow): keep reader concurrency solely in options

* Update crates/iceberg/src/arrow/reader/mod.rs

Co-authored-by: blackmwk <257669749+blackmwk@users.noreply.github.com>

* docs(arrow): clarify reader options migration and preload behavior

---------

Co-authored-by: blackmwk <257669749+blackmwk@users.noreply.github.com>
…e previous file (apache#3695)

* fix(spec): validate the metadata log in build() with the entry for the previous file

* test(spec): metadata change after a branch snapshot within the clock-skew tolerance builds
* feat: add TableMetadataBuilder::build_replacement

Backs `REPLACE TABLE`: swaps the schema, partition spec and sort order
while keeping the table's uuid, properties and snapshots, and drops the
main branch so the replaced table has no current snapshot.

Columns the table already has keep their field ids, so data files written
before the replacement stay readable through the shared columns. New
columns draw ids above `last_column_id`, never reusing a dropped column's
id. Partition field ids are likewise reused across historical specs.

Extracts `rebind_to_schema` out of `reassign_ids` so the partition spec
and sort order can be rebound to a schema whose ids were assigned either
from scratch or by `assign_fresh_ids`.

Format version 1 is not supported.

* fix docs

* tighten up tests

* nits

* test: cover build_replacement error paths and refs, document id matching

* refactor: take a typed format_version in build_replacement

* refactor: make build_replacement pub(crate) until RTAS uses it

Moves the doc example into a unit test, since a doctest cannot call a
pub(crate) function.

* fix: return an error when build_replacement runs out of partition ids

Reuses equivalent ids through reuse_partition_field_ids and assigns the
rest with PartitionSpecBuilder, which errors on overflow instead of
wrapping. The AddSpec change still carries every field id.
…rted load (apache#3697)

* fix(arrow): wake equality-delete waiters with the notifier of the started load

Co-authored-by: JeonDaehong <90895144+JeonDaehong@users.noreply.github.com>

* Assert that insert_equality_delete gets the notifier of the started load

---------

Co-authored-by: JeonDaehong <90895144+JeonDaehong@users.noreply.github.com>
…e predicate (apache#3507)

* fix(reader): keep pages whose index bounds cannot be compared with the predicate

Page-index bounds are built with the table field type but the file's
physical literal. When the two differ, e.g. a long field read against
INT32 pages written before an int -> long promotion, a double field
against FLOAT pages, or a UINT_32 column mapped to long, the bounds
never compare with the predicate literal. Inequality and IN predicates
treated that as "cannot match" and skipped pages holding matching rows.

Keep a page whenever its bound cannot be compared with the literal: the
inequality check now treats an incomparable bound as a possible match,
and IN uses all(lt) / all(gt) instead of !any(ge) / !any(le).

* test(reader): cover a mixed UINT_32 page in the page-index evaluator

A page holding both 100 and 3_000_000_000 has a raw signed max below its
min, which pins the unsigned column order that a later bound widening has
to zero-extend rather than sign-extend.

* test(reader): cover <= and > on INT32 pages of a promoted long field

* test(reader): cover all four inequality operators on promoted float and timestamp pages

* test(reader): pass RowGroupPageIndex to the page-index evaluator in the promoted-type tests

Arrow 60 (apache#3257) changed PageIndexEvaluator::eval to take a RowGroupPageIndex
and removed ParquetMetaData::column_index, so the tests added here no longer
built after merging main.
Defer Moka cache construction until the first manifest or manifest-list
read, and skip it entirely when caching is disabled. Share lazy cache
state across clones and retain existing cache policy and encryption.

Adapted from #34 for upstream main, without the
fork-specific FileIO rebinding helper.

Co-authored-by: Geoffrey Claude <geoffrey.claude@datadoghq.com>
@alexanderbianchi alexanderbianchi changed the title Sync main with apache/iceberg-rust main (58af4edf) Sync main with apache/iceberg-rust main (c9d52179) Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.