Repository navigation
Sync main with apache/iceberg-rust main (c9d52179) - #35
Open
alexanderbianchi wants to merge 891 commits into
Open
alexanderbianchi wants to merge 891 commits into
alexanderbianchi wants to merge 891 commits into
Conversation
* docs: fix references in public docs * chore: make private items that are publicly referenced public * feat: make TransactionAction public and correct more issues with docs * docs: change Schema method docs to be worded more naturally * docs: fix incorrect links when rendering docs * feat: make actions public again * chore: remove default from transactions * Remove default implementation on actions, remove public visibility of TransactionAction --------- Co-authored-by: Daniel Carl Jones <djonesoa@amazon.com>
…leteFile (apache#2868) * feat(scan): carry deletion-vector coordinates on FileScanTaskDeleteFile Add referenced_data_file, content_offset, and content_size_in_bytes to FileScanTaskDeleteFile, populated from the delete file's manifest entry. These locate a deletion-vector blob and scope it to its data file, which the delete loader needs to read and apply V3 deletion vectors. Refs apache#2792. * fix after merging in main * fix public-api.txt * fix test name * update * Address PR feedback
…r-row clones (apache#3080) * perf(arrow): build repeated constant string/binary columns without per-row clones create_primitive_array_repeated built a throwaway `vec![value.clone(); num_rows]` for the Utf8/Binary/LargeBinary/FixedSizeBinary arms before handing it to the array constructor. For strings that clones the value into num_rows separate heap allocations per batch; the binary arms allocate a throwaway intermediate Vec. Stream the single value straight into the Arrow buffer via `from_iter_values` with `std::iter::repeat_n` instead. Benchmarks (release, throwaway harness, old vs new toggled on this file only): - leaf create_primitive_array_repeated (Utf8, 8192-row batch): 23.0 -> 4.1 ns/row (5.6x) - full scan select(x, _partition) over a real manifest + 262k-row parquet file, string-partitioned: 20.9 -> 5.7 ns/row (3.7x) Note that the gain is allocation-counts and scales with how much of the projection is the string partition column. Fixes apache#3079 * Feedback
…ield (apache#2826) * chore(repo): Update Apache Iceberg Rust version options Updated the version options for Apache Iceberg Rust in the bug report template. * Replace optional version list with required freeform text
* fix(sql): propagate catalog commit errors * test(sql): clarify transaction commit coverage * test(sql): exercise commit errors through catalog API * test(sql): keep commit regression catalog-focused
* Refs apache#2879 Read-side synthesis of the v3 _row_id metadata column, similar to the _last_updated_sequence_number materialization. We do a per-row coalesce of the physically-stored column against a per-row positional fallback of first_row_id plus the row position, gated on first_row_id (null first_row_id nulls the whole column). The positional fallback consumes the RowNumber virtual column, which is now added whenever _row_id is synthesized. Added tests * Fix CI failure * Fix typo flagged by lint * Synthesize _row_id in a reader-level stream step Reject a name-only physical _row_id column unconditionally (it never threads), fixing a false reject of user columns named _row_id under positional fallback. Add tests for global position across row groups, survival through positional deletes, and name collision under fallback. * Fix pr feedback * Fix merge collision
Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.12.28 to 0.13.4. - [Release notes](https://github.com/seanmonstar/reqwest/releases) - [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md) - [Commits](seanmonstar/reqwest@v0.12.28...v0.13.4) --- updated-dependencies: - dependency-name: reqwest dependency-version: 0.13.4 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…e#3115) Bumps [taiki-e/install-action](https://github.com/taiki-e/install-action) from 2.86.1 to 2.86.6. - [Release notes](https://github.com/taiki-e/install-action/releases) - [Changelog](https://github.com/taiki-e/install-action/blob/main/CHANGELOG.md) - [Commits](taiki-e/install-action@288e746...6cd1350) --- updated-dependencies: - dependency-name: taiki-e/install-action dependency-version: 2.86.6 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps the codeql-action group with 2 updates: [github/codeql-action/init](https://github.com/github/codeql-action) and [github/codeql-action/analyze](https://github.com/github/codeql-action). Updates `github/codeql-action/init` from 4.37.7 to 4.37.8 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@ff2f1c6...db488dd) Updates `github/codeql-action/analyze` from 4.37.7 to 4.37.8 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@ff2f1c6...db488dd) --- updated-dependencies: - dependency-name: github/codeql-action/init dependency-version: 4.37.8 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: codeql-action - dependency-name: github/codeql-action/analyze dependency-version: 4.37.8 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: codeql-action ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [uuid](https://github.com/uuid-rs/uuid) from 1.24.0 to 1.25.0. - [Release notes](https://github.com/uuid-rs/uuid/releases) - [Commits](uuid-rs/uuid@v1.24.0...1.25.0) --- updated-dependencies: - dependency-name: uuid dependency-version: 1.25.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…ngs/python (apache#3112) chore(deps-dev): Bump huggingface-hub in /bindings/python Bumps [huggingface-hub](https://github.com/huggingface/huggingface_hub) from 1.27.0 to 1.28.0. - [Release notes](https://github.com/huggingface/huggingface_hub/releases) - [Commits](huggingface/huggingface_hub@v1.27.0...v1.28.0) --- updated-dependencies: - dependency-name: huggingface-hub dependency-version: 1.28.0 dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Prune data columns for metadata-only projections A projection of only metadata columns (e.g. SELECT _row_id) reads no data columns for their values, since they are synthesized or constant. Without an independent row-count source, the projection mask fell back to reading every data column just to recover the row count. Install the RowNumber virtual column (derived from row-group metadata, no column I/O) as that source for any metadata-only projection, so the read is pruned to zero data columns. COUNT(*) (an empty projection) is left unchanged. Applies to all metadata-only projections: _row_id, _last_updated_sequence_number, _file, _spec_id, and _partition. Measured on a 58 MB, 200k-row, 30-column Parquet file, projecting only _last_updated_sequence_number: this change (pruned): 524 KB read, 6 ms before (read all columns): 58.6 MB read, 1289 ms * Incorporate PR feedback
…e#3101) * refactor(memory): derive catalog configuration from properties Replace manual catalog property extraction with the shared derive while preserving warehouse validation and forwarding unmodeled properties. Tracks [apache#3095](apache#3095). Generated-by: Codex * refactor(memory): retain typed catalog properties Remove the redundant config wrapper and catch-all property field, expose the typed properties, and retain the catalog name and typed properties on MemoryCatalog. Addresses review feedback on [apache#3101](apache#3101). Generated-by: Codex * refactor(memory): retain raw catalog properties * fix(memory): defer missing warehouse validation * refactor(memory): move raw properties into file io * fix(memory): restore catalog validation contract * refactor(memory): retain public catalog properties * refactor(memory): forward raw catalog properties
* chore: Add LICENSE and NOTICE files for crates missing them in 0.10.0 (apache#2830) ## Which issue does this PR close? For release: apache#2527 ## What changes are included in this PR? These changes are cherry-picks of apache#2823 and apache#2828. The crates were missing the LICENSE and NOTICE files required for the distributed source code. ## Are these changes tested? Tested using the following script provided by @kevinjqliu. Introducing this in CI is left as a follow-up (apache#2827). ```bash set -Eeuo pipefail failed=0 while IFS= read -r crate; do echo "Checking ${crate}..." files="$(cargo package \ --package "${crate}" \ --list \ --locked)" for required in LICENSE NOTICE; do if ! grep -Fxq "${required}" <<<"${files}"; then echo "::error::${crate} package is missing top-level ${required}" failed=1 fi done done < <( cargo metadata --no-deps --format-version 1 | jq -r '.packages[] | select(.publish != []) | .name' ) exit "${failed}" ``` --------- Co-authored-by: Andrew Kane <andrew@ankane.org> * chore: Update changelog for 0.10.0 RC4 (apache#2831) ## Which issue does this PR close? Follow up for 0.10.0 RC4. ## What changes are included in this PR? Change log entries for missing changes in 0.10.0 RC4. The changelog is worded for the original commits to `main`, and not as the squashed cherry pick in apache#2830. ## Are these changes tested? N/A * chore: Update crossbeam-epoch to 0.9.20, pin CI ruff version to 0.15.22 (apache#2911) ## Which issue does this PR close? Small changes needed for 0.10.1. ## What changes are included in this PR? First, address advisory in crossbeam-epoch with patch version update. Second, to address CI, pin Ruff linter for Python to version available when iceberg-rust 0.10.0 was released. ## Are these changes tested? Already covered by CI. * chore: Bump version to 0.10.1, update changelog (apache#2909) ## Which issue does this PR close? This bumps the crate and pyiceberg-core version to 0.10.1, for the 0.10.1 release. Additionally, it pins ruff to 0.15.22 which was the version used for iceberg-rust 0.10.0. ## What changes are included in this PR? Version bump, updated dependency TSV manifests, updated changelog. Pins CI version of ruff Python linter. ## Are these changes tested? N/A --------- Co-authored-by: Andrew Kane <andrew@ankane.org>
…ICE (apache#3084) * chore(ci): Fail CI when a publishable crate is missing LICENSE or NOTICE Missing LICENSE and NOTICE files for newly introduced crates (apache#2823) were only discovered manually during release preparation, because nothing asserted their presence. Add dev/check_license_notice.sh, which lists the packaged files of every publishable crate with `cargo package --list` and fails when a top-level LICENSE or NOTICE is absent. Checking the packaged file list rather than the crate directory also catches files present on disk but excluded from the published crate. Because `cargo package --list` reports an entry even when a symlinked target is gone, the file contents are checked separately so a removed or emptied root copy fails too. Each crate's copy must also resolve to the repository root file itself, so the license text cannot drift from one crate to the next. Run it from the CI lint job so pull requests fail when a crate is missing either file, and stop excluding the root LICENSE and NOTICE from the pull_request path filter, since every crate symlinks to those copies and a PR removing them must still run the job. Also wire it into `make check` so it can be run locally the same way as the other lint checks. The check does not pass at this commit. It reports iceberg-property-macro, which is published but ships neither file, and iceberg-catalog-s3tables, whose LICENSE and NOTICE are real copies rather than the repository root files. Both gaps are fixed in the two commits that follow. Closes apache#2827 * chore: Add missing LICENSE and NOTICE to iceberg-property-macro iceberg-property-macro is published to crates.io but shipped neither a LICENSE nor a NOTICE, so the crate was distributed without the files an Apache release requires. It was missed because the crate was added after apache#1601, which introduced the files for the crates publishable at the time. Add the same relative symlinks to the repository root copies that every other publishable crate uses, which is what the new check in the preceding commit reports. * chore: Symlink s3tables LICENSE and NOTICE to the repository root Every publishable crate references the root LICENSE and NOTICE through a relative symlink. That convention was established in apache#1601, which added the files to the crates that were publishable at the time. `iceberg-catalog-s3tables` was `publish = false` back then so it was skipped, and apache#1916 later added real copies when the crate was made publishable. Every crate added since has followed the symlink convention, leaving s3tables as the only outlier. Replace the copies with symlinks so the root files remain the single source of truth and per-crate copies cannot drift from them. This is also required for the check added earlier in this branch to pass, which rejects a per-crate copy that is not the root file itself: the s3tables copies are byte-identical to the root files today, but nothing kept them that way. `cargo package` dereferences the symlinks, so the published crate is unchanged: LICENSE and NOTICE are still regular files with byte-identical content. * Update based on PR feedback
* refactor(scan): extract scan planning utilities * comments * comments * remove allocation
* refactor(scan): Validate FileScanTask construction Make FileScanTask fields private, expose read-only getters, and validate builder output so non-empty partition data always has an associated partition spec. Generated-by: Codex (GPT-5) * refactor(scan): Simplify FileScanTask validation Use typed-builder output conversion for validation instead of coupling the custom build method to generated typestate arguments. Generated-by: Codex (GPT-5) * fix(scan): Validate FileScanTask partition data Validate partition arity and transformed value types while preserving support for historical partition specs whose source columns were dropped. Port the partition coverage from PR apache#3091 to builder regression tests. Generated-by: Codex (GPT-5) * fix(scan): Refine partition validation Enforce symmetric partition/spec presence and tuple arity while avoiding per-value transform validation during task construction. Update the spec-id scan fixture to provide its partition tuple. Generated-by: Codex (GPT-5) * fix(scan): Validate partition specs against schema Preserve historical specs with dropped source columns while validating resolvable partition specs against the task schema. Add regression coverage for an incompatible evolved schema. Generated-by: Codex (GPT-5) * fix(scan): Always validate partition specs Remove the dropped-source shortcut and rely directly on PartitionSpec::partition_type for schema validation. Update dropped-source coverage to assert the resulting planning error. Generated-by: Codex (GPT-5)
…pache#3086) Take an owned AuthManager on catalog construction
…ache#3022) * fix(partition): Align DateTime Transform to Java's Implementation * Add Doc Tests and Update Match Branching * Adding Spark Equivalence Tests * Update comments and return early * Removes requested integration tests --------- Co-authored-by: Kurtis Wright <kurwrigh@amazon.com>
* feat(scan): carry deletion-vector coordinates on FileScanTaskDeleteFile Add referenced_data_file, content_offset, and content_size_in_bytes to FileScanTaskDeleteFile, populated from the delete file's manifest entry. These locate a deletion-vector blob and scope it to its data file, which the delete loader needs to read and apply V3 deletion vectors. Refs apache#2792. * fix after merging in main * fix public-api.txt * fix test name * update * apply V3 deletion vectors during scan * identify deletion vectors by content type and file format * Fix after merging upstream/main * Update comment wording.
* bump df to 55.0 (head on main) and arrow to 59.1 * upmerge main after opendal bump * fix some CI failures * python version bumps * bump to branch-55 and rc2 commit. bump to arrow and parquet 59.2 * bump to branch-55 and rc2 commit. bump to arrow and parquet 59.2 * bump to final crates, fix API changes * update public API * gate test
…che#3143) * refactor(python): remove DataFusion integration Remove the DataFusion table provider export, its tests, and its Rust and Python dependency trees from pyiceberg-core. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * chore(python): leave dependency inventory unchanged Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * docs(python): clarify PyArrow test dependency Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * docs(python): explain PyIceberg core conflict Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…action (apache#3121) Unify the reserved/metadata column handling behind a single set of transform actions, per review feedback on apache#3058. Behavior is unchanged.
…#3435) * fix(arrow): release a positional delete file whose load fails try_start_pos_del_load marks a positional delete file Loading, and only the success path marked it Loaded. If opening or parsing the file failed, or the load was dropped because another delete file failed, the entry stayed Loading: a retry on the same loader, a clone of the same ArrowReader, or a concurrent load of the same file waited for a notification that never came. The Load action now carries a PosDelLoadGuard. finish() marks the file loaded; dropping the guard without finishing removes the Loading marker and wakes the waiters. A woken waiter checks the state again and loads the file itself if it isn't loaded, instead of assuming it is. Closes apache#3434. * refactor(arrow): assert the Loading invariant when abandoning a load Only try_start_pos_del_load inserts into positional_deletes, and only the guard that it returns changes the entry afterwards, so the entry is always the guard's Loading marker when it is abandoned. * test(arrow): cover waiters that load after an abandoned positional load Hold the Load guard, poll load_file_for_task for the same file until it waits, then drop the guard: the waiter loads the file itself, or returns the open error when the file is gone. Also load a positional delete file that opens but fails to parse, then rewrite it and load it again on the same loader, which hangs without the guard. * fix(arrow): return a failed positional delete load's error to its waiters Waiters on a failed load used to retry it one after another, so a missing file shared by many tasks was opened once per task. The load now records its error and the tasks waiting on that attempt return it. A load dropped without an error, when a sibling's error stops load_deletes, still lets its waiters retry, and later calls start a new attempt either way. Abandoning a load also tolerates a poisoned lock, since it runs from Drop. * test(arrow): keep the sibling's load in flight when another delete file fails On the current-thread test runtime the local filesystem never yields, so the valid file was only loaded on the second call and its guard was never dropped mid-load. A test storage now yields before reads and holds one file until the test opens a gate. * fix(arrow): pass the whole load error to waiters
* docs: ask agents to consider public API maintenance * docs: lead public API guidance with justification
* add bypass capability for PR limits and document, add dannycjones and xanderbailey * Update CONTRIBUTING.md Co-authored-by: Kevin Liu <kevinjqliu@users.noreply.github.com> * Update .asf.yaml Co-authored-by: Kevin Liu <kevinjqliu@users.noreply.github.com> --------- Co-authored-by: Kevin Liu <kevinjqliu@users.noreply.github.com>
…ache#3582) * feat(catalog): support write.metadata.delete-after-commit.enabled Add the `write.metadata.delete-after-commit.enabled` table property (default false). When it is set, catalogs that write metadata files themselves delete the previous metadata files that fall out of the metadata log (bounded by `write.metadata.previous-versions-max`) after a successful commit, matching Java's `CatalogUtil.deleteRemovedMetadataFiles`. The shared `delete_removed_metadata_files` helper diffs the base and committed metadata logs and deletes best-effort, logging failures without failing the commit. It is wired into the memory, SQL and Glue catalogs. REST and S3 Tables are left out because the server or service owns metadata file cleanup, and HMS does not support commits yet. Co-authored-by: Isaac <no-reply@databricks.com> * test(catalog): cover enabling delete-after-commit in the same commit Address review: test that cleanup reads the property from the new metadata, and say the property must be true in the helper's doc. Co-authored-by: Isaac <no-reply@databricks.com> --------- Co-authored-by: Isaac <no-reply@databricks.com>
…and s3.retry.* (apache#3605) * feat(storage-opendal): read the s3.retry.* properties The S3 operator was always wrapped in `RetryLayer::new()`: 3 retries, a 1 s first delay, no jitter. A throttled bucket (S3 `SlowDown`) answers a burst of requests faster than that policy gives up, and the error reaches the caller after about 7 s. Define Iceberg Java's `s3.retry.num-retries`, `s3.retry.min-wait-ms` and `s3.retry.max-wait-ms` in `iceberg::io` so every S3 backend can read the same keys, and apply them to the opendal S3 operator. Setting any of them also turns on jitter; with none set the layer stays `RetryLayer::new()`. Generated-by: Claude Code (Sonnet 5.5) * feat(storage-opendal): add backend-independent opendal.retry.* properties `opendal.retry.num-retries`, `opendal.retry.min-wait-ms` and `opendal.retry.max-wait-ms` configure the `RetryLayer` of every backend, in the place where the `TimeoutLayer` is applied. For S3 each `s3.retry.*` key overrides the matching `opendal.retry.*` key, resolved per key; unset keeps OpenDAL's default. Any set key turns jitter on. Generated-by: Claude Code (Sonnet 5.5) * refactor(storage-opendal): keep the reviewed retry tests, add only the new cases Restore the four tests of a767ae1 as they were (their S3-only semantics still hold) and add only the generic-keys and per-key precedence cases; `s3_retry_layer` stays as a thin wrapper of `retry_layer(true)`. Generated-by: Claude Code (Sonnet 5.5) * refactor(storage-opendal): resolve s3.retry.* overrides when building the S3 storage Parse the s3.retry.* properties next to the other s3.* properties and overlay them, per key, onto the opendal.retry.* settings of the S3 variant's client config. OpenDalClientConfig no longer knows about S3, so retry_layer takes no backend flag and other backends neither parse nor serialize s3.retry.*. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FhHg5U2mpkFVSMgVtmhThk --------- Co-authored-by: Claude <noreply@anthropic.com>
Document the dev-only SQLite usage and remove the exception once sqlx supports a patched libsqlite3-sys release. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…3653) * build(python): build pyiceberg-core wheel with opt-level = 3 * update docs
…pache#3643) * fix(spec): write manifests and manifest lists with the deflate codec They were written without an avro.codec entry (uncompressed). Readers that assume a default for a missing entry refuse them: PyIceberg 0.12 reads it as "gzip" and raises 'Unsupported codec: gzip', so a table with one snapshot from iceberg-rust cannot be scanned from PyIceberg. Deflate is what other writers use for manifests, and it makes them smaller. * fix(avro): centralize deflate writer defaults Use a shared internal factory for Avro container writers and verify codec metadata in existing round-trip tests. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Kevin Liu <kevin.jq.liu@gmail.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Bump reqwest to 0.13.5
* Bump to Arrow 60 * fix(arrow): borrow page indexes and cover missing offsets --------- Co-authored-by: Kevin Liu <kevinjqliu@users.noreply.github.com>
…3668) The REST catalog spec recommends a standard User-Agent format whose library token for Rust is iceberg-rust (apache/iceberg#17727), so send iceberg-rust/<version> instead of iceberg-rs/<version>. A header.user-agent property still overrides the default.
* fix(encryption): reject encryption.key-id on pre-v3 tables EncryptionManager::from_table_metadata returned Ok(None) for v1/v2 tables before reading encryption.key-id, so commits to such tables silently wrote plaintext manifest lists and manifests. Return DataInvalid instead, matching iceberg-python. * fix(encryption): reject pre-v3 encryption properties on metadata updates Reject encryption.key-id and encryption.data-key-length when creating a table or setting properties below v3, before anything is persisted. Pre-v3 tables that already carry them still load, so they can be repaired by removing the properties or upgrading to v3, and snapshot production on them fails before any manifest is written. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(encryption): reject pre-v3 encryption properties only on table creation Match Java, which runs EncryptionUtil.checkCompatibility only when creating new table metadata: check in TableMetadataBuilder::new and leave property updates, loading, and snapshot writes unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * refactor(spec): inline pre-v3 encryption property check Only TableMetadataBuilder::new uses it, so check inline like the reserved-property check instead of adding a TableProperties method. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * refactor(spec): move pre-v3 encryption check into a helper Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * test(spec): cover pre-v3 encryption check at the builder level Test v1, v2, and v3 with each encryption property and both together in TableMetadataBuilder, where the check lives, instead of through the memory catalog. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(spec): reject pre-v3 encryption properties in set_properties Check in TableMetadataBuilder::set_properties so both table creation and property updates reject encryption.key-id and encryption.data-key-length below v3, as iceberg-python does. Only the properties being set are checked, so existing pre-v3 tables that already carry them can still be loaded and updated. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * refactor(spec): rename property format-version check helper Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(spec): check property format compatibility on every metadata build Run check_properties_format_compatibility in TableMetadataBuilder::build against the final properties and format version, so every metadata mutation is checked rather than only set_properties. A pre-v3 table that already has encryption properties must remove them or upgrade to v3 in its next commit; loading it is unaffected. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * refactor(spec): shorten format compatibility helper name Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(spec): reject encryption keys on pre-v3 table metadata v1 and v2 metadata do not serialize encryption-keys, so a key added below v3 was silently dropped on write. Check it in check_format_compatibility so every metadata build rejects it. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * test(spec): consolidate pre-v3 encryption tests Merge the property and encryption-key tests into one table-driven test, and keep only the unrelated-update and removal cases for pre-v3 tables that already have encryption.key-id. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * test(spec): tidy pre-v3 encryption tests --------- Co-authored-by: Kevin Liu <kevin.jq.liu@gmail.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
fix(arrow): rewrite FixedSizeList file hints to List parquet-rs picks its list reader from the embedded Arrow schema. Iceberg has no fixed-size list type, so coerce FixedSizeList hints to List before name mapping and fallback IDs. The 3-level LIST reader then handles null rows without PromotePlan conversion or pad filtering.
…ing (apache#3438) * fix(arrow): keep struct presence when its requested children are missing When none of a struct's requested children are in a Parquet file and another column is projected, the projection mask selected no leaf under the struct, so it read as NULL on every row. Read one leaf under such a struct for its definition level, like Java's PresenceColumnSelector: inner structs first, then the leaf with the lowest repetition level, then the first. A leaf under a map reads the whole map, and a candidate must pass the projection's type check. Closes apache#3378 * fix(arrow): count list views in the presence walk, group projected field ids * fix(arrow): record no presence leaf under a list view
* feat(arrow): read unshredded variant columns Rebased onto main. * fix(arrow): address review on unshredded variant read - RecordBatchTransformer matches a variant's metadata/value by name, casts binary storage to the table type and sets arrow.parquet.variant on output - projection validates projected variants and tags only projected ones - files without field ids or name mapping reject variant projection - shredded variants fail with FeatureUnsupported linking apache#3606 - tests assert the extension end to end, cover a VARIANT-annotated file, dictionary and LargeBinary storage, name mapping and every malformed shape * fix(arrow): share variant storage check and cover null reads - projection and RecordBatchTransformer validate variant storage with one check, so both report shredded storage as FeatureUnsupported and malformed storage as DataInvalid naming the column and field id - an optional variant metadata field is DataInvalid - tests read null variant rows and a file written before the variant column was added * test(arrow): read a nested variant under a null struct row
…ds columns (apache#3665) * feat(transaction): update the name mapping when UpdateSchemaAction adds columns When a table has schema.name-mapping.default, UpdateSchemaAction now adds entries for added columns to it in the same commit, as Java's MappingUtil.update does: new entries go under the parent's entry, an existing entry in the same struct loses a name an added column takes, and deleted columns keep their entries. A mapping that fails to parse is logged and left unchanged. Co-authored-by: Isaac <no-reply@databricks.com> * fix(transaction): skip name mapping entries for additions under deleted ancestors An addition whose parent is deleted in the same update is dropped from the new schema, and its ID can be assigned again later. Giving it a name mapping entry would let id-less files resolve that name to an unrelated column. Also look up added names through a map instead of scanning the additions for every alias. Co-authored-by: Isaac <no-reply@databricks.com> --------- Co-authored-by: Isaac <no-reply@databricks.com>
* fix: support fast append across partition specs * fix: address multi-spec append review feedback
…gBuilderFactory (apache#3407) The factories in the registry return a Box<dyn BoxedCatalogBuilder>, not a CatalogBuilder, so name the alias after the trait it produces. Suggested in review of apache#3286.
…he#3424) * feat(scan): optionally retain column statistics on FileScanTask Add TableScanBuilder::with_column_stats, mirroring Java's Scan.includeColumnStats. The named columns resolve to field ids at build time with the scan's case sensitivity, and each planned FileScanTask carries a FileScanTaskColumnStats holding the data file's value, null and NaN counts and lower and upper bounds for only those field ids. Retention is opt-in: by default column_stats is None, nothing is copied from the manifest entry, and the field is omitted when a task is serialized, so serialized tasks are unchanged. * fix(scan): serialize column bounds as spec binary values Bounds went through Datum's JSON encoding, which writes non-finite floats as null and rejects many finite float values when reading them back, so a task with such a bound serialized but failed to deserialize. Serialize each bound as its type and the spec's single-value binary encoding, as the DataFile JSON serde already does, so every bound round-trips exactly. * refactor(scan): address review of column statistics retention Retain column_sizes for the requested ids too, matching Java's BaseFile(copy, requestedColumnIds). Share the column-name lookup and its error with column selection, keep the requested names as an Option like column_names so an empty list yields empty statistics, and pass the ids as Arc<[i32]>. Skip work for an empty map when filtering, trim the struct docs, and cover nested field names. * fix(scan): reject column names that cannot have statistics Struct, list and map fields have no statistics of their own, and metadata columns are not stored in data files, so requesting stats for them silently returned empty maps or failed with a misleading "not found" error. Fail build() with an error that says why. Also cover bounds that keep the manifest's type after a type promotion. * fix(scan): promote column stats bounds to the scan schema type A manifest written before a type promotion keeps the narrower type, so one scan could return an int bound for one file and a long bound for another, and Datum cannot order values of different types. Promote bounds to the column's type in the scan schema (int to long, float to double, decimal to a wider precision) and omit a bound that cannot be promoted. Box the retained statistics so tasks of scans that never request them do not grow, and add a doctest for with_column_stats. * refactor(scan): resolve column stats types once and trim tests and docs Resolve each requested column's type when the scan is built, so planning no longer looks the column up in the schema for every bound of every manifest entry. Rebuild the column stats tests on the builder, one behavior per test, and cover list names. Drop private field docs that repeated the getter docs, and state the decimal and date-to-timestamp promotion rules in the struct docs. * docs(scan): drop the date to timestamp claim from the column stats docs A date bound under a timestamp column is stored in 4 bytes, which Datum::try_from_bytes cannot decode as a timestamp, so the manifest read fails before planning sees the bound. The docs described a case that planning never handles.
build(deps): update quinn-udp for RUSTSEC-2026-0538
* fix(arrow): skip data files with a known zero record count * test(arrow): simplify empty-file reader regressions
…#3514) * feat(scan): expose portable reader options for planned tasks * refactor(arrow): separate reader options from scan planning * feat(arrow): include Parquet tuning in shared reader options * refactor(arrow): refine reader options API after review * refactor(arrow): address reader configuration review feedback * fix: preserve scan concurrency defaults for Arrow reads * refactor(arrow): clarify options API and focus reader tests * test(arrow): migrate upstream scheduling tests to reader options * refactor(arrow): retain reader options and address review feedback * fix(arrow): serialize Parquet preload settings * fix(arrow): expose preload settings on Parquet options builder * fix(arrow): expose Parquet preload getters * refactor(arrow): resolve reader concurrency independently of scans * refactor(arrow): keep reader concurrency solely in options * Update crates/iceberg/src/arrow/reader/mod.rs Co-authored-by: blackmwk <257669749+blackmwk@users.noreply.github.com> * docs(arrow): clarify reader options migration and preload behavior --------- Co-authored-by: blackmwk <257669749+blackmwk@users.noreply.github.com>
…e previous file (apache#3695) * fix(spec): validate the metadata log in build() with the entry for the previous file * test(spec): metadata change after a branch snapshot within the clock-skew tolerance builds
* feat: add TableMetadataBuilder::build_replacement Backs `REPLACE TABLE`: swaps the schema, partition spec and sort order while keeping the table's uuid, properties and snapshots, and drops the main branch so the replaced table has no current snapshot. Columns the table already has keep their field ids, so data files written before the replacement stay readable through the shared columns. New columns draw ids above `last_column_id`, never reusing a dropped column's id. Partition field ids are likewise reused across historical specs. Extracts `rebind_to_schema` out of `reassign_ids` so the partition spec and sort order can be rebound to a schema whose ids were assigned either from scratch or by `assign_fresh_ids`. Format version 1 is not supported. * fix docs * tighten up tests * nits * test: cover build_replacement error paths and refs, document id matching * refactor: take a typed format_version in build_replacement * refactor: make build_replacement pub(crate) until RTAS uses it Moves the doc example into a unit test, since a doctest cannot call a pub(crate) function. * fix: return an error when build_replacement runs out of partition ids Reuses equivalent ids through reuse_partition_field_ids and assigns the rest with PartitionSpecBuilder, which errors on overflow instead of wrapping. The AddSpec change still carries every field id.
…rted load (apache#3697) * fix(arrow): wake equality-delete waiters with the notifier of the started load Co-authored-by: JeonDaehong <90895144+JeonDaehong@users.noreply.github.com> * Assert that insert_equality_delete gets the notifier of the started load --------- Co-authored-by: JeonDaehong <90895144+JeonDaehong@users.noreply.github.com>
…e predicate (apache#3507) * fix(reader): keep pages whose index bounds cannot be compared with the predicate Page-index bounds are built with the table field type but the file's physical literal. When the two differ, e.g. a long field read against INT32 pages written before an int -> long promotion, a double field against FLOAT pages, or a UINT_32 column mapped to long, the bounds never compare with the predicate literal. Inequality and IN predicates treated that as "cannot match" and skipped pages holding matching rows. Keep a page whenever its bound cannot be compared with the literal: the inequality check now treats an incomparable bound as a possible match, and IN uses all(lt) / all(gt) instead of !any(ge) / !any(le). * test(reader): cover a mixed UINT_32 page in the page-index evaluator A page holding both 100 and 3_000_000_000 has a raw signed max below its min, which pins the unsigned column order that a later bound widening has to zero-extend rather than sign-extend. * test(reader): cover <= and > on INT32 pages of a promoted long field * test(reader): cover all four inequality operators on promoted float and timestamp pages * test(reader): pass RowGroupPageIndex to the page-index evaluator in the promoted-type tests Arrow 60 (apache#3257) changed PageIndexEvaluator::eval to take a RowGroupPageIndex and removed ParquetMetaData::column_index, so the tests added here no longer built after merging main.
Defer Moka cache construction until the first manifest or manifest-list read, and skip it entirely when caching is disabled. Share lazy cache state across clones and retain existing cache policy and encryption. Adapted from #34 for upstream main, without the fork-specific FileIO rebinding helper. Co-authored-by: Geoffrey Claude <geoffrey.claude@datadoghq.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fast-forwards the fork's
mainto apache/iceberg-rust@c9d52179bb (upstreammainon 2026-11-16). The fork'smainhas no commits of its own: it isdc349284, an older upstream commit, so this adds the 891 upstream commits since then and nothing else.It includes, among others:
4dd8c946, the upstream base dd-source's Iceberg defork currently pins.Please merge with a merge commit, not squash, so the upstream SHAs stay reachable from
mainand later syncs remain fast-forwards.Note: upstream
mainis on Arrow 60 and reqwest 0.13.