Promote columnar Column/ColumnBuilder into the container crate#809
Open
frankmcsherry wants to merge 1 commit into
Open
Promote columnar Column/ColumnBuilder into the container crate#809frankmcsherry wants to merge 1 commit into
Column/ColumnBuilder into the container crate#809frankmcsherry wants to merge 1 commit into
Conversation
Moves the columnar container out of `examples/columnar.rs` and into
`timely_container`, where it becomes a first-class, documented container
that users can reach for without copying it out of the example.
`Column<C>` is now a type alias for `columnar`'s `Stash<C, Bytes>`, with
the container traits (`Accountable`, `DrainContainer`, `SizableContainer`,
`PushInto`) implemented directly on it — no wrapper struct, and no
re-statement of `Stash`'s typed/bytes/aligned representation. This makes
`container` depend on `columnar` (and `timely_bytes`); it already carries
concrete impls such as `Vec<T>`, so this is not a new kind of dependency.
The bespoke `ColumnBuilder` moves across unchanged, now with a comment
recording why it encodes eagerly: it keeps the (expensive to rebuild)
typed accumulator resident in the builder, so an exchanged batch — which
is sent to a peer and never returns — need not round-trip to be recycled.
The wire encoding (`dataflow::channels::ContainerBytes`) is a `timely`
trait, so its impl stays in the `timely` crate, implemented on the alias.
`examples/columnar.rs` shrinks to `use timely::container::{Column,
ColumnBuilder}` over the same dataflow (~125 fewer lines).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Moves the columnar container out of
examples/columnar.rsand intotimely_container, so it becomes a first-class, documented container rather than something users copy out of the example.What changes
Column<C>is now a type alias forcolumnar'sStash<C, Bytes>, with the container traits (Accountable,DrainContainer,SizableContainer,PushInto) implemented directly on it. No wrapper struct, and no re-statement ofStash's typed/bytes/aligned representation — the previous example newtype only delegated toStash.containergains acolumnar(+timely_bytes) dependency. It already carries concrete container impls (Vec<T>, …), so this isn't a new kind of dependency; it teaches the container layer about columnar, which lower crates (e.g.communication, already acolumnardependent) can also use.timely.dataflow::channels::ContainerBytesis atimelytrait, so its impl for the alias lives there (a local trait on a foreign type — no orphan issue), next to the existingVec<T>impl.ColumnBuildermoves across unchanged, now with a comment recording why it encodes eagerly: it keeps the (expensive-to-rebuild) typed accumulator resident, so an exchanged batch — sent to a peer and never returned — need not round-trip to be recycled. Behavior is identical.examples/columnar.rsshrinks touse timely::container::{Column, ColumnBuilder}over the same dataflow (~125 fewer lines), so it now exercises the public API.Convenience alias
ColumnOf<T>and a namedCOLUMN_CAPACITY_BYTESconstant (previously a magic1 << 20) are included.Notes
Vec<T>+ bincode remains the default container. This is purely additive.at_capacitythreshold (1 MiB) is preserved as a documented default; tuning it is deliberately left as a bring-your-own-builder concern.Verified locally:
cargo build --workspace --all-targets, clippy clean on the new module, andexamples/columnarruns and produces the expected wordcount output.🤖 Generated with Claude Code