Skip to content

Promote columnar Column/ColumnBuilder into the container crate#809

Open
frankmcsherry wants to merge 1 commit into
masterfrom
columnar-container
Open

Promote columnar Column/ColumnBuilder into the container crate#809
frankmcsherry wants to merge 1 commit into
masterfrom
columnar-container

Conversation

@frankmcsherry

Copy link
Copy Markdown
Member

Moves the columnar container out of examples/columnar.rs and into timely_container, so it becomes a first-class, documented container rather than something users copy out of the example.

What changes

  • Column<C> is now a type alias for columnar's Stash<C, Bytes>, with the container traits (Accountable, DrainContainer, SizableContainer, PushInto) implemented directly on it. No wrapper struct, and no re-statement of Stash's typed/bytes/aligned representation — the previous example newtype only delegated to Stash.
  • container gains a columnar (+ timely_bytes) dependency. It already carries concrete container impls (Vec<T>, …), so this isn't a new kind of dependency; it teaches the container layer about columnar, which lower crates (e.g. communication, already a columnar dependent) can also use.
  • The wire encoding stays in timely. dataflow::channels::ContainerBytes is a timely trait, so its impl for the alias lives there (a local trait on a foreign type — no orphan issue), next to the existing Vec<T> impl.
  • ColumnBuilder moves across unchanged, now with a comment recording why it encodes eagerly: it keeps the (expensive-to-rebuild) typed accumulator resident, so an exchanged batch — sent to a peer and never returned — need not round-trip to be recycled. Behavior is identical.
  • examples/columnar.rs shrinks to use timely::container::{Column, ColumnBuilder} over the same dataflow (~125 fewer lines), so it now exercises the public API.

Convenience alias ColumnOf<T> and a named COLUMN_CAPACITY_BYTES constant (previously a magic 1 << 20) are included.

Notes

  • No default is changed: Vec<T> + bincode remains the default container. This is purely additive.
  • The at_capacity threshold (1 MiB) is preserved as a documented default; tuning it is deliberately left as a bring-your-own-builder concern.

Verified locally: cargo build --workspace --all-targets, clippy clean on the new module, and examples/columnar runs and produces the expected wordcount output.

🤖 Generated with Claude Code

Moves the columnar container out of `examples/columnar.rs` and into
`timely_container`, where it becomes a first-class, documented container
that users can reach for without copying it out of the example.

`Column<C>` is now a type alias for `columnar`'s `Stash<C, Bytes>`, with
the container traits (`Accountable`, `DrainContainer`, `SizableContainer`,
`PushInto`) implemented directly on it — no wrapper struct, and no
re-statement of `Stash`'s typed/bytes/aligned representation. This makes
`container` depend on `columnar` (and `timely_bytes`); it already carries
concrete impls such as `Vec<T>`, so this is not a new kind of dependency.

The bespoke `ColumnBuilder` moves across unchanged, now with a comment
recording why it encodes eagerly: it keeps the (expensive to rebuild)
typed accumulator resident in the builder, so an exchanged batch — which
is sent to a peer and never returns — need not round-trip to be recycled.

The wire encoding (`dataflow::channels::ContainerBytes`) is a `timely`
trait, so its impl stays in the `timely` crate, implemented on the alias.

`examples/columnar.rs` shrinks to `use timely::container::{Column,
ColumnBuilder}` over the same dataflow (~125 fewer lines).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant