Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 2 additions & 54 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -1,57 +1,5 @@
# C² Engineering Guidelines

## Repository Workflow
Before planning or modifying this repository, read [CONTRIBUTING.md](CONTRIBUTING.md) in full and treat its development, design, Rust style, documentation, changelog, and pull request guidance as repository requirements.

Before planning or modifying this repository, read `CONTRIBUTING.md` and treat its workspace layout and validation guidance as repository requirements. Use `cargo x` as the source of truth for routine check, test, lint, and benchmark workflows.

## Priorities

Use this order when correctness and performance goals compete:

1. Keep the request path simple and fast.
2. Preserve or improve hit rate.
3. Prefer newer values when sequence information makes that cheap.

C² uses best-effort consistency and may return stale hits. Sequence numbers provide advisory ordering. Each L2 attempt uses one index lookup and one local validation pass.

## Global bounds

- Bound every cache-owned allocation, queue, buffer, probe, scan, retry, and eviction decision.
- Resolve saturation through L1 bypass, a cache miss, mutation throttling, or explicit overload.
- Keep critical sections short and shard-local. Release locks before device I/O.
- Keep read, write, and reclaim I/O pools bounded and independent.
- Compute the full-key hash and record size once at the public boundary.
- Give compare-exchange bookkeeping a small retry budget, then bypass or miss.
- Keep statistics optional and implement enabled counters with low-cost atomics.

## Read path

- Probe L1 through shard routing and one short critical section. Bound same-hash chains and victim work; pressure bypasses insertion and promotion.
- Consult L2 after every L1 miss. Index misses and index or warm-page contention return a miss before read-buffer allocation.
- Admit one Region- and size-class-bounded read. Reserve one read slot, charge one managed buffer, and validate the record locally.
- Expand direct-I/O reads to 4 KiB boundaries within the selected Region.
- Use the full read-pool depth for immediate admission.
- Optional waiting begins after candidate selection and is limited by a short deadline and an explicit waiter bound. Wait capacity is independent of execution capacity and defaults to the aggregate read-pool depth. A queued request retains its plan and allocates its buffer after admission.
- Queue, memory, and timeout pressure return overload in wait mode. Memory and engine pressure return a miss in immediate mode.
- Validate address, generation, size class, hash, full key, lengths, checksums, and sequence structure in one pass.
- Successful promotion returns an L1-backed value, preserves Region as the hit source, and releases the transient read buffer. Bypassed promotion may return a zero-copy Region value that owns the aligned allocation until drop.
- Heat updates are single, bounded, lossy relaxed-atomic operations independent of the read result.

## Mutation path

- Finish foreground mutations after bounded in-memory work. Accepted writes publish asynchronously.
- Preflight shard capacity before allocating an append receipt. Reserve the Region tail and open span under one manager try-lock; encode under the shard mutation gate after releasing the manager guard.
- Batch Region writes. Background shard workers own write-slot waits; explicit completion barriers may wait for those workers.
- Publish an L2 index entry after its data write completes.
- Make L1 admission immediately readable and evictable. L1 bypass continues through Region.
- Route `put_l2` payloads through Region, apply best-effort L1 cleanup, and make them visible after L2 publication.
- Preserve logical sequence numbers across reinsertion and use them to cheaply prefer newer L1 values. L2 stores compact location metadata.
- Accept older valid values from contention, eviction, delayed publication, and promotion. Validate the complete key after every hash match.
- Treat `drain` as the completion barrier for accepted writes.

## Recovery and failure

- Treat eviction, bypass, rejection, throttling, misses, stale reads, overload, and cache loss as valid cache outcomes.
- Publish a clean recovery image during warm close. Fast close and unclean exit reopen empty.
- Move unsafe I/O, index, or metadata failures to miss-only. Reads then return misses, while mutations report errors.
- Return values that pass bounds, key, and checksum validation. Failed validation returns a miss.
Keep the documented workflow synchronized with structural changes and run the applicable validation commands before handing work back. `CONTRIBUTING.md` is the single source of truth for contribution rules.
11 changes: 4 additions & 7 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,11 @@

### Breaking Changes

- Error types are exported only from the crate root. Replace imports from `cache2::error` with `cache2::{Error, ErrorKind, ErrorOperation, Result}`.
- Configuration now separates editable `StorageOptions` / `RuntimeOptions` from immutable `StorageLayout` / `CacheConfig`. Build the layout, construct `CacheConfig::new(layout, options)`, and call `Cache::open(path, config)` or `Cache::open_with_handle(path, config, handle)`. `StaticConfig`, `RuntimeConfig`, `CacheBuilder`, and the standalone `validate` method are removed.
- The public I/O configuration enum is now named `IoEngineConfig`; replace `IoEngine` imports and variant paths with `IoEngineConfig`.
- Error types are exported only from the crate root, and the `cache2::Result` alias is removed. Replace imports from `cache2::error` with `cache2::{Error, ErrorKind, ErrorOperation}` and use the standard `Result<T, Error>`; public operation error types are unchanged.
- Configuration now separates editable `StorageOptions` / `RuntimeOptions` from immutable `StorageLayout` / `CacheConfig`. Replace `StaticConfig`, `RuntimeConfig`, and `CacheBuilder` by building a layout, constructing `CacheConfig::new(layout, options)`, and calling `Cache::open(path, config)` or `Cache::open_with_handle(path, config, handle)`. Construction validates and retains geometry and memory requirements; the standalone `validate` method is removed. Configurations can be inspected and reused without file access or an active Tokio runtime, with paths and runtime handles supplied separately at open.
- `ReadAdmission::Immediate` and `ReadAdmission::Wait { timeout, max_waiters }` replace the separate read-wait setters. Waiting requires a positive timeout; an omitted waiter bound follows the selected read execution capacity.
- Configuration errors identify `ErrorOperation::BuildStorage` or `BuildConfig`. `StorageLayout::peak_disk_bytes()` is now an infallible query.

### Improvements

- Geometry and memory requirements are retained through startup. Configurations can be inspected and reused without file access or an active Tokio runtime; paths and runtime handles are supplied separately when opening each cache.
- Configuration errors identify `ErrorOperation::BuildStorage` or `BuildConfig` instead of `ValidateConfig` or `PeakDiskBytes`. Replace fallible `StaticConfig::peak_disk_bytes()` calls with the infallible `StorageLayout::peak_disk_bytes()` query.

## v0.3.0 (2026-09-04)

Expand Down
4 changes: 2 additions & 2 deletions CONFIGURATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -270,14 +270,14 @@ io_uring is feature-gated and experimental. Its three pools configure physical r

```rust
use cache2::{
IoEngine, IoMode, IoUringConfig, IoUringPoolConfig,
IoEngineConfig, IoMode, IoUringConfig, IoUringPoolConfig,
IoUringSqPollConfig, RuntimeOptions,
};

let read = IoUringPoolConfig::new(1, 128)
.with_sq_poll(IoUringSqPollConfig::new(2_000).with_cpu(4));
let runtime = RuntimeOptions {
io_engine: IoEngine::IoUring(IoUringConfig::new(
io_engine: IoEngineConfig::IoUring(IoUringConfig::new(
read,
IoUringPoolConfig::new(1, 64),
IoUringPoolConfig::new(1, 1),
Expand Down
72 changes: 23 additions & 49 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,75 +1,49 @@
# Contributing to C²

C² requires Rust 1.98.0. Run development commands from the repository root so Cargo uses the complete workspace and the shared dependency and lint policy.
## Workspace

## Workspace layout
`cache2/` contains the published library and its private implementation tests. `tests-integration/` exercises the public API; `benchmarks/` and `examples/` contain workloads and runnable integrations. `xtask/` implements the `cargo x` development commands.

The repository separates published code from development-only consumers:
## Development

| Path | Purpose |
|----------------------|-----------------------------------------------------------------------------------------------|
| `cache2/` | The publishable `cache2` crate and private implementation tests. |
| `tests-integration/` | End-to-end tests that exercise only the public `cache2` API. |
| `benchmarks/` | Standalone benchmark targets and workload-specific harnesses. |
| `examples/` | Runnable programs that demonstrate complete integrations. |
| `xtask/` | The `cargo x` repository workflow entrypoint. |
Run commands from the repository root. Use `cargo x` as the source of truth for repository workflows. Read `cargo x --help` and the relevant subcommand's `--help` before running build, test, lint, or formatting commands.

Keep unit tests beside the implementation when they need private access. Behavior visible to callers belongs in `tests-integration/tests`.
Use a Rust toolchain at or above the `rust-version` declared in [Cargo.toml](Cargo.toml). Linting also requires nightly Rust and these tools:

## Repository workflows
```sh
rustup toolchain install nightly --profile minimal --component rustfmt,clippy
cargo install --locked cargo-deny hawkeye taplo-cli typos-cli
```

The `.cargo/config.toml` alias maps `cargo x` to the `x` package in `xtask/`. Use these commands before opening a pull request:
Before submitting a pull request, run:

```sh
cargo x check
cargo x test
cargo x lint
```

`cargo x check` verifies the workspace and each optional `cache2` feature. `cargo x test` runs workspace tests with all features and the ignored extended library tests. These commands use the selected toolchain. `cargo x lint` explicitly uses nightly for Rust formatting, Clippy, and public documentation, and also checks TOML formatting, spelling, the publishable package, license headers, dependency licenses, advisories, and sources.
`check` covers the workspace and optional features, `test` includes the extended library tests, and `lint` checks formatting, code, documentation, packaging, and dependencies. Use `cargo x lint --fix` to apply supported automatic fixes, then review the diff.

The lint workflow uses nightly Rust and the latest releases of the lint tools so new diagnostics are caught early:
Cover observable behavior changes with tests. See [BENCHMARK.md](BENCHMARK.md) for performance workloads and qualification.

```sh
rustup toolchain install nightly --profile minimal --component rustfmt,clippy
cargo install cargo-deny --locked
cargo install hawkeye --locked
cargo install taplo-cli --locked
cargo install typos-cli --locked
```
## Design and Rust Style

Run `cargo x lint --fix` to apply Clippy fixes, Rust and TOML formatting, and license headers. Review the changes, then run `cargo x lint` to verify the result. The Rust import and comment style follows `rustfmt.toml`; TOML formatting follows `taplo.toml`.

CI runs nightly lint and feature checks in `check`, tests on Linux and macOS with Rust 1.98.0 and stable in `test`, and the pinned ASan/TSan suite in `safety`. The final `Required` job succeeds only when all three jobs succeed; failures, cancellations, and skipped dependencies fail the gate. Pull requests and pushes to `main`, including documentation changes, run the workflow.

Use the underlying Cargo commands directly when isolating a failure. The release-mode test pass used by CI is:

```sh
cargo test --workspace --release --all-features
```
Follow the surrounding code and the design constraints in [ARCHITECTURE.md](ARCHITECTURE.md), including bounded resource use, best-effort consistency, request-path priorities, and recovery guarantees.

## Rust Style

Declare restricted visibility at the module boundary and use `pub` for items in that module's API.
Declare restricted visibility at module boundaries and use `pub` for items in those modules' APIs. Keep items private when only their defining module and its descendants need them. For items reachable through public modules or re-exported public types, reserve `pub` for intentional public API and use narrower visibility for internal callers.

## Documentation

Keep each Markdown prose paragraph and list item on one source line.

## Benchmarks and property tests

Each benchmark is an explicit target in the `benchmarks` package. Run one target with:

```sh
cargo x bench --bench cache
```
Keep public documentation current and describe observable contracts. Keep each Markdown prose paragraph and list item on one source line. Format Markdown tables so their columns and separators align in the source.

See `BENCHMARK.md` for workload controls and qualification requirements. The normal test workflow also runs 10,000 QuickCheck cases for each of four properties: persistent decoders, record round trips, the fixed-map state machine, and the Region-index state machine. Inputs are capped at 16 KiB. Run that group directly with:
## Changelog

```sh
cargo test --package cache2 --lib property_tests::
```
- Update [CHANGELOG.md](CHANGELOG.md) for significant user-visible changes by comparing the final behavior with the latest release tag, not the sequence of commits in the current development cycle. Add entries under `Unreleased`, using only categories that contain entries.
- Before adding a bug-fix entry, verify from the latest release tag that the faulty behavior was shipped. If the affected API or behavior is unreleased, describe only its final contract in the relevant feature entry and omit the development-only correction.
- Include public API migrations, new capabilities, correctness or compatibility changes, and meaningful performance improvements. Exclude tests, internal refactors, documentation, CI, tooling, dependency maintenance, discarded intermediate APIs, and implementation history unless they change supported or observable behavior relative to the latest release.
- Write each entry from the user's perspective as one coherent observable change, including required migration guidance for breaking changes. Scope performance claims to the workloads supported by evidence.

## Changelog
## Pull Requests

Update `CHANGELOG.md` for user-visible API, correctness, compatibility, performance, or operational changes. Internal refactors, tests, documentation, CI, tooling, and dependency maintenance do not need an entry unless they alter observable behavior.
Format pull request titles according to [.github/semantic.yml](.github/semantic.yml) and keep the description concise. Use a `Summary` section for routine changes and add `Design Notes` only when the design needs explanation.
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,9 @@ C² (`cache2`) provides bounded, disposable acceleration for large file chunks.
## Quick start

```rust
use cache2::{Cache, CacheConfig, ErrorKind, Result, RuntimeOptions, StorageOptions};
use cache2::{Cache, CacheConfig, Error, ErrorKind, RuntimeOptions, StorageOptions};

async fn run() -> Result<()> {
async fn run() -> Result<(), Error> {
let storage = StorageOptions::new(1024 * 1024 * 1024).build()?;
let config = CacheConfig::new(storage, RuntimeOptions::default())?;
let cache = Cache::open("/var/tmp/cache2.data", config).await?;
Expand Down Expand Up @@ -77,7 +77,7 @@ See the [configuration guide](CONFIGURATION.md#configuration-lifecycle) for exam
| Area | Controls | Default and behavior |
|-----------|----------------------------------------------------------------------------|---------------------------------------------------------------------------------------------|
| L1 | `l1_capacity_bytes`, `l1_shards`, `l1_eviction_policy` | 256 MiB, 32 shards, CLOCK. Zero capacity disables L1; entries charged above 256 KiB use L2. |
| I/O pools | `io_engine: IoEngine::Posix(...)` or `IoEngine::IoUring(...)` | Four POSIX read workers, four write workers, and one reclaimer; io_uring is experimental. |
| I/O pools | `io_engine: IoEngineConfig::Posix(...)` or `IoEngineConfig::IoUring(...)` | Four POSIX read workers, four write workers, and one reclaimer; io_uring is experimental. |
| Read wait | `read_admission: ReadAdmission::Immediate` or `ReadAdmission::Wait { .. }` | Immediate admission; wait capacity defaults to aggregate read capacity. |
| Writes | `append_shards`, `write_flush_threshold_bytes` | Four append shards and a 4 MiB flush threshold. |
| Memory | `managed_memory_limit_bytes` | 1 GiB across cache-managed allocations. |
Expand Down
Loading