diff --git a/AGENTS.md b/AGENTS.md index 4ff74c6..27cd76e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,57 +1,5 @@ # C² Engineering Guidelines -## Repository Workflow +Before planning or modifying this repository, read [CONTRIBUTING.md](CONTRIBUTING.md) in full and treat its development, design, Rust style, documentation, changelog, and pull request guidance as repository requirements. -Before planning or modifying this repository, read `CONTRIBUTING.md` and treat its workspace layout and validation guidance as repository requirements. Use `cargo x` as the source of truth for routine check, test, lint, and benchmark workflows. - -## Priorities - -Use this order when correctness and performance goals compete: - -1. Keep the request path simple and fast. -2. Preserve or improve hit rate. -3. Prefer newer values when sequence information makes that cheap. - -C² uses best-effort consistency and may return stale hits. Sequence numbers provide advisory ordering. Each L2 attempt uses one index lookup and one local validation pass. - -## Global bounds - -- Bound every cache-owned allocation, queue, buffer, probe, scan, retry, and eviction decision. -- Resolve saturation through L1 bypass, a cache miss, mutation throttling, or explicit overload. -- Keep critical sections short and shard-local. Release locks before device I/O. -- Keep read, write, and reclaim I/O pools bounded and independent. -- Compute the full-key hash and record size once at the public boundary. -- Give compare-exchange bookkeeping a small retry budget, then bypass or miss. -- Keep statistics optional and implement enabled counters with low-cost atomics. - -## Read path - -- Probe L1 through shard routing and one short critical section. Bound same-hash chains and victim work; pressure bypasses insertion and promotion. -- Consult L2 after every L1 miss. Index misses and index or warm-page contention return a miss before read-buffer allocation. -- Admit one Region- and size-class-bounded read. Reserve one read slot, charge one managed buffer, and validate the record locally. -- Expand direct-I/O reads to 4 KiB boundaries within the selected Region. -- Use the full read-pool depth for immediate admission. -- Optional waiting begins after candidate selection and is limited by a short deadline and an explicit waiter bound. Wait capacity is independent of execution capacity and defaults to the aggregate read-pool depth. A queued request retains its plan and allocates its buffer after admission. -- Queue, memory, and timeout pressure return overload in wait mode. Memory and engine pressure return a miss in immediate mode. -- Validate address, generation, size class, hash, full key, lengths, checksums, and sequence structure in one pass. -- Successful promotion returns an L1-backed value, preserves Region as the hit source, and releases the transient read buffer. Bypassed promotion may return a zero-copy Region value that owns the aligned allocation until drop. -- Heat updates are single, bounded, lossy relaxed-atomic operations independent of the read result. - -## Mutation path - -- Finish foreground mutations after bounded in-memory work. Accepted writes publish asynchronously. -- Preflight shard capacity before allocating an append receipt. Reserve the Region tail and open span under one manager try-lock; encode under the shard mutation gate after releasing the manager guard. -- Batch Region writes. Background shard workers own write-slot waits; explicit completion barriers may wait for those workers. -- Publish an L2 index entry after its data write completes. -- Make L1 admission immediately readable and evictable. L1 bypass continues through Region. -- Route `put_l2` payloads through Region, apply best-effort L1 cleanup, and make them visible after L2 publication. -- Preserve logical sequence numbers across reinsertion and use them to cheaply prefer newer L1 values. L2 stores compact location metadata. -- Accept older valid values from contention, eviction, delayed publication, and promotion. Validate the complete key after every hash match. -- Treat `drain` as the completion barrier for accepted writes. - -## Recovery and failure - -- Treat eviction, bypass, rejection, throttling, misses, stale reads, overload, and cache loss as valid cache outcomes. -- Publish a clean recovery image during warm close. Fast close and unclean exit reopen empty. -- Move unsafe I/O, index, or metadata failures to miss-only. Reads then return misses, while mutations report errors. -- Return values that pass bounds, key, and checksum validation. Failed validation returns a miss. +Keep the documented workflow synchronized with structural changes and run the applicable validation commands before handing work back. `CONTRIBUTING.md` is the single source of truth for contribution rules. diff --git a/CHANGELOG.md b/CHANGELOG.md index 1aa5952..fe9a6cf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,14 +4,11 @@ ### Breaking Changes -- Error types are exported only from the crate root. Replace imports from `cache2::error` with `cache2::{Error, ErrorKind, ErrorOperation, Result}`. -- Configuration now separates editable `StorageOptions` / `RuntimeOptions` from immutable `StorageLayout` / `CacheConfig`. Build the layout, construct `CacheConfig::new(layout, options)`, and call `Cache::open(path, config)` or `Cache::open_with_handle(path, config, handle)`. `StaticConfig`, `RuntimeConfig`, `CacheBuilder`, and the standalone `validate` method are removed. +- The public I/O configuration enum is now named `IoEngineConfig`; replace `IoEngine` imports and variant paths with `IoEngineConfig`. +- Error types are exported only from the crate root, and the `cache2::Result` alias is removed. Replace imports from `cache2::error` with `cache2::{Error, ErrorKind, ErrorOperation}` and use the standard `Result`; public operation error types are unchanged. +- Configuration now separates editable `StorageOptions` / `RuntimeOptions` from immutable `StorageLayout` / `CacheConfig`. Replace `StaticConfig`, `RuntimeConfig`, and `CacheBuilder` by building a layout, constructing `CacheConfig::new(layout, options)`, and calling `Cache::open(path, config)` or `Cache::open_with_handle(path, config, handle)`. Construction validates and retains geometry and memory requirements; the standalone `validate` method is removed. Configurations can be inspected and reused without file access or an active Tokio runtime, with paths and runtime handles supplied separately at open. - `ReadAdmission::Immediate` and `ReadAdmission::Wait { timeout, max_waiters }` replace the separate read-wait setters. Waiting requires a positive timeout; an omitted waiter bound follows the selected read execution capacity. -- Configuration errors identify `ErrorOperation::BuildStorage` or `BuildConfig`. `StorageLayout::peak_disk_bytes()` is now an infallible query. - -### Improvements - -- Geometry and memory requirements are retained through startup. Configurations can be inspected and reused without file access or an active Tokio runtime; paths and runtime handles are supplied separately when opening each cache. +- Configuration errors identify `ErrorOperation::BuildStorage` or `BuildConfig` instead of `ValidateConfig` or `PeakDiskBytes`. Replace fallible `StaticConfig::peak_disk_bytes()` calls with the infallible `StorageLayout::peak_disk_bytes()` query. ## v0.3.0 (2026-09-04) diff --git a/CONFIGURATION.md b/CONFIGURATION.md index 6767efa..7deb233 100644 --- a/CONFIGURATION.md +++ b/CONFIGURATION.md @@ -270,14 +270,14 @@ io_uring is feature-gated and experimental. Its three pools configure physical r ```rust use cache2::{ - IoEngine, IoMode, IoUringConfig, IoUringPoolConfig, + IoEngineConfig, IoMode, IoUringConfig, IoUringPoolConfig, IoUringSqPollConfig, RuntimeOptions, }; let read = IoUringPoolConfig::new(1, 128) .with_sq_poll(IoUringSqPollConfig::new(2_000).with_cpu(4)); let runtime = RuntimeOptions { - io_engine: IoEngine::IoUring(IoUringConfig::new( + io_engine: IoEngineConfig::IoUring(IoUringConfig::new( read, IoUringPoolConfig::new(1, 64), IoUringPoolConfig::new(1, 1), diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e5165c7..e0f92bc 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,24 +1,21 @@ # Contributing to C² -C² requires Rust 1.98.0. Run development commands from the repository root so Cargo uses the complete workspace and the shared dependency and lint policy. +## Workspace -## Workspace layout +`cache2/` contains the published library and its private implementation tests. `tests-integration/` exercises the public API; `benchmarks/` and `examples/` contain workloads and runnable integrations. `xtask/` implements the `cargo x` development commands. -The repository separates published code from development-only consumers: +## Development -| Path | Purpose | -|----------------------|-----------------------------------------------------------------------------------------------| -| `cache2/` | The publishable `cache2` crate and private implementation tests. | -| `tests-integration/` | End-to-end tests that exercise only the public `cache2` API. | -| `benchmarks/` | Standalone benchmark targets and workload-specific harnesses. | -| `examples/` | Runnable programs that demonstrate complete integrations. | -| `xtask/` | The `cargo x` repository workflow entrypoint. | +Run commands from the repository root. Use `cargo x` as the source of truth for repository workflows. Read `cargo x --help` and the relevant subcommand's `--help` before running build, test, lint, or formatting commands. -Keep unit tests beside the implementation when they need private access. Behavior visible to callers belongs in `tests-integration/tests`. +Use a Rust toolchain at or above the `rust-version` declared in [Cargo.toml](Cargo.toml). Linting also requires nightly Rust and these tools: -## Repository workflows +```sh +rustup toolchain install nightly --profile minimal --component rustfmt,clippy +cargo install --locked cargo-deny hawkeye taplo-cli typos-cli +``` -The `.cargo/config.toml` alias maps `cargo x` to the `x` package in `xtask/`. Use these commands before opening a pull request: +Before submitting a pull request, run: ```sh cargo x check @@ -26,50 +23,27 @@ cargo x test cargo x lint ``` -`cargo x check` verifies the workspace and each optional `cache2` feature. `cargo x test` runs workspace tests with all features and the ignored extended library tests. These commands use the selected toolchain. `cargo x lint` explicitly uses nightly for Rust formatting, Clippy, and public documentation, and also checks TOML formatting, spelling, the publishable package, license headers, dependency licenses, advisories, and sources. +`check` covers the workspace and optional features, `test` includes the extended library tests, and `lint` checks formatting, code, documentation, packaging, and dependencies. Use `cargo x lint --fix` to apply supported automatic fixes, then review the diff. -The lint workflow uses nightly Rust and the latest releases of the lint tools so new diagnostics are caught early: +Cover observable behavior changes with tests. See [BENCHMARK.md](BENCHMARK.md) for performance workloads and qualification. -```sh -rustup toolchain install nightly --profile minimal --component rustfmt,clippy -cargo install cargo-deny --locked -cargo install hawkeye --locked -cargo install taplo-cli --locked -cargo install typos-cli --locked -``` +## Design and Rust Style -Run `cargo x lint --fix` to apply Clippy fixes, Rust and TOML formatting, and license headers. Review the changes, then run `cargo x lint` to verify the result. The Rust import and comment style follows `rustfmt.toml`; TOML formatting follows `taplo.toml`. - -CI runs nightly lint and feature checks in `check`, tests on Linux and macOS with Rust 1.98.0 and stable in `test`, and the pinned ASan/TSan suite in `safety`. The final `Required` job succeeds only when all three jobs succeed; failures, cancellations, and skipped dependencies fail the gate. Pull requests and pushes to `main`, including documentation changes, run the workflow. - -Use the underlying Cargo commands directly when isolating a failure. The release-mode test pass used by CI is: - -```sh -cargo test --workspace --release --all-features -``` +Follow the surrounding code and the design constraints in [ARCHITECTURE.md](ARCHITECTURE.md), including bounded resource use, best-effort consistency, request-path priorities, and recovery guarantees. -## Rust Style - -Declare restricted visibility at the module boundary and use `pub` for items in that module's API. +Declare restricted visibility at module boundaries and use `pub` for items in those modules' APIs. Keep items private when only their defining module and its descendants need them. For items reachable through public modules or re-exported public types, reserve `pub` for intentional public API and use narrower visibility for internal callers. ## Documentation -Keep each Markdown prose paragraph and list item on one source line. - -## Benchmarks and property tests - -Each benchmark is an explicit target in the `benchmarks` package. Run one target with: - -```sh -cargo x bench --bench cache -``` +Keep public documentation current and describe observable contracts. Keep each Markdown prose paragraph and list item on one source line. Format Markdown tables so their columns and separators align in the source. -See `BENCHMARK.md` for workload controls and qualification requirements. The normal test workflow also runs 10,000 QuickCheck cases for each of four properties: persistent decoders, record round trips, the fixed-map state machine, and the Region-index state machine. Inputs are capped at 16 KiB. Run that group directly with: +## Changelog -```sh -cargo test --package cache2 --lib property_tests:: -``` +- Update [CHANGELOG.md](CHANGELOG.md) for significant user-visible changes by comparing the final behavior with the latest release tag, not the sequence of commits in the current development cycle. Add entries under `Unreleased`, using only categories that contain entries. +- Before adding a bug-fix entry, verify from the latest release tag that the faulty behavior was shipped. If the affected API or behavior is unreleased, describe only its final contract in the relevant feature entry and omit the development-only correction. +- Include public API migrations, new capabilities, correctness or compatibility changes, and meaningful performance improvements. Exclude tests, internal refactors, documentation, CI, tooling, dependency maintenance, discarded intermediate APIs, and implementation history unless they change supported or observable behavior relative to the latest release. +- Write each entry from the user's perspective as one coherent observable change, including required migration guidance for breaking changes. Scope performance claims to the workloads supported by evidence. -## Changelog +## Pull Requests -Update `CHANGELOG.md` for user-visible API, correctness, compatibility, performance, or operational changes. Internal refactors, tests, documentation, CI, tooling, and dependency maintenance do not need an entry unless they alter observable behavior. +Format pull request titles according to [.github/semantic.yml](.github/semantic.yml) and keep the description concise. Use a `Summary` section for routine changes and add `Design Notes` only when the design needs explanation. diff --git a/README.md b/README.md index 2f4a7db..aa78223 100644 --- a/README.md +++ b/README.md @@ -11,9 +11,9 @@ C² (`cache2`) provides bounded, disposable acceleration for large file chunks. ## Quick start ```rust -use cache2::{Cache, CacheConfig, ErrorKind, Result, RuntimeOptions, StorageOptions}; +use cache2::{Cache, CacheConfig, Error, ErrorKind, RuntimeOptions, StorageOptions}; -async fn run() -> Result<()> { +async fn run() -> Result<(), Error> { let storage = StorageOptions::new(1024 * 1024 * 1024).build()?; let config = CacheConfig::new(storage, RuntimeOptions::default())?; let cache = Cache::open("/var/tmp/cache2.data", config).await?; @@ -77,7 +77,7 @@ See the [configuration guide](CONFIGURATION.md#configuration-lifecycle) for exam | Area | Controls | Default and behavior | |-----------|----------------------------------------------------------------------------|---------------------------------------------------------------------------------------------| | L1 | `l1_capacity_bytes`, `l1_shards`, `l1_eviction_policy` | 256 MiB, 32 shards, CLOCK. Zero capacity disables L1; entries charged above 256 KiB use L2. | -| I/O pools | `io_engine: IoEngine::Posix(...)` or `IoEngine::IoUring(...)` | Four POSIX read workers, four write workers, and one reclaimer; io_uring is experimental. | +| I/O pools | `io_engine: IoEngineConfig::Posix(...)` or `IoEngineConfig::IoUring(...)` | Four POSIX read workers, four write workers, and one reclaimer; io_uring is experimental. | | Read wait | `read_admission: ReadAdmission::Immediate` or `ReadAdmission::Wait { .. }` | Immediate admission; wait capacity defaults to aggregate read capacity. | | Writes | `append_shards`, `write_flush_threshold_bytes` | Four append shards and a 4 MiB flush threshold. | | Memory | `managed_memory_limit_bytes` | 1 GiB across cache-managed allocations. | diff --git a/benchmarks/cache/main.rs b/benchmarks/cache/main.rs index 3f4bee2..1328956 100644 --- a/benchmarks/cache/main.rs +++ b/benchmarks/cache/main.rs @@ -13,12 +13,14 @@ // limitations under the License. use std::env; +use std::fmt; +use std::fs; use std::hint::black_box; use std::io; +use std::ops::Range; use std::path::Path; use std::path::PathBuf; use std::sync::Arc; -use std::thread; use std::time::Duration; use std::time::Instant; use std::time::SystemTime; @@ -32,8 +34,7 @@ use benchmarks::report::emit_cache_report; use cache2::Cache; use cache2::CacheConfig; use cache2::CacheTier; -use cache2::ErrorKind as CacheErrorKind; -use cache2::IoEngine; +use cache2::IoEngineConfig; use cache2::IoMode; use cache2::IoUringConfig; use cache2::IoUringPoolConfig; @@ -77,7 +78,7 @@ struct BenchConfig { reclaim_workers: usize, write_clients: usize, clients: usize, - io_engine: IoEngine, + io_engine: IoEngineConfig, io_mode: IoMode, l1_eviction_policy: L1EvictionPolicy, statistics_enabled: bool, @@ -112,12 +113,12 @@ impl BenchConfig { .unwrap_or_else(|_| "posix".to_owned()) .as_str() { - "posix" => IoEngine::Posix(PosixIoConfig::new( + "posix" => IoEngineConfig::Posix(PosixIoConfig::new( read_io_workers, write_io_workers, reclaim_workers, )), - "io-uring" => IoEngine::IoUring(IoUringConfig::new( + "io-uring" => IoEngineConfig::IoUring(IoUringConfig::new( IoUringPoolConfig::new( read_io_workers, read_io_workers @@ -300,7 +301,7 @@ impl BenchConfig { match (self.io_engine, self.read_io_wait_timeout.is_zero()) { // Keep the benchmark at the POSIX engine's exact admission depth. // Saturation misses belong in the soak, not the device-rate phase. - (IoEngine::Posix(_), true) => self.clients.min(self.read_io_workers), + (IoEngineConfig::Posix(_), true) => self.clients.min(self.read_io_workers), _ => self.clients, } } @@ -333,7 +334,7 @@ impl Drop for BenchFiles { sidecar(&self.data, ".image"), sidecar(&self.data, ".image.next"), ] { - let _ = std::fs::remove_file(path); + let _ = fs::remove_file(path); } } } @@ -381,7 +382,7 @@ fn main() -> io::Result<()> { result .as_ref() .err() - .map(|error| error as &dyn std::fmt::Display), + .map(|error| error as &dyn fmt::Display), ); result } @@ -752,7 +753,7 @@ fn concurrent_writes( clients: usize, ) -> io::Result { let barrier = Arc::new(std::sync::Barrier::new(clients + 1)); - thread::scope(|scope| { + std::thread::scope(|scope| { let mut handles = Vec::with_capacity(clients); for client in 0..clients { let cache = Arc::clone(&cache); @@ -804,7 +805,7 @@ fn concurrent_writes( async fn concurrent_reads( cache: Arc, - key_range: std::ops::Range, + key_range: Range, operations: usize, clients: usize, expected_tier: CacheTier, @@ -985,7 +986,7 @@ fn put_eventually(cache: &Cache, key: &[u8], value: &[u8]) -> io::Result<(u64, u attempts = attempts.saturating_add(1); match cache.put(key, value) { Ok(receipt) => return Ok((receipt, attempts)), - Err(error) if error.kind() == CacheErrorKind::Overloaded => { + Err(error) if error.kind() == cache2::ErrorKind::Overloaded => { if Instant::now() >= deadline { return Err(io::Error::new( io::ErrorKind::TimedOut, @@ -993,9 +994,9 @@ fn put_eventually(cache: &Cache, key: &[u8], value: &[u8]) -> io::Result<(u64, u )); } if attempts <= WRITE_YIELD_RETRIES { - thread::yield_now(); + std::thread::yield_now(); } else { - thread::sleep(RETRY_DELAY); + std::thread::sleep(RETRY_DELAY); } } Err(error) => return Err(error.into()), diff --git a/benchmarks/cache_soak/main.rs b/benchmarks/cache_soak/main.rs index 3ffff1e..fb82ffb 100644 --- a/benchmarks/cache_soak/main.rs +++ b/benchmarks/cache_soak/main.rs @@ -12,14 +12,17 @@ // See the License for the specific language governing permissions and // limitations under the License. +use std::cmp::min; use std::env; +use std::fmt; +use std::fs; use std::io; +use std::mem::MaybeUninit; use std::path::Path; use std::path::PathBuf; use std::sync::atomic::AtomicBool; use std::sync::atomic::AtomicU64; use std::sync::atomic::Ordering; -use std::thread; use std::time::Duration; use std::time::Instant; use std::time::SystemTime; @@ -34,8 +37,7 @@ use cache2::Cache; use cache2::CacheConfig; use cache2::CacheHealth; use cache2::DetailedCacheSnapshot; -use cache2::ErrorKind as CacheErrorKind; -use cache2::IoEngine; +use cache2::IoEngineConfig; use cache2::IoMode; use cache2::IoUringConfig; use cache2::IoUringPoolConfig; @@ -83,7 +85,7 @@ struct SoakConfig { final_warm_verify: bool, require_path_coverage: bool, require_reinsert_coverage: bool, - io_engine: IoEngine, + io_engine: IoEngineConfig, io_mode: IoMode, l1_eviction_policy: L1EvictionPolicy, directory: PathBuf, @@ -263,7 +265,7 @@ impl SoakFiles { sidecar(&self.data, ".image.next"), ] .into_iter() - .try_fold(0_u64, |total, path| match std::fs::metadata(path) { + .try_fold(0_u64, |total, path| match fs::metadata(path) { Ok(metadata) => total .checked_add(metadata.len()) .ok_or_else(|| invalid("logical disk byte count overflow")), @@ -288,7 +290,7 @@ impl Drop for SoakFiles { sidecar(&self.data, ".image"), sidecar(&self.data, ".image.next"), ] { - let _ = std::fs::remove_file(path); + let _ = fs::remove_file(path); } } } @@ -359,7 +361,7 @@ fn main() -> io::Result<()> { result .as_ref() .err() - .map(|error| error as &dyn std::fmt::Display), + .map(|error| error as &dyn fmt::Display), ); result } @@ -446,7 +448,7 @@ fn run_benchmark() -> io::Result<()> { files.data.display(), ); - thread::scope(|scope| -> io::Result<()> { + std::thread::scope(|scope| -> io::Result<()> { let mut workers = Vec::with_capacity(client_count); for _ in 0..config.writers { workers.push(scope.spawn(|| { @@ -499,9 +501,9 @@ fn run_benchmark() -> io::Result<()> { .ok_or_else(|| invalid("soak sample deadline is too far in the future"))?; let mut sample_error = None; while Instant::now() < deadline && !stop.load(Ordering::Acquire) { - let wake_at = std::cmp::min(next_sample, deadline); + let wake_at = min(next_sample, deadline); if let Some(remaining) = wake_at.checked_duration_since(Instant::now()) { - thread::sleep(remaining); + std::thread::sleep(remaining); } let now = Instant::now(); if now >= next_sample && now < deadline { @@ -677,8 +679,8 @@ fn populate_for_warm_reopen( loop { match cache.put(key, &value[..value_bytes]) { Ok(_) => break, - Err(error) if error.kind() == CacheErrorKind::Overloaded => { - thread::sleep(OVERLOAD_DELAY); + Err(error) if error.kind() == cache2::ErrorKind::Overloaded => { + std::thread::sleep(OVERLOAD_DELAY); } Err(error) => return Err(error.into()), } @@ -729,10 +731,10 @@ fn run_writer( record_latency(&counters.put_latency, put_started); counters.writes.fetch_add(1, Ordering::Relaxed); } - Err(error) if error.kind() == CacheErrorKind::Overloaded => { + Err(error) if error.kind() == cache2::ErrorKind::Overloaded => { record_latency(&counters.put_latency, put_started); counters.write_rejections.fetch_add(1, Ordering::Relaxed); - thread::sleep(OVERLOAD_DELAY); + std::thread::sleep(OVERLOAD_DELAY); continue; } Err(error) => return Err(error.into()), @@ -747,10 +749,10 @@ fn run_writer( record_latency(&counters.delete_latency, delete_started); counters.deletes.fetch_add(1, Ordering::Relaxed); } - Err(error) if error.kind() == CacheErrorKind::Overloaded => { + Err(error) if error.kind() == cache2::ErrorKind::Overloaded => { record_latency(&counters.delete_latency, delete_started); counters.delete_rejections.fetch_add(1, Ordering::Relaxed); - thread::sleep(OVERLOAD_DELAY); + std::thread::sleep(OVERLOAD_DELAY); } Err(error) => return Err(error.into()), } @@ -1131,13 +1133,13 @@ fn record_latency(histogram: &AtomicLatencyHistogram, started: Option) fn pace(interval: Duration) { if !interval.is_zero() { - thread::sleep(interval); + std::thread::sleep(interval); } } #[cfg(unix)] fn peak_rss_bytes() -> u64 { - let mut usage = std::mem::MaybeUninit::::zeroed(); + let mut usage = MaybeUninit::::zeroed(); // SAFETY: `usage` points to writable storage for one `rusage` value. if unsafe { libc::getrusage(libc::RUSAGE_SELF, usage.as_mut_ptr()) } != 0 { return 0; @@ -1158,7 +1160,7 @@ fn peak_rss_bytes() -> u64 { #[cfg(target_os = "linux")] fn current_rss_bytes() -> io::Result { - let status = std::fs::read_to_string("/proc/self/status")?; + let status = fs::read_to_string("/proc/self/status")?; let kib = status .lines() .find_map(|line| line.strip_prefix("VmRSS:")) @@ -1241,17 +1243,17 @@ fn parse_io_engine( read_workers: usize, write_workers: usize, reclaim_workers: usize, -) -> io::Result { +) -> io::Result { match env::var(name) .unwrap_or_else(|_| "posix".to_owned()) .as_str() { - "posix" => Ok(IoEngine::Posix(PosixIoConfig::new( + "posix" => Ok(IoEngineConfig::Posix(PosixIoConfig::new( read_workers, write_workers, reclaim_workers, ))), - "io-uring" => Ok(IoEngine::IoUring(IoUringConfig::new( + "io-uring" => Ok(IoEngineConfig::IoUring(IoUringConfig::new( io_uring_read_pool(read_workers)?, io_uring_write_pool(write_workers)?, IoUringPoolConfig::new(reclaim_workers, reclaim_workers), diff --git a/benchmarks/mixed_workloads/main.rs b/benchmarks/mixed_workloads/main.rs index b6956a9..01d705a 100644 --- a/benchmarks/mixed_workloads/main.rs +++ b/benchmarks/mixed_workloads/main.rs @@ -13,6 +13,9 @@ // limitations under the License. use std::env; +use std::f64::consts::TAU; +use std::fmt; +use std::fs; use std::hint::black_box; use std::io; use std::path::Path; @@ -33,8 +36,9 @@ use benchmarks::report::emit_cache_report; use cache2::Cache; use cache2::CacheConfig; use cache2::CacheHealth; -use cache2::ErrorKind as CacheErrorKind; -use cache2::IoEngine; +use cache2::CacheSnapshot; +use cache2::DetailedCacheSnapshot; +use cache2::IoEngineConfig; use cache2::IoMode; use cache2::IoUringConfig; use cache2::IoUringPoolConfig; @@ -209,7 +213,7 @@ struct HarnessConfig { reclaim_workers: usize, latency_sample_interval: usize, seed: u64, - io_engine: IoEngine, + io_engine: IoEngineConfig, io_mode: IoMode, l1_eviction_policy: L1EvictionPolicy, directory: PathBuf, @@ -236,12 +240,12 @@ impl HarnessConfig { .unwrap_or_else(|_| "posix".to_owned()) .as_str() { - "posix" => IoEngine::Posix(PosixIoConfig::new( + "posix" => IoEngineConfig::Posix(PosixIoConfig::new( read_io_workers, write_io_workers, reclaim_workers, )), - "io-uring" => IoEngine::IoUring(IoUringConfig::new( + "io-uring" => IoEngineConfig::IoUring(IoUringConfig::new( IoUringPoolConfig::new( read_io_workers, read_io_workers @@ -410,7 +414,7 @@ struct EffectiveConfig { reclaim_workers: usize, latency_sample_interval: usize, seed: u64, - io_engine: IoEngine, + io_engine: IoEngineConfig, io_mode: IoMode, l1_eviction_policy: L1EvictionPolicy, directory: PathBuf, @@ -467,7 +471,7 @@ impl Drop for BenchFiles { sidecar(&self.data, ".image"), sidecar(&self.data, ".image.next"), ] { - let _ = std::fs::remove_file(file); + let _ = fs::remove_file(file); } } } @@ -602,7 +606,7 @@ async fn run_scenario(config: EffectiveConfig) -> io::Result<()> { result .as_ref() .err() - .map(|error| error as &dyn std::fmt::Display), + .map(|error| error as &dyn fmt::Display), ); result } @@ -744,7 +748,7 @@ async fn run_worker( )); } Ok(None) => result.misses = result.misses.saturating_add(1), - Err(error) if error.kind() == CacheErrorKind::Overloaded => { + Err(error) if error.kind() == cache2::ErrorKind::Overloaded => { result.get_overloaded = result.get_overloaded.saturating_add(1); } Err(error) => return Err(error.into()), @@ -781,7 +785,7 @@ async fn run_worker( )); } Ok(None) => result.misses = result.misses.saturating_add(1), - Err(error) if error.kind() == CacheErrorKind::Overloaded => { + Err(error) if error.kind() == cache2::ErrorKind::Overloaded => { result.get_overloaded = result.get_overloaded.saturating_add(1); } Err(error) => return Err(error.into()), @@ -808,7 +812,7 @@ async fn run_worker( .accepted_value_bytes .saturating_add(value_size as u64); } - Err(error) if error.kind() == CacheErrorKind::Overloaded => { + Err(error) if error.kind() == cache2::ErrorKind::Overloaded => { result.set_overloaded = result.set_overloaded.saturating_add(1); } Err(error) => return Err(error.into()), @@ -823,7 +827,7 @@ async fn run_worker( Ok(_) => { result.delete_accepted = result.delete_accepted.saturating_add(1); } - Err(error) if error.kind() == CacheErrorKind::Overloaded => { + Err(error) if error.kind() == cache2::ErrorKind::Overloaded => { result.delete_overloaded = result.delete_overloaded.saturating_add(1); } Err(error) => return Err(error.into()), @@ -854,7 +858,7 @@ fn sample_normal_key(left: usize, right: usize, rng: &mut DeterministicRng) -> u let standard_deviation = (right - left) as f64 * 0.25; for _ in 0..NORMAL_SAMPLE_ATTEMPTS { let radius = (-2.0 * rng.open_unit_f64().ln()).sqrt(); - let angle = std::f64::consts::TAU * rng.open_unit_f64(); + let angle = TAU * rng.open_unit_f64(); let sampled = (mean + standard_deviation * radius * angle.cos()).round(); if sampled >= left as f64 && sampled <= right as f64 { return sampled as usize; @@ -961,7 +965,7 @@ fn should_sample(operation: Operation, result: &WorkloadResult, interval: usize) } } -fn validate_snapshot(result: &WorkloadResult, snapshot: &cache2::CacheSnapshot) -> io::Result<()> { +fn validate_snapshot(result: &WorkloadResult, snapshot: &CacheSnapshot) -> io::Result<()> { let cache_hits = snapshot.l1_hits.saturating_add(snapshot.l2_hits); if snapshot.health != CacheHealth::Running || snapshot.io_failures != 0 { return Err(io::Error::other( @@ -993,7 +997,7 @@ fn report( result: &WorkloadResult, workload_elapsed: Duration, drain_elapsed: Duration, - detailed: &cache2::DetailedCacheSnapshot, + detailed: &DetailedCacheSnapshot, ) { let snapshot = detailed.summary; let seconds = workload_elapsed.as_secs_f64(); diff --git a/benchmarks/recovery_scale/main.rs b/benchmarks/recovery_scale/main.rs index 0e553dc..f5cc0a9 100644 --- a/benchmarks/recovery_scale/main.rs +++ b/benchmarks/recovery_scale/main.rs @@ -13,10 +13,12 @@ // limitations under the License. use std::env; +use std::fmt; +use std::fs; use std::io; +use std::mem::MaybeUninit; use std::path::Path; use std::path::PathBuf; -use std::thread; use std::time::Duration; use std::time::Instant; use std::time::SystemTime; @@ -26,8 +28,7 @@ use benchmarks::report::JobReport; use benchmarks::report::RunReporter; use cache2::Cache; use cache2::CacheConfig; -use cache2::ErrorKind as CacheErrorKind; -use cache2::IoEngine; +use cache2::IoEngineConfig; use cache2::IoMode; use cache2::PosixIoConfig; use cache2::RuntimeOptions; @@ -91,7 +92,7 @@ impl ScaleConfig { fn runtime_options(&self) -> RuntimeOptions { RuntimeOptions { - io_engine: IoEngine::Posix(PosixIoConfig::new(1, 1, 1)), + io_engine: IoEngineConfig::Posix(PosixIoConfig::new(1, 1, 1)), io_mode: IoMode::Buffered, append_shards: 4, l1_capacity_bytes: self.memory_bytes, @@ -129,7 +130,7 @@ impl ScaleFiles { fn logical_bytes(&self) -> io::Result { self.paths() .into_iter() - .try_fold(0_u64, |total, path| match std::fs::metadata(path) { + .try_fold(0_u64, |total, path| match fs::metadata(path) { Ok(metadata) => total .checked_add(metadata.len()) .ok_or_else(|| invalid("logical file size overflow")), @@ -144,7 +145,7 @@ impl ScaleFiles { self.paths() .into_iter() - .try_fold(0_u64, |total, path| match std::fs::metadata(path) { + .try_fold(0_u64, |total, path| match fs::metadata(path) { Ok(metadata) => total .checked_add(metadata.blocks().saturating_mul(512)) .ok_or_else(|| invalid("allocated file size overflow")), @@ -178,7 +179,7 @@ impl Drop for ScaleFiles { return; } for path in self.paths() { - let _ = std::fs::remove_file(path); + let _ = fs::remove_file(path); } } } @@ -190,7 +191,7 @@ fn main() -> io::Result<()> { result .as_ref() .err() - .map(|error| error as &dyn std::fmt::Display), + .map(|error| error as &dyn fmt::Display), ); result } @@ -279,11 +280,7 @@ async fn run(config: ScaleConfig) -> io::Result<()> { Ok(()) } -async fn verify_sentinels( - cache: &cache2::Cache, - keys: &[[u8; 16]], - value_bytes: usize, -) -> io::Result<()> { +async fn verify_sentinels(cache: &Cache, keys: &[[u8; 16]], value_bytes: usize) -> io::Result<()> { let started = Instant::now(); for (ordinal, key) in keys.iter().enumerate() { let observed = cache @@ -307,19 +304,19 @@ async fn verify_sentinels( Ok(()) } -fn put_eventually(cache: &cache2::Cache, key: &[u8], value: &[u8]) -> io::Result<()> { +fn put_eventually(cache: &Cache, key: &[u8], value: &[u8]) -> io::Result<()> { let deadline = Instant::now() + WRITE_RETRY_TIMEOUT; loop { match cache.put(key, value) { Ok(_) => return Ok(()), - Err(error) if error.kind() == CacheErrorKind::Overloaded => { + Err(error) if error.kind() == cache2::ErrorKind::Overloaded => { if Instant::now() >= deadline { return Err(io::Error::new( io::ErrorKind::TimedOut, "recovery benchmark write did not enter bounded staging", )); } - thread::sleep(Duration::from_micros(50)); + std::thread::sleep(Duration::from_micros(50)); } Err(error) => return Err(error.into()), } @@ -357,7 +354,7 @@ fn emit(phase: &str, operation: &str, elapsed: Duration, operations: u64, bytes: #[cfg(unix)] fn peak_rss_bytes() -> u64 { - let mut usage = std::mem::MaybeUninit::::zeroed(); + let mut usage = MaybeUninit::::zeroed(); // SAFETY: `usage` points to writable storage for one `rusage` value. if unsafe { libc::getrusage(libc::RUSAGE_SELF, usage.as_mut_ptr()) } != 0 { return 0; @@ -383,7 +380,7 @@ fn peak_rss_bytes() -> u64 { #[cfg(target_os = "linux")] fn current_rss_bytes() -> u64 { - std::fs::read_to_string("/proc/self/status") + fs::read_to_string("/proc/self/status") .ok() .and_then(|status| { status.lines().find_map(|line| { diff --git a/benchmarks/region_index_turnover/main.rs b/benchmarks/region_index_turnover/main.rs index f108590..122abb3 100644 --- a/benchmarks/region_index_turnover/main.rs +++ b/benchmarks/region_index_turnover/main.rs @@ -13,6 +13,7 @@ // limitations under the License. use std::env; +use std::fmt; use std::io; use benchmarks::report::JobReport; @@ -28,7 +29,7 @@ fn main() -> io::Result<()> { result .as_ref() .err() - .map(|error| error as &dyn std::fmt::Display), + .map(|error| error as &dyn fmt::Display), ); result } diff --git a/benchmarks/src/report.rs b/benchmarks/src/report.rs index 4a07e63..810027d 100644 --- a/benchmarks/src/report.rs +++ b/benchmarks/src/report.rs @@ -14,7 +14,11 @@ //! Bounded benchmark measurements and fio-style reporting. +use std::array; +use std::env::consts::ARCH; +use std::env::consts::OS; use std::fmt; +use std::mem::MaybeUninit; use std::sync::atomic::AtomicU64; use std::sync::atomic::Ordering; use std::time::Duration; @@ -142,7 +146,7 @@ pub struct AtomicLatencyHistogram { impl Default for AtomicLatencyHistogram { fn default() -> Self { Self { - buckets: std::array::from_fn(|_| AtomicU64::new(0)), + buckets: array::from_fn(|_| AtomicU64::new(0)), minimum_ns: AtomicU64::new(u64::MAX), maximum_ns: AtomicU64::new(0), } @@ -160,7 +164,7 @@ impl AtomicLatencyHistogram { /// Takes a non-transactional snapshot suitable for periodic reporting. pub fn snapshot(&self) -> LatencyHistogram { - let buckets = std::array::from_fn(|index| self.buckets[index].load(Ordering::Relaxed)); + let buckets = array::from_fn(|index| self.buckets[index].load(Ordering::Relaxed)); let samples = buckets.iter().copied().sum(); LatencyHistogram { buckets, @@ -359,15 +363,11 @@ impl RunReporter { println!("C² benchmark report"); println!( " benchmark={}, scenario={}, os={}, arch={}", - benchmark, - scenario, - std::env::consts::OS, - std::env::consts::ARCH, + benchmark, scenario, OS, ARCH, ); println!( "report version=1 type=header benchmark={benchmark} scenario={scenario} os={} arch={}", - std::env::consts::OS, - std::env::consts::ARCH, + OS, ARCH, ); Self { benchmark, @@ -594,7 +594,7 @@ struct ProcessUsage { impl ProcessUsage { #[cfg(unix)] fn capture() -> Self { - let mut usage = std::mem::MaybeUninit::::zeroed(); + let mut usage = MaybeUninit::::zeroed(); // SAFETY: `usage` points to writable storage for one `rusage` value. if unsafe { libc::getrusage(libc::RUSAGE_SELF, usage.as_mut_ptr()) } != 0 { return Self::default(); diff --git a/cache2/ERRORS.md b/cache2/ERRORS.md index d05a166..c30512c 100644 --- a/cache2/ERRORS.md +++ b/cache2/ERRORS.md @@ -1,6 +1,6 @@ # Error handling -C² separates normal cache outcomes from failures. A miss, stale hit, L1 bypass, eviction, rejected recovery image, or transition to read miss-only mode is not an error. Public operations return `cache2::Result`, whose error type is `cache2::Error`. +C² separates normal cache outcomes from failures. A miss, stale hit, L1 bypass, eviction, rejected recovery image, or transition to read miss-only mode is not an error. Public operations return `Result`, where `Error` is the C² error type. An error has three independent pieces of information: @@ -15,9 +15,9 @@ Display text is intended for people and logs. Do not parse it or use it as a met `Overloaded` is an expected result of bounded admission. A lookaside-cache caller should normally continue through its authoritative data path or perform a bounded retry rather than fail the application request. ```rust -use cache2::{Cache, ErrorKind, Result}; +use cache2::{Cache, Error, ErrorKind}; -fn cache_value(cache: &Cache, key: &[u8], value: &[u8]) -> Result<()> { +fn cache_value(cache: &Cache, key: &[u8], value: &[u8]) -> Result<(), Error> { match cache.put(key, value) { Ok(_sequence) => Ok(()), Err(error) if error.kind() == ErrorKind::Overloaded => { diff --git a/cache2/src/benchmarking.rs b/cache2/src/benchmarking.rs index 063ef22..79399fd 100644 --- a/cache2/src/benchmarking.rs +++ b/cache2/src/benchmarking.rs @@ -20,18 +20,18 @@ use std::io; use std::time::Duration; use std::time::Instant; -use crate::index::IndexEntry; -use crate::index::MAX_INDEX_PROBES; -use crate::index::MAX_PACKED_REGION_COUNT; -use crate::index::MAX_REGION_OFFSET; -use crate::index::PackedLocation; -use crate::index_storage::PartitionedIndexStorage; -use crate::index_storage::validated_index_partition_ranges; -use crate::record_codec::hash_key; -use crate::region_index::BenchmarkProbeStats; -use crate::region_index::RegionIndex; -use crate::region_index::reset_benchmark_probe_stats; -use crate::region_index::take_benchmark_probe_stats; +use crate::region::index::BenchmarkProbeStats; +use crate::region::index::RegionIndex; +use crate::region::index::packed::IndexEntry; +use crate::region::index::packed::MAX_INDEX_PROBES; +use crate::region::index::packed::MAX_PACKED_REGION_COUNT; +use crate::region::index::packed::MAX_REGION_OFFSET; +use crate::region::index::packed::PackedLocation; +use crate::region::index::reset_benchmark_probe_stats; +use crate::region::index::storage::PartitionedIndexStorage; +use crate::region::index::storage::validated_index_partition_ranges; +use crate::region::index::take_benchmark_probe_stats; +use crate::region::record::codec::hash_key; use crate::snapshot::CacheIndexSnapshot; const BENCHMARK_HASH_SEED: u64 = 0x6a09_e667_f3bc_c909; diff --git a/cache2/src/cache.rs b/cache2/src/cache.rs index 7812c40..9baffaf 100644 --- a/cache2/src/cache.rs +++ b/cache2/src/cache.rs @@ -33,23 +33,25 @@ use std::time::Instant; use std::time::SystemTime; use std::time::UNIX_EPOCH; +use tokio::task::JoinError; + use crate::config::CacheConfig; -use crate::config::KEY_HASH_SEED; +use crate::config::storage::KEY_HASH_SEED; use crate::config::storage_fingerprint; use crate::config::storage_geometry; +use crate::error::Error; use crate::error::ErrorOperation; -use crate::error::Result; use crate::error::from_io; -use crate::recovery::DataSuperblock; -use crate::recovery::PersistentId; -use crate::recovery::RECOVERY_IMAGE_INDEX_OFFSET; -use crate::recovery::recovery_image_index_len; -use crate::region::FileRegionBackend; -use crate::region::RegionFiles; -use crate::region::SystemRegionFileSystem; -use crate::region_runtime::HybridValueRead; -use crate::region_runtime::RegionDataPlane; -use crate::region_store::RegionStore; +use crate::region::file_backend::FileRegionBackend; +use crate::region::file_backend::RegionFiles; +use crate::region::file_backend::SystemRegionFileSystem; +use crate::region::recovery::DataSuperblock; +use crate::region::recovery::PersistentId; +use crate::region::recovery::RECOVERY_IMAGE_INDEX_OFFSET; +use crate::region::recovery::recovery_image_index_len; +use crate::region::runtime::HybridValueRead; +use crate::region::runtime::RegionDataPlane; +use crate::region::store::RegionStore; use crate::snapshot::CacheSnapshot; use crate::snapshot::DetailedCacheSnapshot; use crate::snapshot::StartupMode; @@ -141,7 +143,7 @@ impl Cache { /// Returns [`ErrorOperation::Open`] for file locking, recovery, allocation, /// device support, runtime binding, or worker startup failures. Configuration /// has already been checked by [`CacheConfig::new`]. - pub async fn open(path: impl AsRef, config: CacheConfig) -> Result { + pub async fn open(path: impl AsRef, config: CacheConfig) -> Result { let handle = tokio::runtime::Handle::try_current().map_err(|error| { from_io( ErrorOperation::Open, @@ -162,7 +164,7 @@ impl Cache { path: impl AsRef, config: CacheConfig, tokio_handle: tokio::runtime::Handle, - ) -> Result { + ) -> Result { let path = path.as_ref().to_path_buf(); let cache_handle = tokio_handle.clone(); let started = Instant::now(); @@ -269,12 +271,12 @@ impl Cache { /// /// # Errors /// - /// Returns [`crate::ErrorKind::InvalidInput`] for an oversized key or - /// record and [`crate::ErrorKind::Overloaded`] when bounded mutation - /// admission is busy. Returns [`crate::ErrorKind::Unavailable`] after close - /// starts. Runtime and device failures use their corresponding structured + /// Returns [`ErrorKind::InvalidInput`](crate::ErrorKind::InvalidInput) for an oversized key or + /// record and [`ErrorKind::Overloaded`](crate::ErrorKind::Overloaded) when bounded mutation + /// admission is busy. Returns [`ErrorKind::Unavailable`](crate::ErrorKind::Unavailable) after + /// close starts. Runtime and device failures use their corresponding structured /// classifications. - pub fn put(&self, key: impl AsRef<[u8]>, value: impl AsRef<[u8]>) -> Result { + pub fn put(&self, key: impl AsRef<[u8]>, value: impl AsRef<[u8]>) -> Result { self.ensure_open(ErrorOperation::Put)?; public_result( ErrorOperation::Put, @@ -293,8 +295,8 @@ impl Cache { /// /// Uses the same input, overload, runtime, and device classifications as /// [`Self::put`], including unavailable after close starts, with - /// [`crate::ErrorOperation::PutL2`] as its context. - pub fn put_l2(&self, key: impl AsRef<[u8]>, value: impl AsRef<[u8]>) -> Result { + /// [`ErrorOperation::PutL2`](crate::ErrorOperation::PutL2) as its context. + pub fn put_l2(&self, key: impl AsRef<[u8]>, value: impl AsRef<[u8]>) -> Result { self.ensure_open(ErrorOperation::PutL2)?; public_result( ErrorOperation::PutL2, @@ -309,11 +311,11 @@ impl Cache { /// /// # Errors /// - /// Returns [`crate::ErrorKind::InvalidInput`] for an oversized key and - /// [`crate::ErrorKind::Overloaded`] when bounded mutation admission is - /// busy. Returns [`crate::ErrorKind::Unavailable`] after close starts. + /// Returns [`ErrorKind::InvalidInput`](crate::ErrorKind::InvalidInput) for an oversized key and + /// [`ErrorKind::Overloaded`](crate::ErrorKind::Overloaded) when bounded mutation admission is + /// busy. Returns [`ErrorKind::Unavailable`](crate::ErrorKind::Unavailable) after close starts. /// Runtime and device failures remain explicit. - pub fn delete(&self, key: impl AsRef<[u8]>) -> Result { + pub fn delete(&self, key: impl AsRef<[u8]>) -> Result { self.ensure_open(ErrorOperation::Delete)?; public_result(ErrorOperation::Delete, self.data_plane.delete(key.as_ref())) } @@ -329,11 +331,11 @@ impl Cache { /// /// # Errors /// - /// Returns [`crate::ErrorKind::Overloaded`] for explicit read pressure when - /// waiting is enabled. Cache data and device failures that can safely fail + /// Returns [`ErrorKind::Overloaded`](crate::ErrorKind::Overloaded) for explicit read pressure + /// when waiting is enabled. Cache data and device failures that can safely fail /// open transition reads to misses instead of surfacing an application /// error. - pub async fn get(&self, key: impl AsRef<[u8]> + Send) -> Result> { + pub async fn get(&self, key: impl AsRef<[u8]> + Send) -> Result, Error> { if self.is_closed() { return Ok(None); } @@ -352,10 +354,10 @@ impl Cache { /// /// # Errors /// - /// Returns [`crate::ErrorKind::Overloaded`] if another drain is active, or - /// [`crate::ErrorKind::Unavailable`] after close starts. Accepted work that - /// cannot complete returns a structured runtime/device failure. - pub async fn drain(&self) -> Result<()> { + /// Returns [`ErrorKind::Overloaded`](crate::ErrorKind::Overloaded) if another drain is active, + /// or [`ErrorKind::Unavailable`](crate::ErrorKind::Unavailable) after close starts. + /// Accepted work that cannot complete returns a structured runtime/device failure. + pub async fn drain(&self) -> Result<(), Error> { self.ensure_open(ErrorOperation::Drain)?; public_result(ErrorOperation::Drain, self.data_plane.drain_async().await) } @@ -366,9 +368,9 @@ impl Cache { /// /// # Errors /// - /// Returns [`crate::ErrorKind::Unavailable`] after close starts, or a + /// Returns [`ErrorKind::Unavailable`](crate::ErrorKind::Unavailable) after close starts, or a /// structured runtime failure if the snapshot cannot be read. - pub fn snapshot(&self) -> Result { + pub fn snapshot(&self) -> Result { self.ensure_open(ErrorOperation::Snapshot)?; let mut snapshot = public_result(ErrorOperation::Snapshot, self.data_plane.snapshot())?; snapshot.logical_disk_peak_bytes = self.logical_disk_peak_bytes; @@ -382,10 +384,10 @@ impl Cache { /// /// # Errors /// - /// Returns [`crate::ErrorKind::Unavailable`] after close starts, or a + /// Returns [`ErrorKind::Unavailable`](crate::ErrorKind::Unavailable) after close starts, or a /// structured runtime failure if any diagnostic partition cannot be /// sampled. - pub fn detailed_snapshot(&self) -> Result { + pub fn detailed_snapshot(&self) -> Result { self.ensure_open(ErrorOperation::DetailedSnapshot)?; let mut snapshot = public_result( ErrorOperation::DetailedSnapshot, @@ -402,10 +404,10 @@ impl Cache { /// /// # Errors /// - /// Returns [`crate::ErrorKind::Unavailable`] if close already started, or a - /// structured runtime, worker, or filesystem failure with - /// [`crate::ErrorOperation::CloseFast`]. - pub fn close_fast(&self) -> impl Future> + Send + 'static { + /// Returns [`ErrorKind::Unavailable`](crate::ErrorKind::Unavailable) if close already started, + /// or a structured runtime, worker, or filesystem failure with + /// [`ErrorOperation::CloseFast`](crate::ErrorOperation::CloseFast). + pub fn close_fast(&self) -> impl Future> + Send + 'static { self.close(false) } @@ -417,16 +419,16 @@ impl Cache { /// /// # Errors /// - /// Returns [`crate::ErrorKind::Unavailable`] if close already started, or a - /// structured runtime, worker, filesystem, or device failure with - /// [`crate::ErrorOperation::CloseWarm`]. A failed warm close does not - /// publish a recoverable image. - pub fn close_warm(&self) -> impl Future> + Send + 'static { + /// Returns [`ErrorKind::Unavailable`](crate::ErrorKind::Unavailable) if close already started, + /// or a structured runtime, worker, filesystem, or device failure with + /// [`ErrorOperation::CloseWarm`](crate::ErrorOperation::CloseWarm). A failed warm close does + /// not publish a recoverable image. + pub fn close_warm(&self) -> impl Future> + Send + 'static { self.close(true) } #[inline(always)] - fn ensure_open(&self, operation: ErrorOperation) -> Result<()> { + fn ensure_open(&self, operation: ErrorOperation) -> Result<(), Error> { if self.is_closed() { return public_result(operation, Err(cache_closed_error())); } @@ -438,7 +440,7 @@ impl Cache { self.closed.load(Ordering::Acquire) } - fn close(&self, warm: bool) -> impl Future> + Send + 'static { + fn close(&self, warm: bool) -> impl Future> + Send + 'static { let (operation, mode) = if warm { (ErrorOperation::CloseWarm, "warm") } else { @@ -554,7 +556,7 @@ fn log_cache_close(path: &Path, mode: &'static str, elapsed: Duration, result: & } } -fn blocking_task_error(operation: &'static str, error: tokio::task::JoinError) -> io::Error { +fn blocking_task_error(operation: &'static str, error: JoinError) -> io::Error { io::Error::other(format!("{operation} task failed: {error}")) } @@ -564,7 +566,7 @@ fn sidecar_path(path: &Path, suffix: &str) -> PathBuf { PathBuf::from(value) } -fn public_result(operation: ErrorOperation, result: io::Result) -> Result { +fn public_result(operation: ErrorOperation, result: io::Result) -> Result { result.map_err(|error| from_io(operation, error)) } diff --git a/cache2/src/checksum.rs b/cache2/src/checksum.rs index 40cdfbe..1bbb1d0 100644 --- a/cache2/src/checksum.rs +++ b/cache2/src/checksum.rs @@ -12,11 +12,11 @@ // See the License for the specific language governing permissions and // limitations under the License. -//! CRC32C (Castagnoli) used by the on-disk format. +//! CRC32C used by the on-disk format. //! //! The dependency selects hardware acceleration when the host supports it and //! retains a portable software fallback. This wrapper keeps the cache's codec -//! API and checksum values independent from that implementation detail. +//! API and checksum values independent of that implementation detail. use crc_fast::CrcAlgorithm; use crc_fast::Digest; @@ -27,8 +27,8 @@ pub fn crc32c(bytes: &[u8]) -> u32 { crc32_iscsi(bytes) } -/// Incremental CRC32C state, useful for checksumming a key and value without -/// first joining them in a temporary allocation. +/// Incremental CRC32C state, useful for checksum a key and value without first joining them in a +/// temporary allocation. pub struct Crc32c { digest: Digest, } @@ -57,7 +57,7 @@ impl Default for Crc32c { #[cfg(test)] mod tests { - use super::crc32c; + use crate::checksum::crc32c; #[test] fn matches_the_crc32c_check_value() { diff --git a/cache2/src/config.rs b/cache2/src/config/mod.rs similarity index 79% rename from cache2/src/config.rs rename to cache2/src/config/mod.rs index 63a4eb3..6d3cd3a 100644 --- a/cache2/src/config.rs +++ b/cache2/src/config/mod.rs @@ -14,31 +14,13 @@ //! Configuration construction, independent of file paths and runtime handles. -use crate::recovery::DataGeometry; - -mod runtime; -pub use self::runtime::IoEngine; -pub use self::runtime::IoMode; -pub use self::runtime::IoPoolTopology; -pub use self::runtime::IoUringConfig; -pub use self::runtime::IoUringPoolConfig; -pub use self::runtime::IoUringSqPollConfig; -pub use self::runtime::L1EvictionPolicy; -#[cfg(test)] -pub use self::runtime::MAX_WRITE_FLUSH_THRESHOLD_BYTES; -pub use self::runtime::PosixIoConfig; -pub use self::runtime::ReadAdmission; -pub use self::runtime::RuntimeOptions; -pub use self::runtime::read_io_wait_capacity; -pub use self::runtime::read_io_wait_timeout; - -mod storage; -pub use self::storage::KEY_HASH_SEED; -pub use self::storage::StorageOptions; -#[cfg(test)] -pub use self::storage::cache_config; - -/// Complete, immutable configuration for opening a [`crate::Cache`]. +use crate::config::runtime::RuntimeOptions; +use crate::region::recovery::DataGeometry; + +pub mod runtime; +pub mod storage; + +/// Complete, immutable configuration for opening a [`Cache`](crate::Cache). /// /// Construction checks runtime settings against the storage layout and managed /// memory limit. It performs bounded calculations without opening files, starting @@ -77,8 +59,8 @@ impl CacheConfig { /// Immutable persistent geometry with a checked logical disk bound. /// -/// Created by [`StorageOptions::build`]. Changing the geometry or index size -/// changes the disk identity, so an incompatible recovery image opens empty. +/// Created by [`StorageOptions::build`](crate::StorageOptions::build). Changing the geometry or +/// index size changes the disk identity, so an incompatible recovery image opens empty. /// Layout construction neither reserves disk space nor requires a Tokio runtime. #[derive(Clone, Debug, Eq, PartialEq)] pub struct StorageLayout { diff --git a/cache2/src/config/runtime.rs b/cache2/src/config/runtime.rs index 724cbd2..ce309af 100644 --- a/cache2/src/config/runtime.rs +++ b/cache2/src/config/runtime.rs @@ -15,18 +15,20 @@ use std::io; use std::time::Duration; -use super::CacheConfig; -use super::StorageLayout; +use crate::config::CacheConfig; +use crate::config::StorageLayout; +use crate::error::Error; use crate::error::ErrorOperation; -use crate::error::Result; use crate::error::from_io; -use crate::io_engine::IO_QUEUE_ENTRY_RESERVATION_BYTES; -use crate::io_engine::MAX_IO_REQUESTS_PER_ENGINE; -use crate::io_engine::io_uring_extra_memory_bytes; +use crate::io::engine::IO_QUEUE_ENTRY_RESERVATION_BYTES; +use crate::io::engine::MAX_IO_REQUESTS_PER_ENGINE; +use crate::io::engine::io_uring_extra_memory_bytes; use crate::memory::MemoryStore; -use crate::recovery::DataGeometry; -use crate::region_runtime::ActivityMetrics; -use crate::region_staging::RegionStaging; +use crate::region::recovery::DataGeometry; +use crate::region::runtime::metrics::ActivityMetrics; +use crate::region::runtime_fixed_memory_bytes; +use crate::region::staging::RegionStaging; +use crate::resources::BUFFER_ALIGNMENT; use crate::resources::CACHE_THREAD_STACK_BYTES; use crate::resources::MAX_CONFIG_COUNT; @@ -92,7 +94,7 @@ impl Default for PosixIoConfig { /// experimental io_uring engine. /// /// `max_in_flight` is distributed as evenly as possible across `rings`. This -/// keeps admission capacity independent from the number of driver threads. +/// keeps admission capacity independent of the number of driver threads. #[derive(Clone, Copy, Debug, Eq, PartialEq)] pub struct IoUringPoolConfig { rings: usize, @@ -244,7 +246,7 @@ impl Default for IoUringConfig { /// I/O pools. #[non_exhaustive] #[derive(Clone, Copy, Debug, Eq, PartialEq)] -pub enum IoEngine { +pub enum IoEngineConfig { /// Worker-backed POSIX positioned I/O with explicit thread counts. Posix(PosixIoConfig), /// Experimental Linux io_uring engine with independent ring and in-flight @@ -258,13 +260,13 @@ pub enum IoEngine { IoUring(IoUringConfig), } -impl Default for IoEngine { +impl Default for IoEngineConfig { fn default() -> Self { Self::Posix(PosixIoConfig::default()) } } -impl IoEngine { +impl IoEngineConfig { const fn is_available(self) -> bool { match self { Self::Posix(_) => true, @@ -292,24 +294,24 @@ pub struct IoPoolTopology { } impl IoPoolTopology { - pub const fn read(engine: IoEngine) -> Self { + pub const fn read(engine: IoEngineConfig) -> Self { match engine { - IoEngine::Posix(config) => Self::posix(config.read_workers), - IoEngine::IoUring(config) => Self::io_uring(config.read), + IoEngineConfig::Posix(config) => Self::posix(config.read_workers), + IoEngineConfig::IoUring(config) => Self::io_uring(config.read), } } - pub const fn write(engine: IoEngine) -> Self { + pub const fn write(engine: IoEngineConfig) -> Self { match engine { - IoEngine::Posix(config) => Self::posix(config.write_workers), - IoEngine::IoUring(config) => Self::io_uring(config.write), + IoEngineConfig::Posix(config) => Self::posix(config.write_workers), + IoEngineConfig::IoUring(config) => Self::io_uring(config.write), } } - pub const fn reclaim(engine: IoEngine) -> Self { + pub const fn reclaim(engine: IoEngineConfig) -> Self { match engine { - IoEngine::Posix(config) => Self::posix(config.reclaim_workers), - IoEngine::IoUring(config) => Self::io_uring(config.reclaim), + IoEngineConfig::Posix(config) => Self::posix(config.reclaim_workers), + IoEngineConfig::IoUring(config) => Self::io_uring(config.reclaim), } } @@ -387,12 +389,12 @@ pub enum ReadAdmission { /// Maximum wait, greater than zero and no longer than five seconds. timeout: Duration, /// Maximum queued readers, from one through 65536. `None` follows the - /// aggregate read in-flight limit when [`super::CacheConfig`] is built. + /// aggregate read in-flight limit when [`CacheConfig`] is built. max_waiters: Option, }, } -/// Process-local resource choices, checked together by [`super::CacheConfig::new`]. +/// Process-local resource choices, checked together by [`CacheConfig::new`]. /// /// These values may change across opens. Warm recovery rebinds append shards /// from recovered Active and Free Regions when the requested topology fits. @@ -402,7 +404,7 @@ pub enum ReadAdmission { pub struct RuntimeOptions { /// Independent read, write, and reclaim pools. Defaults to POSIX with 4, 4, /// and 1 workers. - pub io_engine: IoEngine, + pub io_engine: IoEngineConfig, /// Record I/O mode. Defaults to buffered; direct I/O requires supported Linux storage. pub io_mode: IoMode, /// Admission policy after an L2 candidate has been selected. @@ -436,7 +438,7 @@ pub struct RuntimeOptions { impl Default for RuntimeOptions { fn default() -> Self { Self { - io_engine: IoEngine::default(), + io_engine: IoEngineConfig::default(), io_mode: IoMode::Buffered, read_admission: ReadAdmission::Immediate, append_shards: DEFAULT_APPEND_SHARDS, @@ -479,7 +481,7 @@ impl CacheConfig { /// Checks the complete combination and resolves dependent runtime defaults. /// /// ```no_run - /// # async fn example() -> cache2::Result<()> { + /// # async fn example() -> Result<(), cache2::Error> { /// use cache2::Cache; /// use cache2::CacheConfig; /// use cache2::RuntimeOptions; @@ -499,7 +501,7 @@ impl CacheConfig { /// Returns [`ErrorOperation::BuildConfig`] for incompatible Region/shard /// counts, invalid runtime settings, unavailable build/platform features, /// or insufficient managed memory. Device capabilities are checked at open. - pub fn new(storage: StorageLayout, mut runtime: RuntimeOptions) -> Result { + pub fn new(storage: StorageLayout, mut runtime: RuntimeOptions) -> Result { let build = || -> io::Result { let geometry = storage.geometry; let index_slots = storage.index_slots; @@ -516,10 +518,9 @@ impl CacheConfig { runtime.l1_shards, runtime.l1_eviction_policy, )?; - let fixed_bytes = - crate::region::runtime_fixed_memory_bytes(index_slots, geometry.region_count)? - .checked_add(l1_metadata_bytes) - .ok_or_else(|| invalid_config("fixed memory requirements overflow"))?; + let fixed_bytes = runtime_fixed_memory_bytes(index_slots, geometry.region_count)? + .checked_add(l1_metadata_bytes) + .ok_or_else(|| invalid_config("fixed memory requirements overflow"))?; let (reserved_memory_bytes, minimum_memory_bytes) = runtime.memory_requirements(geometry, fixed_bytes)?; if minimum_memory_bytes > runtime.managed_memory_limit_bytes { @@ -571,12 +572,12 @@ impl RuntimeOptions { let write_topology = IoPoolTopology::write(self.io_engine); let reclaim_topology = IoPoolTopology::reclaim(self.io_engine); match self.io_engine { - IoEngine::Posix(_) => { + IoEngineConfig::Posix(_) => { validate_posix_pool("read", read_topology)?; validate_posix_pool("write", write_topology)?; validate_posix_pool("reclaim", reclaim_topology)?; } - IoEngine::IoUring(config) => { + IoEngineConfig::IoUring(config) => { validate_io_uring_pool("read", config.read())?; validate_io_uring_pool("write", config.write())?; validate_io_uring_pool("reclaim", config.reclaim())?; @@ -638,7 +639,7 @@ impl RuntimeOptions { || self.write_flush_threshold_bytes > MAX_WRITE_FLUSH_THRESHOLD_BYTES || !self .write_flush_threshold_bytes - .is_multiple_of(crate::resources::BUFFER_ALIGNMENT) + .is_multiple_of(BUFFER_ALIGNMENT) { return Err(io::Error::new( io::ErrorKind::InvalidInput, @@ -748,7 +749,7 @@ fn runtime_topology_memory_bytes(config: &RuntimeOptions) -> Option { .checked_add(config.l1_shards)? .checked_add(reclaim.max_in_flight)? .checked_mul(RUNTIME_CONTROL_RESERVATION_BYTES)?; - let metrics = shard_count.checked_mul(std::mem::size_of::())?; + let metrics = shard_count.checked_mul(size_of::())?; stacks .checked_add(queue)? .checked_add(uring)? @@ -805,7 +806,7 @@ mod tests { #[test] fn optional_read_wait_queue_is_memory_accounted() { let base = RuntimeOptions { - io_engine: crate::config::IoEngine::Posix(crate::config::PosixIoConfig::new(7, 4, 1)), + io_engine: IoEngineConfig::Posix(PosixIoConfig::new(7, 4, 1)), ..RuntimeOptions::default() }; let no_wait = runtime_topology_memory_bytes(&base).unwrap(); @@ -835,7 +836,7 @@ mod tests { }; let (_, base_minimum) = base.memory_requirements(geometry, 0).unwrap(); let (_, parallel_minimum) = RuntimeOptions { - io_engine: crate::config::IoEngine::Posix(crate::config::PosixIoConfig::new(4, 4, 2)), + io_engine: IoEngineConfig::Posix(PosixIoConfig::new(4, 4, 2)), ..base } .memory_requirements(geometry, 0) @@ -852,7 +853,7 @@ mod tests { #[test] fn io_engine_topology_matches_backend_shape() { - let posix = IoEngine::Posix(PosixIoConfig::new(7, 5, 2)); + let posix = IoEngineConfig::Posix(PosixIoConfig::new(7, 5, 2)); assert_eq!( IoPoolTopology::read(posix), IoPoolTopology { @@ -863,7 +864,7 @@ mod tests { } ); - let io_uring = IoEngine::IoUring(IoUringConfig::new( + let io_uring = IoEngineConfig::IoUring(IoUringConfig::new( IoUringPoolConfig::new(3, 8), IoUringPoolConfig::new(2, 5), IoUringPoolConfig::new(1, 2), @@ -881,11 +882,11 @@ mod tests { fn io_uring_depth_reserves_more_than_common_request_bookkeeping() { let pool = IoUringPoolConfig::new(1, 1); let shallow = RuntimeOptions { - io_engine: IoEngine::IoUring(IoUringConfig::new(pool, pool, pool)), + io_engine: IoEngineConfig::IoUring(IoUringConfig::new(pool, pool, pool)), ..RuntimeOptions::default() }; let deep = RuntimeOptions { - io_engine: IoEngine::IoUring(IoUringConfig::new( + io_engine: IoEngineConfig::IoUring(IoUringConfig::new( IoUringPoolConfig::new(1, MAX_IO_REQUESTS_PER_ENGINE), pool, pool, @@ -929,7 +930,7 @@ mod tests { fn io_poll_requires_direct_mode() { let pool = IoUringPoolConfig::default().with_io_poll(true); let mut config = RuntimeOptions { - io_engine: IoEngine::IoUring(crate::config::IoUringConfig::new( + io_engine: IoEngineConfig::IoUring(IoUringConfig::new( pool, IoUringPoolConfig::default(), IoUringPoolConfig::new(1, 1), diff --git a/cache2/src/config/storage.rs b/cache2/src/config/storage.rs index c8b13cb..804b640 100644 --- a/cache2/src/config/storage.rs +++ b/cache2/src/config/storage.rs @@ -17,25 +17,25 @@ use std::io; #[cfg(test)] -use super::CacheConfig; +use crate::config::CacheConfig; +use crate::config::StorageLayout; #[cfg(test)] -use super::RuntimeOptions; -use super::StorageLayout; +use crate::config::runtime::RuntimeOptions; +use crate::error::Error; use crate::error::ErrorOperation; -use crate::error::Result; use crate::error::from_io; -use crate::index::MAX_PACKED_REGION_COUNT; -use crate::index::MAX_PACKED_REGION_SIZE; -use crate::index_storage::IndexStorageError; -use crate::index_storage::validated_index_partition_ranges; -use crate::recovery::DataGeometry; -use crate::recovery::KEY_HASH_ALGORITHM_XXH3_64; -use crate::recovery::RECOVERY_IMAGE_INDEX_OFFSET; -use crate::recovery::STATE_FILE_SIZE; -use crate::recovery::recovery_image_index_len; -use crate::region_metadata::REGION_METADATA_PAGE_SIZE; -use crate::region_metadata::REGION_METADATA_PARTITIONS_PER_PAGE; -use crate::region_metadata::REGION_METADATA_REGIONS_PER_PAGE; +use crate::region::index::packed::MAX_PACKED_REGION_COUNT; +use crate::region::index::packed::MAX_PACKED_REGION_SIZE; +use crate::region::index::storage::IndexStorageError; +use crate::region::index::storage::validated_index_partition_ranges; +use crate::region::recovery::DataGeometry; +use crate::region::recovery::KEY_HASH_ALGORITHM_XXH3_64; +use crate::region::recovery::RECOVERY_IMAGE_INDEX_OFFSET; +use crate::region::recovery::STATE_FILE_SIZE; +use crate::region::recovery::metadata::REGION_METADATA_PAGE_SIZE; +use crate::region::recovery::metadata::REGION_METADATA_PARTITIONS_PER_PAGE; +use crate::region::recovery::metadata::REGION_METADATA_REGIONS_PER_PAGE; +use crate::region::recovery::recovery_image_index_len; const DEFAULT_REGION_SIZE: u64 = 32 * 1024 * 1024; const DEFAULT_EXPECTED_ENTRY_BYTES: u64 = 16 * 1024; @@ -70,14 +70,14 @@ impl StorageOptions { /// Checks the inputs and computes an immutable layout without opening files. /// Use [`StorageLayout::peak_disk_bytes`] to compare a candidate with a disk - /// budget, then pass the chosen layout to [`super::CacheConfig::new`]. + /// budget, then pass the chosen layout to [`CacheConfig::new`](crate::CacheConfig::new). /// /// # Errors /// /// Returns [`ErrorOperation::BuildStorage`] if the geometry or index cannot /// be represented, disk accounting overflows, or bounded layout allocation /// fails. - pub fn build(self) -> Result { + pub fn build(self) -> Result { let build = || { let entries = match self.expected_entries { Some(entries) => entries, @@ -209,8 +209,8 @@ pub fn cache_config( #[cfg(test)] mod tests { use super::*; - use crate::recovery::DataSuperblock; - use crate::recovery::PersistentId; + use crate::region::recovery::DataSuperblock; + use crate::region::recovery::PersistentId; #[test] fn constructed_layouts_encode_at_format_boundaries() { diff --git a/cache2/src/error.rs b/cache2/src/error.rs index 1244818..f8a7641 100644 --- a/cache2/src/error.rs +++ b/cache2/src/error.rs @@ -12,13 +12,9 @@ // See the License for the specific language governing permissions and // limitations under the License. -use std::error::Error as StdError; use std::fmt; use std::io; -/// A result returned by a public C² operation. -pub type Result = std::result::Result; - /// Stable, actionable classification for a C² failure. /// /// Match this value instead of parsing [`Error`]'s display text or branching @@ -74,29 +70,30 @@ impl fmt::Display for ErrorKind { #[non_exhaustive] #[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)] pub enum ErrorOperation { - /// [`crate::CacheConfig::new`]. + /// [`CacheConfig::new`](crate::CacheConfig::new). BuildConfig, - /// [`crate::StorageOptions::build`]. + /// [`StorageOptions::build`](crate::StorageOptions::build). BuildStorage, - /// [`crate::Cache::open`] or [`crate::Cache::open_with_handle`]. + /// [`Cache::open`](crate::Cache::open) or + /// [`Cache::open_with_handle`](crate::Cache::open_with_handle). Open, - /// [`crate::Cache::put`]. + /// [`Cache::put`](crate::Cache::put). Put, - /// [`crate::Cache::put_l2`]. + /// [`Cache::put_l2`](crate::Cache::put_l2). PutL2, - /// [`crate::Cache::delete`]. + /// [`Cache::delete`](crate::Cache::delete). Delete, - /// [`crate::Cache::get`]. + /// [`Cache::get`](crate::Cache::get). Get, - /// [`crate::Cache::drain`]. + /// [`Cache::drain`](crate::Cache::drain). Drain, - /// [`crate::Cache::snapshot`]. + /// [`Cache::snapshot`](crate::Cache::snapshot). Snapshot, - /// [`crate::Cache::detailed_snapshot`]. + /// [`Cache::detailed_snapshot`](crate::Cache::detailed_snapshot). DetailedSnapshot, - /// [`crate::Cache::close_fast`]. + /// [`Cache::close_fast`](crate::Cache::close_fast). CloseFast, - /// [`crate::Cache::close_warm`]. + /// [`Cache::close_warm`](crate::Cache::close_warm). CloseWarm, } @@ -130,7 +127,7 @@ impl fmt::Display for ErrorOperation { /// /// The classification and operation are stable programmatic fields. The /// wrapped [`io::Error`] retains the detailed cause, its raw OS error when one -/// exists, and the complete [`StdError::source`] chain. +/// exists, and the complete [`Error::source`](std::error::Error::source) chain. #[doc = include_str!("../ERRORS.md")] #[derive(Debug)] pub struct Error { @@ -199,8 +196,8 @@ impl fmt::Display for Error { } } -impl StdError for Error { - fn source(&self) -> Option<&(dyn StdError + 'static)> { +impl std::error::Error for Error { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { Some(&self.source) } } @@ -220,26 +217,28 @@ pub fn from_io(operation: ErrorOperation, source: io::Error) -> Error { } fn classify(operation: ErrorOperation, source: &io::Error) -> ErrorKind { - use io::ErrorKind as IoKind; - match source.kind() { - IoKind::InvalidInput if source.raw_os_error().is_some() => ErrorKind::Io, - IoKind::InvalidInput if accepts_caller_input(operation) => ErrorKind::InvalidInput, - IoKind::InvalidInput => ErrorKind::Internal, - IoKind::Unsupported => ErrorKind::Unsupported, - IoKind::WouldBlock | IoKind::AlreadyExists if operation == ErrorOperation::Open => { + io::ErrorKind::InvalidInput if source.raw_os_error().is_some() => ErrorKind::Io, + io::ErrorKind::InvalidInput if accepts_caller_input(operation) => ErrorKind::InvalidInput, + io::ErrorKind::InvalidInput => ErrorKind::Internal, + io::ErrorKind::Unsupported => ErrorKind::Unsupported, + io::ErrorKind::WouldBlock | io::ErrorKind::AlreadyExists + if operation == ErrorOperation::Open => + { ErrorKind::Busy } - IoKind::WouldBlock if source.raw_os_error().is_some() => ErrorKind::Io, - IoKind::WouldBlock if has_bounded_admission(operation) => ErrorKind::Overloaded, - IoKind::WouldBlock | IoKind::AlreadyExists => ErrorKind::Internal, - IoKind::TimedOut if operation == ErrorOperation::Get => ErrorKind::Overloaded, - IoKind::TimedOut => ErrorKind::Io, - IoKind::OutOfMemory if operation == ErrorOperation::Get => ErrorKind::Overloaded, - IoKind::OutOfMemory => ErrorKind::ResourceExhausted, - IoKind::BrokenPipe | IoKind::NotConnected | IoKind::Interrupted => ErrorKind::Unavailable, - IoKind::InvalidData | IoKind::UnexpectedEof => ErrorKind::CorruptData, - IoKind::Other if source.raw_os_error().is_none() => ErrorKind::Internal, + io::ErrorKind::WouldBlock if source.raw_os_error().is_some() => ErrorKind::Io, + io::ErrorKind::WouldBlock if has_bounded_admission(operation) => ErrorKind::Overloaded, + io::ErrorKind::WouldBlock | io::ErrorKind::AlreadyExists => ErrorKind::Internal, + io::ErrorKind::TimedOut if operation == ErrorOperation::Get => ErrorKind::Overloaded, + io::ErrorKind::TimedOut => ErrorKind::Io, + io::ErrorKind::OutOfMemory if operation == ErrorOperation::Get => ErrorKind::Overloaded, + io::ErrorKind::OutOfMemory => ErrorKind::ResourceExhausted, + io::ErrorKind::BrokenPipe | io::ErrorKind::NotConnected | io::ErrorKind::Interrupted => { + ErrorKind::Unavailable + } + io::ErrorKind::InvalidData | io::ErrorKind::UnexpectedEof => ErrorKind::CorruptData, + io::ErrorKind::Other if source.raw_os_error().is_none() => ErrorKind::Internal, _ => ErrorKind::Io, } } diff --git a/cache2/src/fixtures/mod.rs b/cache2/src/fixtures.rs similarity index 84% rename from cache2/src/fixtures/mod.rs rename to cache2/src/fixtures.rs index fb5fef5..3839388 100644 --- a/cache2/src/fixtures/mod.rs +++ b/cache2/src/fixtures.rs @@ -12,7 +12,14 @@ // See the License for the specific language governing permissions and // limitations under the License. -//! Versioned byte fixtures for private persistent-format tests. +//! Shared byte assertions for module-local persistent-format fixtures. +//! +//! Golden fixtures pin versioned on-disk bytes. Changes require an explicit format-version +//! decision; tests never regenerate them. Each fixture lives beside the module that owns its +//! format. +//! +//! The sparse representation starts with the complete byte length. Each following line contains a +//! hexadecimal offset and hexadecimal bytes; unspecified bytes are zero. /// Checks every byte, including zero padding, and returns the committed bytes /// for decoder compatibility checks. diff --git a/cache2/src/fixtures/format_v1/README.md b/cache2/src/fixtures/format_v1/README.md deleted file mode 100644 index b4fbce5..0000000 --- a/cache2/src/fixtures/format_v1/README.md +++ /dev/null @@ -1,5 +0,0 @@ -# Format 1 golden fixtures - -These fixtures pin the version 1 on-disk bytes. Changes require an explicit format-version decision; tests never regenerate them. - -The sparse representation starts with the complete byte length. Each following line contains a hexadecimal offset and hexadecimal bytes; unspecified bytes are zero. diff --git a/cache2/src/hashing.rs b/cache2/src/hashing.rs index 290d62e..d75b1df 100644 --- a/cache2/src/hashing.rs +++ b/cache2/src/hashing.rs @@ -61,7 +61,7 @@ impl FixedPrehashedMap { pub fn allocation_bytes(maximum_entries: usize) -> io::Result { Self::slot_count(maximum_entries)? - .checked_mul(std::mem::size_of::()) + .checked_mul(size_of::()) .ok_or_else(|| io::Error::new(io::ErrorKind::InvalidInput, "fixed map is too large")) } @@ -237,7 +237,7 @@ mod tests { #[test] fn fixed_map_slot_is_one_u64() { - assert_eq!(std::mem::size_of::(), 8); + assert_eq!(size_of::(), 8); } #[test] @@ -247,7 +247,7 @@ mod tests { assert_eq!(FixedPrehashedMap::slot_count(40_960).unwrap(), 81_920); assert_eq!( FixedPrehashedMap::allocation_bytes(40_960).unwrap(), - 40_960 * 2 * std::mem::size_of::() + 40_960 * 2 * size_of::() ); } diff --git a/cache2/src/io_backend.rs b/cache2/src/io/backend.rs similarity index 99% rename from cache2/src/io_backend.rs rename to cache2/src/io/backend.rs index 10ab216..c8eae6a 100644 --- a/cache2/src/io_backend.rs +++ b/cache2/src/io/backend.rs @@ -31,12 +31,13 @@ use std::os::unix::fs::MetadataExt; #[cfg(unix)] use std::os::unix::fs::OpenOptionsExt; use std::path::Path; +use std::slice; use std::sync::Arc; use std::sync::atomic::AtomicBool; use std::sync::atomic::AtomicU64; use std::sync::atomic::Ordering; -use crate::config::IoMode; +use crate::config::runtime::IoMode; use crate::snapshot::CacheIoPathSnapshot; pub const DIRECT_IO_ALIGNMENT: usize = 4096; @@ -346,7 +347,7 @@ pub trait IoBackend: Send + Sync { // before constructing the mutable slice required by `read_at`. unsafe { buffer.write_bytes(0, length); - self.read_at(std::slice::from_raw_parts_mut(buffer, length), offset) + self.read_at(slice::from_raw_parts_mut(buffer, length), offset) } } fn write_at(&self, point: WritePoint, buffer: &[u8], offset: u64) -> io::Result; @@ -965,6 +966,7 @@ unsafe extern "C" { #[cfg(test)] mod tests { + use std::env; use std::path::PathBuf; use std::sync::atomic::AtomicU64; use std::sync::atomic::AtomicUsize; @@ -979,7 +981,7 @@ mod tests { impl TestFile { fn new(label: &str) -> Self { let nonce = NEXT_PATH.fetch_add(1, Ordering::Relaxed); - Self(std::env::temp_dir().join(format!( + Self(env::temp_dir().join(format!( "cache2-{label}-{}-{nonce}.cache", std::process::id() ))) @@ -1302,10 +1304,10 @@ mod tests { #[test] fn one_fault_handle_controls_multiple_recovery_files() { - use super::testing::FaultAction; - use super::testing::FaultBackend; - use super::testing::FaultEvent; - use super::testing::FaultHandle; + use crate::io::backend::testing::FaultAction; + use crate::io::backend::testing::FaultBackend; + use crate::io::backend::testing::FaultEvent; + use crate::io::backend::testing::FaultHandle; let state = TestFile::new("shared-fault-state"); let image = TestFile::new("shared-fault-image"); @@ -1348,10 +1350,12 @@ mod tests { #[test] fn cache_open_rejects_symbolic_links() { + use std::os::unix::fs::symlink; + let target = TestFile::new("symlink-target"); let link = TestFile::new("symlink-link"); drop(target.open()); - std::os::unix::fs::symlink(&target.0, &link.0).unwrap(); + symlink(&target.0, &link.0).unwrap(); assert!(FileBackend::open(&link.0).is_err()); } diff --git a/cache2/src/io_engine.rs b/cache2/src/io/engine/mod.rs similarity index 97% rename from cache2/src/io_engine.rs rename to cache2/src/io/engine/mod.rs index 1c4fa81..07ad2d8 100644 --- a/cache2/src/io_engine.rs +++ b/cache2/src/io/engine/mod.rs @@ -32,10 +32,8 @@ use std::sync::atomic::AtomicBool; use std::sync::atomic::AtomicU64; use std::sync::atomic::AtomicUsize; use std::sync::atomic::Ordering; -use std::sync::mpsc::Receiver; use std::sync::mpsc::SyncSender; use std::sync::mpsc::TrySendError; -use std::sync::mpsc::{self}; use std::task::Context; use std::task::Poll; use std::task::Waker; @@ -46,27 +44,12 @@ use std::time::Instant; use asyncband::semaphore::OwnedSemaphorePermit; use asyncband::semaphore::Semaphore; +use crate::IoEngineConfig; #[cfg(unix)] -use crate::config::IoEngine as ConfiguredIoEngine; +use crate::config::runtime::IoUringPoolConfig; +use crate::io::backend::IoBackend; #[cfg(unix)] -use crate::config::IoUringPoolConfig; -use crate::io_backend::IoBackend; -#[cfg(unix)] -use crate::io_backend::RuntimeFileBackend; -#[cfg(unix)] -use crate::io_backend::RuntimeFileSet; -#[cfg(all( - feature = "io-uring", - target_os = "linux", - any( - target_arch = "x86_64", - target_arch = "aarch64", - target_arch = "riscv64", - target_arch = "loongarch64", - target_arch = "powerpc64" - ) -))] -use crate::io_backend::RuntimeIoDirection; +use crate::io::backend::RuntimeFileSet; #[cfg(all( feature = "io-uring", target_os = "linux", @@ -78,8 +61,7 @@ use crate::io_backend::RuntimeIoDirection; target_arch = "powerpc64" ) ))] -use crate::io_backend::RuntimeIoPath; -use crate::io_backend::RuntimeIoStats; +use crate::io::backend::RuntimeIoDirection; #[cfg(all( feature = "io-uring", target_os = "linux", @@ -91,12 +73,10 @@ use crate::io_backend::RuntimeIoStats; target_arch = "powerpc64" ) ))] -use crate::io_backend::RuntimeIoStatsHandle; -use crate::io_backend::WritePoint; -use crate::io_backend::read_exact_at_uninit_with_progress; -use crate::io_backend::write_all_at_with_progress; +use crate::io::backend::RuntimeIoPath; +use crate::io::backend::RuntimeIoStats; +use crate::io::backend::WritePoint; use crate::resources::BufferLease; -use crate::resources::CACHE_THREAD_STACK_BYTES; use crate::snapshot::CacheIoDirectionSnapshot; mod posix; @@ -113,18 +93,6 @@ mod posix; ) ))] mod uring; -#[cfg(all( - feature = "io-uring", - target_os = "linux", - any( - target_arch = "x86_64", - target_arch = "aarch64", - target_arch = "riscv64", - target_arch = "loongarch64", - target_arch = "powerpc64" - ) -))] -pub use self::uring::UringIoEngine; /// Reference engine: a small fixed worker pool executes exact operations /// through the existing fault-injectable positioned-I/O backend. @@ -1982,13 +1950,13 @@ pub fn build_file_engine( files: RuntimeFileSet, max_in_flight: usize, posix_workers: usize, - kind: ConfiguredIoEngine, + kind: IoEngineConfig, io_uring_config: Option, statistics_enabled: bool, read_wait_enabled: bool, ) -> io::Result> { match kind { - ConfiguredIoEngine::Posix(_) => BackendIoEngine::new_with_files_and_workers( + IoEngineConfig::Posix(_) => BackendIoEngine::new_with_files_and_workers( files, max_in_flight, posix_workers, @@ -1996,7 +1964,7 @@ pub fn build_file_engine( read_wait_enabled, ) .map(|engine| Arc::new(engine) as Arc), - ConfiguredIoEngine::IoUring(_) => { + IoEngineConfig::IoUring(_) => { let _ = posix_workers; #[cfg(all( feature = "io-uring", @@ -2016,7 +1984,7 @@ pub fn build_file_engine( "io_uring pool configuration is missing", ) })?; - UringIoEngine::new_with_files( + uring::UringIoEngine::new_with_files( files, max_in_flight, io_uring_config, @@ -2052,7 +2020,7 @@ fn update_peak(peak: &AtomicUsize, value: usize) { peak.fetch_max(value, Ordering::Relaxed); } -fn add_duration_ns(counter: &AtomicU64, duration: std::time::Duration) { +fn add_duration_ns(counter: &AtomicU64, duration: Duration) { const MAX_DURATION_CAS_ATTEMPTS: usize = 8; let nanos = duration.as_nanos().min(u128::from(u64::MAX)) as u64; let mut current = counter.load(Ordering::Relaxed); diff --git a/cache2/src/io_engine/posix.rs b/cache2/src/io/engine/posix.rs similarity index 84% rename from cache2/src/io_engine/posix.rs rename to cache2/src/io/engine/posix.rs index 10379dc..eec286e 100644 --- a/cache2/src/io_engine/posix.rs +++ b/cache2/src/io/engine/posix.rs @@ -12,7 +12,46 @@ // See the License for the specific language governing permissions and // limitations under the License. -use super::*; +use std::io; +use std::panic; +use std::panic::AssertUnwindSafe; +use std::sync::Arc; +use std::sync::Condvar; +use std::sync::Mutex; +use std::sync::RwLock; +use std::sync::atomic::AtomicBool; +use std::sync::atomic::AtomicU64; +use std::sync::atomic::Ordering; +use std::sync::mpsc; +use std::sync::mpsc::Receiver; +use std::time::Instant; + +use crate::io::backend::IoBackend; +#[cfg(unix)] +use crate::io::backend::RuntimeFileBackend; +#[cfg(unix)] +use crate::io::backend::RuntimeFileSet; +use crate::io::backend::read_exact_at_uninit_with_progress; +use crate::io::backend::write_all_at_with_progress; +use crate::io::engine::BackendIoEngine; +use crate::io::engine::CompletionState; +use crate::io::engine::CompletionStatus; +use crate::io::engine::DriverCommand; +use crate::io::engine::EngineIoSnapshot; +use crate::io::engine::IoEngine; +use crate::io::engine::IoOperation; +use crate::io::engine::IoRequest; +use crate::io::engine::ReadSlot; +use crate::io::engine::ReadSlotWaiter; +use crate::io::engine::RequestId; +use crate::io::engine::RuntimeInner; +use crate::io::engine::RuntimeShared; +use crate::io::engine::ShutdownPhase; +use crate::io::engine::ShutdownState; +use crate::io::engine::SubmitError; +use crate::io::engine::SubmitState; +use crate::io::engine::lock_unpoisoned; +use crate::resources::CACHE_THREAD_STACK_BYTES; impl BackendIoEngine { #[cfg(unix)] @@ -233,16 +272,15 @@ fn backend_driver( shared.finish(task, CompletionStatus::Cancelled, 0); continue; } - let (status, transferred) = - std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { - execute_backend(backend.as_ref(), &mut task.operation) - })) - .unwrap_or_else(|_| { - ( - CompletionStatus::Failed(io::Error::other("I/O backend panicked")), - 0, - ) - }); + let (status, transferred) = panic::catch_unwind(AssertUnwindSafe(|| { + execute_backend(backend.as_ref(), &mut task.operation) + })) + .unwrap_or_else(|_| { + ( + CompletionStatus::Failed(io::Error::other("I/O backend panicked")), + 0, + ) + }); shared.finish(task, status, transferred); } DriverCommand::Cancel(request_id) => { diff --git a/cache2/src/io_engine/tests.rs b/cache2/src/io/engine/tests.rs similarity index 98% rename from cache2/src/io_engine/tests.rs rename to cache2/src/io/engine/tests.rs index 771e3fd..a0fa67c 100644 --- a/cache2/src/io_engine/tests.rs +++ b/cache2/src/io/engine/tests.rs @@ -12,16 +12,20 @@ // See the License for the specific language governing permissions and // limitations under the License. +use std::env; +use std::fs; use std::fs::File; use std::fs::OpenOptions; use std::path::PathBuf; use std::sync::atomic::AtomicU64; +use std::sync::mpsc; use std::time::Duration; use super::*; -use crate::io_backend::FileBackend; -use crate::io_backend::SyncMode; -use crate::io_backend::SyncPoint; +use crate::config::runtime::PosixIoConfig; +use crate::io::backend::FileBackend; +use crate::io::backend::SyncMode; +use crate::io::backend::SyncPoint; use crate::resources::ResourceController; use crate::resources::ResourceLimits; use crate::resources::aligned_buffer_capacity; @@ -74,7 +78,7 @@ impl TestFile { fn new() -> Self { let id = FILE_ID.fetch_add(1, Ordering::Relaxed); let path = - std::env::temp_dir().join(format!("cache2-io-engine-{}-{id}.bin", std::process::id())); + env::temp_dir().join(format!("cache2-io-engine-{}-{id}.bin", std::process::id())); Self { path } } @@ -95,7 +99,7 @@ impl TestFile { impl Drop for TestFile { fn drop(&mut self) { - let _ = std::fs::remove_file(&self.path); + let _ = fs::remove_file(&self.path); } } @@ -760,7 +764,7 @@ fn configured_posix_engine_shares_its_worker_capacity() { files, 4, 4, - ConfiguredIoEngine::Posix(crate::config::PosixIoConfig::new(4, 4, 1)), + IoEngineConfig::Posix(PosixIoConfig::new(4, 4, 1)), None, false, false, diff --git a/cache2/src/io_engine/uring.rs b/cache2/src/io/engine/uring.rs similarity index 96% rename from cache2/src/io_engine/uring.rs rename to cache2/src/io/engine/uring.rs index b502851..02e5426 100644 --- a/cache2/src/io_engine/uring.rs +++ b/cache2/src/io/engine/uring.rs @@ -14,10 +14,26 @@ use std::collections::HashMap; use std::collections::VecDeque; +use std::io; use std::io::Read; use std::io::Write; +use std::mem; use std::os::fd::AsRawFd; use std::os::unix::net::UnixStream; +use std::panic; +use std::panic::AssertUnwindSafe; +use std::sync::Arc; +use std::sync::Condvar; +use std::sync::Mutex; +use std::sync::RwLock; +use std::sync::atomic::AtomicBool; +use std::sync::atomic::AtomicU64; +#[cfg(test)] +use std::sync::atomic::AtomicUsize; +use std::sync::atomic::Ordering; +use std::sync::mpsc; +use std::sync::mpsc::Receiver; +use std::time::Instant; use hashcrew::xxhash::Xxh3_64Builder; use io_uring::IoUring; @@ -26,7 +42,38 @@ use io_uring::opcode; use io_uring::squeue; use io_uring::types; -use super::*; +use crate::config::runtime::IoUringPoolConfig; +use crate::io::backend::RuntimeFileSet; +use crate::io::backend::RuntimeIoPath; +use crate::io::backend::RuntimeIoStatsHandle; +use crate::io::engine::CompletionState; +use crate::io::engine::CompletionStatus; +use crate::io::engine::DriverCommand; +use crate::io::engine::DriverWake; +use crate::io::engine::EngineIoSnapshot; +#[cfg(test)] +use crate::io::engine::IO_QUEUE_ENTRY_RESERVATION_BYTES; +#[cfg(test)] +use crate::io::engine::IoBuffer; +use crate::io::engine::IoEngine; +use crate::io::engine::IoOperation; +use crate::io::engine::IoRequest; +#[cfg(test)] +use crate::io::engine::MAX_IO_REQUESTS_PER_ENGINE; +use crate::io::engine::OperationKind; +use crate::io::engine::ReadSlot; +use crate::io::engine::ReadSlotWaiter; +use crate::io::engine::RequestId; +use crate::io::engine::RuntimeInner; +use crate::io::engine::RuntimeShared; +use crate::io::engine::ShutdownPhase; +use crate::io::engine::ShutdownState; +use crate::io::engine::SubmitError; +use crate::io::engine::SubmitState; +use crate::io::engine::Task; +#[cfg(test)] +use crate::io::engine::io_uring_extra_memory_bytes; +use crate::resources::CACHE_THREAD_STACK_BYTES; const CANCEL_CQE_BIT: u64 = 1_u64 << 63; const INTERNAL_CQE_BIT: u64 = 1_u64 << 62; @@ -83,7 +130,7 @@ impl UringIoEngine { pub fn new_with_files( files: RuntimeFileSet, max_in_flight: usize, - config: crate::config::IoUringPoolConfig, + config: IoUringPoolConfig, statistics_enabled: bool, read_wait_enabled: bool, ) -> io::Result { @@ -352,7 +399,7 @@ fn uring_driver( io_poll, shutting_down: false, }; - let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| driver.run())) + let result = panic::catch_unwind(AssertUnwindSafe(|| driver.run())) .unwrap_or_else(|_| Err(io::Error::other("io_uring driver panicked"))); if let Err(error) = &result { driver.stop_accepting_and_fail_all(error); @@ -860,7 +907,7 @@ impl UringDriver { // and never issue LOCK_UN on another duplicate. self.shared.mark_unfenced_writes(); if let Some(files) = self.files.take() { - std::mem::forget(files); + mem::forget(files); } } diff --git a/cache2/src/io/mod.rs b/cache2/src/io/mod.rs new file mode 100644 index 0000000..9086108 --- /dev/null +++ b/cache2/src/io/mod.rs @@ -0,0 +1,18 @@ +// Copyright 2026 ScopeDB, Inc. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Positioned file access and bounded owned-buffer execution. + +pub mod backend; +pub mod engine; diff --git a/cache2/src/lib.rs b/cache2/src/lib.rs index 969075a..da6a711 100644 --- a/cache2/src/lib.rs +++ b/cache2/src/lib.rs @@ -26,7 +26,6 @@ mod error; pub use self::error::Error; pub use self::error::ErrorKind; pub use self::error::ErrorOperation; -pub use self::error::Result; mod cache; pub use self::cache::Cache; @@ -35,17 +34,17 @@ pub use self::cache::Value; mod config; pub use self::config::CacheConfig; -pub use self::config::IoEngine; -pub use self::config::IoMode; -pub use self::config::IoUringConfig; -pub use self::config::IoUringPoolConfig; -pub use self::config::IoUringSqPollConfig; -pub use self::config::L1EvictionPolicy; -pub use self::config::PosixIoConfig; -pub use self::config::ReadAdmission; -pub use self::config::RuntimeOptions; pub use self::config::StorageLayout; -pub use self::config::StorageOptions; +pub use self::config::runtime::IoEngineConfig; +pub use self::config::runtime::IoMode; +pub use self::config::runtime::IoUringConfig; +pub use self::config::runtime::IoUringPoolConfig; +pub use self::config::runtime::IoUringSqPollConfig; +pub use self::config::runtime::L1EvictionPolicy; +pub use self::config::runtime::PosixIoConfig; +pub use self::config::runtime::ReadAdmission; +pub use self::config::runtime::RuntimeOptions; +pub use self::config::storage::StorageOptions; mod snapshot; pub use self::snapshot::CacheHealth; @@ -61,25 +60,10 @@ pub use self::snapshot::RegionSnapshot; pub use self::snapshot::StartupMode; mod checksum; -mod eviction; -mod format; mod hashing; -mod index; -mod index_storage; -mod io_backend; -mod io_engine; +mod io; mod memory; -mod record_codec; -mod recovery; mod region; -mod region_appender; -mod region_index; -mod region_manager; -mod region_metadata; -mod region_reader; -mod region_runtime; -mod region_staging; -mod region_store; mod resources; #[cfg(test)] diff --git a/cache2/src/eviction.rs b/cache2/src/memory/eviction.rs similarity index 99% rename from cache2/src/eviction.rs rename to cache2/src/memory/eviction.rs index 56ff824..05d587e 100644 --- a/cache2/src/eviction.rs +++ b/cache2/src/memory/eviction.rs @@ -19,7 +19,7 @@ use std::io; -use crate::config::L1EvictionPolicy; +use crate::config::runtime::L1EvictionPolicy; use crate::hashing::FixedPrehashedMap; /// Maximum policy metadata inspected by one complete foreground admission. @@ -329,7 +329,7 @@ impl ClockState { fn allocation_bytes(maximum_entries: usize) -> io::Result { maximum_entries - .checked_mul(std::mem::size_of::()) + .checked_mul(size_of::()) .ok_or_else(|| io::Error::new(io::ErrorKind::InvalidInput, "CLOCK is too large")) } @@ -529,7 +529,7 @@ impl S3FifoState { fn allocation_bytes(maximum_entries: usize) -> io::Result { let links = maximum_entries - .checked_mul(std::mem::size_of::()) + .checked_mul(size_of::()) .ok_or_else(|| io::Error::new(io::ErrorKind::InvalidInput, "S3-FIFO is too large"))?; links .checked_add(GhostQueue::allocation_bytes(maximum_entries)?) @@ -717,12 +717,12 @@ impl GhostQueue { fn allocation_bytes(maximum_entries: usize) -> io::Result { let hashes = maximum_entries - .checked_mul(std::mem::size_of::()) + .checked_mul(size_of::()) .ok_or_else(|| { io::Error::new(io::ErrorKind::InvalidInput, "ghost queue is too large") })?; let links = maximum_entries - .checked_mul(std::mem::size_of::()) + .checked_mul(size_of::()) .ok_or_else(|| { io::Error::new(io::ErrorKind::InvalidInput, "ghost queue is too large") })?; @@ -1075,7 +1075,7 @@ mod tests { #[test] fn ghost_storage_uses_twenty_bytes_per_entry() { const ENTRIES: usize = 5; - assert_eq!(std::mem::size_of::(), 12); + assert_eq!(size_of::(), 12); assert_eq!( GhostQueue::allocation_bytes(ENTRIES).unwrap() - FixedPrehashedMap::allocation_bytes(ENTRIES).unwrap(), diff --git a/cache2/src/memory.rs b/cache2/src/memory/mod.rs similarity index 98% rename from cache2/src/memory.rs rename to cache2/src/memory/mod.rs index 46bfeda..1d7d3ae 100644 --- a/cache2/src/memory.rs +++ b/cache2/src/memory/mod.rs @@ -18,9 +18,12 @@ //! immediately, may be discarded at any time, and use a small bounded eviction //! policy. +use std::hint::spin_loop; use std::io; use std::ops::Deref; use std::sync::Arc; +#[cfg(test)] +use std::sync::Barrier; use std::sync::Mutex; use std::sync::MutexGuard; use std::sync::TryLockError; @@ -28,16 +31,18 @@ use std::sync::atomic::AtomicU64; use std::sync::atomic::AtomicUsize; use std::sync::atomic::Ordering; -use crate::config::L1EvictionPolicy; -use crate::eviction::DetachedPolicy; -use crate::eviction::EvictionState; -use crate::eviction::MAX_POLICY_SCAN_STEPS; -use crate::eviction::MAX_POLICY_SLOT_INDEX; -use crate::eviction::PolicySlot; +use self::eviction::DetachedPolicy; +use self::eviction::EvictionState; +use self::eviction::MAX_POLICY_SCAN_STEPS; +use self::eviction::MAX_POLICY_SLOT_INDEX; +use self::eviction::PolicySlot; +use crate::config::runtime::L1EvictionPolicy; use crate::hashing::FixedPrehashedMap; use crate::hashing::route_hash; use crate::snapshot::CacheL1Snapshot; +mod eviction; + /// Charged retained-value ownership. Fixed entry, policy, and directory /// storage is planned and allocated separately during open. const MEMORY_ENTRY_OVERHEAD_BYTES: usize = 64; @@ -47,7 +52,7 @@ const MEMORY_ENTRY_OVERHEAD_BYTES: usize = 64; const MAX_L1_ENTRY_BYTES: usize = 256 * 1024; /// A sequence trailer uses the spare tail of the fixed entry-overhead charge /// so compact resident slots do not enlarge the hot Arc allocation. -const MEMORY_VALUE_SEQNO_BYTES: usize = std::mem::size_of::(); +const MEMORY_VALUE_SEQNO_BYTES: usize = size_of::(); /// Full-key collision work stays bounded for entries sharing one directory /// fingerprint, including distinct keys with the same 64-bit cache hash. const MAX_SAME_HASH_ENTRIES: usize = 8; @@ -859,10 +864,10 @@ impl MemoryStore { .min(shard_capacity / MEMORY_ENTRY_OVERHEAD_BYTES) .min(MAX_POLICY_SLOT_INDEX.saturating_add(1)); let fixed_slots = shard_entries - .checked_mul(std::mem::size_of::>()) + .checked_mul(size_of::>()) .and_then(|bytes| { shard_entries - .checked_mul(std::mem::size_of::()) + .checked_mul(size_of::()) .and_then(|policy| bytes.checked_add(policy)) }) .ok_or_else(|| invalid_memory_plan("L1 slot memory plan overflow"))?; @@ -938,7 +943,7 @@ impl MemoryStore { Ok(shard) => break shard, Err(TryLockError::WouldBlock) if attempts < MAX_L1_LOOKUP_LOCK_ATTEMPTS => { attempts += 1; - std::hint::spin_loop(); + spin_loop(); } Err(TryLockError::WouldBlock | TryLockError::Poisoned(_)) => { return MemoryLookup::Miss(MemoryReadToken { shard_id }); @@ -1109,9 +1114,9 @@ mod tests { #[test] fn memory_entry_slot_uses_two_machine_words() { - let expected = 2 * std::mem::size_of::(); - assert_eq!(std::mem::size_of::(), expected); - assert_eq!(std::mem::size_of::>(), expected); + let expected = 2 * size_of::(); + assert_eq!(size_of::(), expected); + assert_eq!(size_of::>(), expected); } #[test] @@ -1442,7 +1447,7 @@ mod tests { let clones = (0..8).map(|_| retained.clone()).collect::>(); assert!(!store.publish(22, b"b", &[2; 300], 2)); - let barrier = Arc::new(std::sync::Barrier::new(clones.len() + 1)); + let barrier = Arc::new(Barrier::new(clones.len() + 1)); std::thread::scope(|scope| { for value in clones { let barrier = Arc::clone(&barrier); diff --git a/cache2/src/property_tests.rs b/cache2/src/property_tests.rs index 8e25987..0b16ec1 100644 --- a/cache2/src/property_tests.rs +++ b/cache2/src/property_tests.rs @@ -22,29 +22,29 @@ use quickcheck::QuickCheck; use crate::checksum::Crc32c; use crate::checksum::crc32c; -use crate::format::MAX_KEY_SIZE; -use crate::format::RECORD_ALIGNMENT; -use crate::format::RECORD_HEADER_SIZE; -use crate::format::RecordHeader; use crate::hashing::FixedPrehashedMap; -use crate::index::IndexEntry; -use crate::index::PackedLocation; -use crate::index::record_size_class_upper_bound; -use crate::index_storage::INDEX_IMAGE_SLOT_SIZE; -use crate::index_storage::IndexSlot; -use crate::index_storage::PartitionedIndexStorage; -use crate::record_codec::RecordPayload; -use crate::record_codec::encode_reinsert_into_hashed; -use crate::record_codec::encode_value_into_hashed; -use crate::record_codec::required_record_bytes; -use crate::recovery::DataSuperblock; -use crate::recovery::RECOVERY_PAGE_SIZE; -use crate::recovery::RecoveryImageHeader; -use crate::recovery::StateRecord; -use crate::region_index::ReclaimIndexAction; -use crate::region_index::RegionIndex; -use crate::region_manager::RegionAppendReservation; -use crate::region_metadata::RegionMetadata; +use crate::region::index::ReclaimIndexAction; +use crate::region::index::RegionIndex; +use crate::region::index::packed::IndexEntry; +use crate::region::index::packed::PackedLocation; +use crate::region::index::packed::record_size_class_upper_bound; +use crate::region::index::storage::IndexSlot; +use crate::region::index::storage::PartitionedIndexStorage; +use crate::region::index::storage::page_format::INDEX_IMAGE_SLOT_SIZE; +use crate::region::manager::RegionAppendReservation; +use crate::region::record::MAX_KEY_SIZE; +use crate::region::record::RECORD_ALIGNMENT; +use crate::region::record::RECORD_HEADER_SIZE; +use crate::region::record::RecordHeader; +use crate::region::record::codec::RecordPayload; +use crate::region::record::codec::encode_reinsert_into_hashed; +use crate::region::record::codec::encode_value_into_hashed; +use crate::region::record::codec::required_record_bytes; +use crate::region::recovery::DataSuperblock; +use crate::region::recovery::RECOVERY_PAGE_SIZE; +use crate::region::recovery::RecoveryImageHeader; +use crate::region::recovery::StateRecord; +use crate::region::recovery::metadata::RegionMetadata; const MAX_PROPERTY_INPUT_BYTES: usize = 16 * 1024; const MAX_PROPERTY_MAP_ENTRIES: usize = 64; diff --git a/cache2/src/region_appender.rs b/cache2/src/region/appender.rs similarity index 94% rename from cache2/src/region_appender.rs rename to cache2/src/region/appender.rs index 2850ca7..3c4fb34 100644 --- a/cache2/src/region_appender.rs +++ b/cache2/src/region/appender.rs @@ -21,18 +21,18 @@ use std::fmt; use std::io; -use crate::io_backend::DIRECT_IO_ALIGNMENT; -use crate::io_backend::WritePoint; -use crate::io_engine::BoundedIoRequest; -use crate::io_engine::IoBuffer; -use crate::io_engine::IoEngine; -use crate::io_engine::IoOperation; -use crate::io_engine::OperationKind; -use crate::io_engine::RequestId; -use crate::io_engine::submit_cache_io; -use crate::recovery::DATA_REGION_AREA_OFFSET; -use crate::recovery::DataGeometry; -use crate::region_manager::RegionWriteSpan; +use crate::io::backend::DIRECT_IO_ALIGNMENT; +use crate::io::backend::WritePoint; +use crate::io::engine::BoundedIoRequest; +use crate::io::engine::IoBuffer; +use crate::io::engine::IoEngine; +use crate::io::engine::IoOperation; +use crate::io::engine::OperationKind; +use crate::io::engine::RequestId; +use crate::io::engine::submit_cache_io; +use crate::region::manager::RegionWriteSpan; +use crate::region::recovery::DATA_REGION_AREA_OFFSET; +use crate::region::recovery::DataGeometry; pub struct RegionSpanSubmitError { pub error: io::Error, @@ -242,10 +242,10 @@ mod tests { use std::sync::Mutex; use super::*; - use crate::io_backend::IoBackend; - use crate::io_backend::SyncMode; - use crate::io_backend::SyncPoint; - use crate::io_engine::BackendIoEngine; + use crate::io::backend::IoBackend; + use crate::io::backend::SyncMode; + use crate::io::backend::SyncPoint; + use crate::io::engine::BackendIoEngine; use crate::resources::BufferLease; #[derive(Default)] diff --git a/cache2/src/region/file_backend.rs b/cache2/src/region/file_backend/mod.rs similarity index 94% rename from cache2/src/region/file_backend.rs rename to cache2/src/region/file_backend/mod.rs index 43b8541..93eed41 100644 --- a/cache2/src/region/file_backend.rs +++ b/cache2/src/region/file_backend/mod.rs @@ -14,6 +14,8 @@ //! File ownership, recovery, and lifecycle adapter for the Region core. +use std::fmt; +use std::fs; use std::fs::File; use std::io::Write; use std::io::{self}; @@ -24,73 +26,73 @@ use std::sync::Arc; use std::sync::Mutex; use std::sync::atomic::AtomicU64; -use super::FileRegionCore; -use super::RegionAccessState; -use super::RegionHealthLatch; -use super::RegionManagerAuthority; -use super::RegionShard; -use super::guarded_index_result; -use super::index_storage_io_error; -use super::region_metadata_io_error; use crate::config::CacheConfig; -use crate::config::IoMode; +use crate::config::runtime::IoMode; #[cfg(test)] -use crate::config::RuntimeOptions; +use crate::config::runtime::RuntimeOptions; #[cfg(test)] -use crate::config::cache_config; -use crate::index::MAX_INDEX_PARTITIONS; -use crate::index_storage::IndexImageBinding; -use crate::index_storage::IndexPartitionRange; -use crate::index_storage::IndexPhysicalStats; -use crate::index_storage::PartitionedIndexStorage; -use crate::index_storage::canonical_index_partition_ranges; -use crate::io_backend::ControlIoBackend; -use crate::io_backend::FileBackend; -use crate::io_backend::IoBackend; -use crate::io_backend::RuntimeFileSet; -use crate::io_backend::SyncMode; -use crate::io_backend::SyncPoint; -use crate::io_backend::WritePoint; -use crate::io_backend::read_at_bounded; -use crate::io_backend::read_exact_at; -use crate::io_backend::write_all_at; -use crate::recovery::DataSuperblock; -use crate::recovery::DataSuperblockProbe; -use crate::recovery::PersistentId; -use crate::recovery::RECOVERY_IMAGE_INDEX_OFFSET; -use crate::recovery::RECOVERY_PAGE_SIZE; -use crate::recovery::RecoveryImageHeader; -use crate::recovery::RecoveryImageHeaderProbe; -use crate::recovery::RecoveryState; -use crate::recovery::STATE_FILE_SIZE; -use crate::recovery::STATE_SLOT_COUNT; -use crate::recovery::SelectedState; -use crate::recovery::StateBinding; -use crate::recovery::StatePageWrite; -use crate::recovery::StateRecord; -use crate::recovery::StateSelectionError; -use crate::recovery::clean_image_matches; -use crate::recovery::latest_state; -use crate::recovery::prepare_next_state; -use crate::recovery::prepare_running_barrier; -use crate::recovery::recovery_image_index_len; -use crate::region_index::RegionIndex; -use crate::region_manager::RegionManager; -use crate::region_metadata::PartitionMetadataRecord; -use crate::region_metadata::REGION_METADATA_PAGE_SIZE; -use crate::region_metadata::REGION_METADATA_PARTITIONS_PER_PAGE; -use crate::region_metadata::REGION_METADATA_REGIONS_PER_PAGE; -use crate::region_metadata::RegionMetadata; -use crate::region_metadata::RegionMetadataError; -use crate::region_metadata::RegionMetadataRecord; -use crate::region_metadata::RegionMetadataRoot; -use crate::region_metadata::RegionMetadataState; +use crate::config::storage::cache_config; +use crate::io::backend::ControlIoBackend; +use crate::io::backend::FileBackend; +use crate::io::backend::IoBackend; +use crate::io::backend::RuntimeFileSet; +use crate::io::backend::SyncMode; +use crate::io::backend::SyncPoint; +use crate::io::backend::WritePoint; +use crate::io::backend::read_at_bounded; +use crate::io::backend::read_exact_at; +use crate::io::backend::write_all_at; +use crate::region::FileRegionCore; +use crate::region::RegionAccessState; +use crate::region::RegionHealthLatch; +use crate::region::RegionManagerAuthority; +use crate::region::RegionShard; +use crate::region::guarded_index_result; +use crate::region::index::RegionIndex; +use crate::region::index::packed::MAX_INDEX_PARTITIONS; +use crate::region::index::storage::IndexImageBinding; +use crate::region::index::storage::IndexPartitionRange; +use crate::region::index::storage::IndexPhysicalStats; +use crate::region::index::storage::PartitionedIndexStorage; +use crate::region::index::storage::canonical_index_partition_ranges; +use crate::region::index_storage_io_error; +use crate::region::manager::RegionManager; +use crate::region::recovery::DataSuperblock; +use crate::region::recovery::DataSuperblockProbe; +use crate::region::recovery::PersistentId; +use crate::region::recovery::RECOVERY_IMAGE_INDEX_OFFSET; +use crate::region::recovery::RECOVERY_PAGE_SIZE; +use crate::region::recovery::RecoveryImageHeader; +use crate::region::recovery::RecoveryImageHeaderProbe; +use crate::region::recovery::RecoveryState; +use crate::region::recovery::STATE_FILE_SIZE; +use crate::region::recovery::STATE_SLOT_COUNT; +use crate::region::recovery::SelectedState; +use crate::region::recovery::StateBinding; +use crate::region::recovery::StatePageWrite; +use crate::region::recovery::StateRecord; +use crate::region::recovery::StateSelectionError; +use crate::region::recovery::clean_image_matches; +use crate::region::recovery::latest_state; +use crate::region::recovery::metadata::PartitionMetadataRecord; +use crate::region::recovery::metadata::REGION_METADATA_PAGE_SIZE; +use crate::region::recovery::metadata::REGION_METADATA_PARTITIONS_PER_PAGE; +use crate::region::recovery::metadata::REGION_METADATA_REGIONS_PER_PAGE; +use crate::region::recovery::metadata::RegionMetadata; +use crate::region::recovery::metadata::RegionMetadataError; +use crate::region::recovery::metadata::RegionMetadataRecord; +use crate::region::recovery::metadata::RegionMetadataRoot; +use crate::region::recovery::metadata::RegionMetadataState; +use crate::region::recovery::prepare_next_state; +use crate::region::recovery::prepare_running_barrier; +use crate::region::recovery::recovery_image_index_len; +use crate::region::region_metadata_io_error; #[cfg(test)] -use crate::region_runtime::HybridValueRead; -use crate::region_runtime::RegionDataPlane; -use crate::region_store::RecoveryPlan; -use crate::region_store::RegionBackend; -use crate::region_store::RegionStore; +use crate::region::runtime::HybridValueRead; +use crate::region::runtime::RegionDataPlane; +use crate::region::store::RecoveryPlan; +use crate::region::store::RegionBackend; +use crate::region::store::RegionStore; #[cfg(test)] use crate::snapshot::CacheSnapshot; #[cfg(test)] @@ -339,7 +341,7 @@ impl RegionFileSystem for SystemRegionFileSystem { } fn remove_file(&self, path: &Path) -> io::Result<()> { - match std::fs::remove_file(path) { + match fs::remove_file(path) { Ok(()) => Ok(()), Err(error) if error.kind() == io::ErrorKind::NotFound => Ok(()), Err(error) => Err(error), @@ -347,7 +349,7 @@ impl RegionFileSystem for SystemRegionFileSystem { } fn rename(&self, source: &Path, destination: &Path) -> io::Result<()> { - std::fs::rename(source, destination) + fs::rename(source, destination) } fn sync_parent(&self, path: &Path) -> io::Result<()> { @@ -470,7 +472,7 @@ where reason: &'static str, index_slots: usize, index_mapping_bytes: u64, - error: &impl std::fmt::Display, + error: &impl fmt::Display, ) { log::warn!( target: "cache2::recovery", diff --git a/cache2/src/region/file_backend/tests.rs b/cache2/src/region/file_backend/tests.rs index adab151..d14e2ee 100644 --- a/cache2/src/region/file_backend/tests.rs +++ b/cache2/src/region/file_backend/tests.rs @@ -12,8 +12,13 @@ // See the License for the specific language governing permissions and // limitations under the License. +use std::env; +use std::fs; +use std::future::Future; +use std::future::poll_fn; #[cfg(unix)] use std::os::unix::process::ExitStatusExt; +use std::pin::Pin; #[cfg(unix)] use std::process::Command; #[cfg(unix)] @@ -22,33 +27,42 @@ use std::sync::Arc; use std::sync::Mutex; use std::sync::atomic::AtomicU64; use std::sync::atomic::Ordering; +use std::sync::mpsc; +use std::task::Poll; use std::time::Duration; use std::time::Instant; use super::*; -use crate::config::ReadAdmission; -use crate::index::IndexEntry; -use crate::index::PackedLocation; -use crate::index_storage::INDEX_IMAGE_SLOTS_PER_PAGE; -use crate::index_storage::IndexSlot; -use crate::io_backend::testing::FaultAction; -use crate::io_backend::testing::FaultBackend; -use crate::io_backend::testing::FaultEvent; -use crate::io_backend::testing::FaultHandle; -use crate::io_engine::BackendIoEngine; -use crate::io_engine::IoEngine; -use crate::record_codec::hash_key; -use crate::record_codec::required_record_bytes; -use crate::recovery::DATA_REGION_AREA_OFFSET; -use crate::recovery::DataGeometry; -use crate::recovery::PersistentId; +use crate::IoEngineConfig; +use crate::config::runtime::MAX_WRITE_FLUSH_THRESHOLD_BYTES; +use crate::config::runtime::PosixIoConfig; +use crate::config::runtime::ReadAdmission; +use crate::io::backend::MAX_INTERRUPTED_RETRIES; +use crate::io::backend::testing::FaultAction; +use crate::io::backend::testing::FaultBackend; +use crate::io::backend::testing::FaultEvent; +use crate::io::backend::testing::FaultHandle; +use crate::io::backend::testing::kill_process; +use crate::io::engine::BackendIoEngine; +use crate::io::engine::IoEngine; use crate::region::RegionStageValue; -use crate::region_reader::ReadCandidate; -use crate::region_reader::ReadCompletion; -use crate::region_reader::ReadPlan; -use crate::region_reader::plan_read; -use crate::region_staging::RegionStaging; -use crate::region_staging::StagedRecord; +use crate::region::index::packed::IndexEntry; +use crate::region::index::packed::PackedLocation; +use crate::region::index::storage::IndexSlot; +use crate::region::index::storage::IndexSlotState; +use crate::region::index::storage::page_format::INDEX_IMAGE_SLOTS_PER_PAGE; +use crate::region::reader::ReadCandidate; +use crate::region::reader::ReadCompletion; +use crate::region::reader::ReadPlan; +use crate::region::reader::plan_read; +use crate::region::record::RECORD_ALIGNMENT; +use crate::region::record::codec::hash_key; +use crate::region::record::codec::required_record_bytes; +use crate::region::recovery::DATA_REGION_AREA_OFFSET; +use crate::region::recovery::DataGeometry; +use crate::region::recovery::PersistentId; +use crate::region::staging::RegionStaging; +use crate::region::staging::StagedRecord; use crate::resources::ResourceController; use crate::resources::ResourceLimits; use crate::snapshot::StartupMode; @@ -72,13 +86,11 @@ fn eventually_admitted(mut put: impl FnMut() -> io::Result) -> T { } } -async fn assert_pending(mut future: std::pin::Pin<&mut F>, message: &str) { - std::future::poll_fn( - |context| match std::future::Future::poll(future.as_mut(), context) { - std::task::Poll::Pending => std::task::Poll::Ready(()), - std::task::Poll::Ready(_) => panic!("{message}"), - }, - ) +async fn assert_pending(mut future: Pin<&mut F>, message: &str) { + poll_fn(|context| match Future::poll(future.as_mut(), context) { + Poll::Pending => Poll::Ready(()), + Poll::Ready(_) => panic!("{message}"), + }) .await; } @@ -90,10 +102,9 @@ struct TestDirectory { impl TestDirectory { fn new() -> Self { let ordinal = NEXT_TEST_DIRECTORY.fetch_add(1, Ordering::Relaxed); - let root = - std::env::temp_dir().join(format!("cache2-region-{}-{ordinal}", std::process::id())); - let _ = std::fs::remove_dir_all(&root); - std::fs::create_dir(&root).unwrap(); + let root = env::temp_dir().join(format!("cache2-region-{}-{ordinal}", std::process::id())); + let _ = fs::remove_dir_all(&root); + fs::create_dir(&root).unwrap(); let files = RegionFiles::new( root.join("data"), root.join("state"), @@ -105,7 +116,7 @@ impl TestDirectory { impl Drop for TestDirectory { fn drop(&mut self) { - let _ = std::fs::remove_dir_all(&self.root); + let _ = fs::remove_dir_all(&self.root); } } @@ -125,7 +136,7 @@ fn state_page_reads_stop_after_the_interrupted_retry_budget() { .iter() .filter(|event| **event == FaultEvent::Read) .count(), - crate::io_backend::MAX_INTERRUPTED_RETRIES + 1 + MAX_INTERRUPTED_RETRIES + 1 ); } @@ -193,7 +204,7 @@ impl RegionFileSystem for FaultRegionFileSystem { } fn remove_file(&self, path: &Path) -> io::Result<()> { - match std::fs::remove_file(path) { + match fs::remove_file(path) { Ok(()) => Ok(()), Err(error) if error.kind() == io::ErrorKind::NotFound => Ok(()), Err(error) => Err(error), @@ -202,7 +213,7 @@ impl RegionFileSystem for FaultRegionFileSystem { fn rename(&self, source: &Path, destination: &Path) -> io::Result<()> { self.file_system.check(FileSystemFault::Rename)?; - std::fs::rename(source, destination) + fs::rename(source, destination) } fn sync_parent(&self, path: &Path) -> io::Result<()> { @@ -262,8 +273,8 @@ fn external_process_kill_recovery_contract() { const CHILD_CASE: &str = "CACHE2_CRASH_CHILD_CASE"; const CHILD_ROOT: &str = "CACHE2_CRASH_CHILD_ROOT"; - if let Ok(case) = std::env::var(CHILD_CASE) { - let root = PathBuf::from(std::env::var_os(CHILD_ROOT).expect("child root is set")); + if let Ok(case) = env::var(CHILD_CASE) { + let root = PathBuf::from(env::var_os(CHILD_ROOT).expect("child root is set")); let files = RegionFiles::new( root.join("data"), root.join("state"), @@ -291,7 +302,7 @@ fn external_process_kill_recovery_contract() { initial.drain().unwrap(); initial.close_warm().unwrap(); - let status = Command::new(std::env::current_exe().unwrap()) + let status = Command::new(env::current_exe().unwrap()) .arg("--exact") .arg("region::file_backend::tests::external_process_kill_recovery_contract") .arg("--ignored") @@ -335,7 +346,7 @@ fn run_crash_child(case: &str, files: RegionFiles) -> ! { "open" => { let _store = RegionStore::open(4096, FileRegionBackend::for_test(files, data, 4096)).unwrap(); - crate::io_backend::testing::kill_process(); + kill_process(); } "write" | "drain" => { let store = @@ -344,7 +355,7 @@ fn run_crash_child(case: &str, files: RegionFiles) -> ! { if case == "drain" { store.drain().unwrap(); } - crate::io_backend::testing::kill_process(); + kill_process(); } "warm-data" | "warm-image" | "clean-state" => { let (file_system, faults, _) = FaultRegionFileSystem::new(); @@ -398,7 +409,7 @@ fn configured_read_wait_is_bounded_and_cancel_safe() { let directory = TestDirectory::new(); let data = production_data_superblock(512 * 1024); let runtime_config = RuntimeOptions { - io_engine: crate::config::IoEngine::Posix(crate::config::PosixIoConfig::new(2, 4, 1)), + io_engine: IoEngineConfig::Posix(PosixIoConfig::new(2, 4, 1)), l1_capacity_bytes: 0, statistics: true, read_admission: ReadAdmission::Wait { @@ -482,7 +493,7 @@ fn queued_l2_read_does_not_pin_warm_close() { let directory = TestDirectory::new(); let data = production_data_superblock(512 * 1024); let runtime_config = RuntimeOptions { - io_engine: crate::config::IoEngine::Posix(crate::config::PosixIoConfig::new(1, 4, 1)), + io_engine: IoEngineConfig::Posix(PosixIoConfig::new(1, 4, 1)), l1_capacity_bytes: 0, read_admission: ReadAdmission::Wait { timeout: Duration::from_secs(1), @@ -647,7 +658,7 @@ fn poisoned_runtime_gates_stop_workers_and_reject_warm_close() { let directory = TestDirectory::new(); let data = production_data_superblock(512 * 1024); let runtime_config = RuntimeOptions { - io_engine: crate::config::IoEngine::Posix(crate::config::PosixIoConfig::new(1, 1, 1)), + io_engine: IoEngineConfig::Posix(PosixIoConfig::new(1, 1, 1)), l1_capacity_bytes: 0, managed_memory_limit_bytes: 32 * 1024 * 1024, write_flush_threshold_bytes: 128 * 1024, @@ -756,7 +767,7 @@ fn foreground_stage_fixture() -> (DataSuperblock, FileRegionRuntime, RegionStagi let resources = data_path_resources(); let staging = RegionStaging::try_new( 1, - crate::config::MAX_WRITE_FLUSH_THRESHOLD_BYTES, + MAX_WRITE_FLUSH_THRESHOLD_BYTES, data.geometry.region_size, &resources, ) @@ -788,7 +799,7 @@ fn foreground_stage_rejects_busy_shard_without_reserving_then_stages_once() { let mutation = runtime.core.shards[0].mutation.lock().unwrap(); let hash = hash_key(data.hash_seed, b"key"); let record_bytes = required_record_bytes(b"key".len(), b"value".len()).unwrap(); - let (sender, receiver) = std::sync::mpsc::sync_channel(1); + let (sender, receiver) = mpsc::sync_channel(1); let core = Arc::clone(&runtime.core); let writer = std::thread::spawn(move || { let result = core.try_stage_value(&staging, 0, hash, record_bytes, b"key", b"value"); @@ -833,11 +844,11 @@ fn completed_record_publication_does_not_enter_region_manager() { let record = StagedRecord::new( 7, IndexEntry { - location: crate::index::PackedLocation::new(0, 0, 64).unwrap(), + location: PackedLocation::new(0, 0, 64).unwrap(), }, 1, ); - let (sender, receiver) = std::sync::mpsc::sync_channel(1); + let (sender, receiver) = mpsc::sync_channel(1); let publisher_core = Arc::clone(&core); let publisher = std::thread::spawn(move || { sender @@ -845,7 +856,7 @@ fn completed_record_publication_does_not_enter_region_manager() { .unwrap(); }); - let published = receiver.recv_timeout(std::time::Duration::from_secs(1)); + let published = receiver.recv_timeout(Duration::from_secs(1)); drop(manager); publisher.join().unwrap(); published.unwrap().unwrap(); @@ -864,7 +875,7 @@ fn completed_owned_span_publishes_index_without_a_steady_state_sync() { let resources = data_path_resources(); let staging = RegionStaging::try_new( 1, - crate::config::MAX_WRITE_FLUSH_THRESHOLD_BYTES, + MAX_WRITE_FLUSH_THRESHOLD_BYTES, data.geometry.region_size, &resources, ) @@ -1026,7 +1037,7 @@ fn same_hash_candidate_requires_full_key() { let resources = data_path_resources(); let staging = RegionStaging::try_new( 1, - crate::config::MAX_WRITE_FLUSH_THRESHOLD_BYTES, + MAX_WRITE_FLUSH_THRESHOLD_BYTES, data.geometry.region_size, &resources, ) @@ -1114,7 +1125,7 @@ fn same_hash_candidate_requires_full_key() { let wrong_length_location = PackedLocation::new( current.entry.location.region_id(), current.entry.location.offset(), - current.entry.location.record_len() + crate::format::RECORD_ALIGNMENT, + current.entry.location.record_len() + RECORD_ALIGNMENT, ) .unwrap(); let wrong_length = ReadCandidate { @@ -1168,7 +1179,7 @@ fn failed_span_write_never_publishes_and_latches_miss_only() { let resources = data_path_resources(); let staging = RegionStaging::try_new( 1, - crate::config::MAX_WRITE_FLUSH_THRESHOLD_BYTES, + MAX_WRITE_FLUSH_THRESHOLD_BYTES, data.geometry.region_size, &resources, ) @@ -1430,11 +1441,11 @@ fn complete_warm_image_maps_without_rebuilding_index_slots() { let directory = TestDirectory::new(); let config = INDEX_IMAGE_SLOTS_PER_PAGE + 8; let data = test_data_superblock_with_regions(REGION_SHARDS + 1); - let value = IndexSlot::from_state(crate::index_storage::IndexSlotState::Value { + let value = IndexSlot::from_state(IndexSlotState::Value { fingerprint: 7, displacement: 0, entry: IndexEntry { - location: crate::index::PackedLocation::new(0, 0, 32).unwrap(), + location: PackedLocation::new(0, 0, 32).unwrap(), }, }); @@ -1747,8 +1758,8 @@ fn data_and_state_inode_alias_is_rejected_without_truncation() { let config = 8; let data = test_data_superblock(); let marker = b"do-not-truncate"; - std::fs::write(&directory.files.data, marker).unwrap(); - std::fs::hard_link(&directory.files.data, &directory.files.state).unwrap(); + fs::write(&directory.files.data, marker).unwrap(); + fs::hard_link(&directory.files.data, &directory.files.state).unwrap(); let opened = RegionStore::open( config, @@ -1758,7 +1769,7 @@ fn data_and_state_inode_alias_is_rejected_without_truncation() { opened, Err(error) if error.kind() == io::ErrorKind::InvalidInput )); - assert_eq!(std::fs::read(&directory.files.data).unwrap(), marker); + assert_eq!(fs::read(&directory.files.data).unwrap(), marker); } #[test] @@ -1767,7 +1778,7 @@ fn recovery_temporary_path_cannot_name_the_data_or_state_file() { let marker = b"keep-data"; let image = directory.root.join("recovery"); let data_path = directory.root.join("recovery.next"); - std::fs::write(&data_path, marker).unwrap(); + fs::write(&data_path, marker).unwrap(); let files = RegionFiles::new(&data_path, directory.root.join("state"), image); let opened = RegionStore::open( @@ -1782,14 +1793,14 @@ fn recovery_temporary_path_cannot_name_the_data_or_state_file() { opened, Err(error) if error.kind() == io::ErrorKind::InvalidInput )); - assert_eq!(std::fs::read(data_path).unwrap(), marker); + assert_eq!(fs::read(data_path).unwrap(), marker); } #[test] fn recovery_sidecars_must_share_one_directory() { let directory = TestDirectory::new(); let other = directory.root.join("other"); - std::fs::create_dir(&other).unwrap(); + fs::create_dir(&other).unwrap(); let files = RegionFiles::new( directory.root.join("data"), other.join("state"), diff --git a/cache2/src/region_index.rs b/cache2/src/region/index/mod.rs similarity index 98% rename from cache2/src/region_index.rs rename to cache2/src/region/index/mod.rs index 19a0f97..c04afa1 100644 --- a/cache2/src/region_index.rs +++ b/cache2/src/region/index/mod.rs @@ -21,6 +21,7 @@ //! There are no probe chains, //! tombstones, generation tables, retries, or request-time allocations. +use std::array; #[cfg(feature = "benchmarking")] use std::cell::Cell; use std::io; @@ -28,16 +29,19 @@ use std::sync::atomic::AtomicBool; use std::sync::atomic::AtomicU64; use std::sync::atomic::Ordering; +use self::storage::IndexPartitionWriteGuard; +use self::storage::IndexSlotState; +use self::storage::IndexStorageError; +use self::storage::PartitionedIndexStorage; use crate::hashing::route_hash; -use crate::index::INDEX_CANDIDATES; -use crate::index::IndexEntry; -use crate::index::PackedLocation; -use crate::index_storage::IndexPartitionWriteGuard; -use crate::index_storage::IndexSlotState; -use crate::index_storage::IndexStorageError; -use crate::index_storage::PartitionedIndexStorage; +use crate::region::index::packed::INDEX_CANDIDATES; +use crate::region::index::packed::IndexEntry; +use crate::region::index::packed::PackedLocation; use crate::snapshot::CacheIndexSnapshot; +pub mod packed; +pub mod storage; + const CANDIDATE_OFFSETS: [usize; INDEX_CANDIDATES] = [0, 23, 61, 97]; const FINGERPRINT_MASK: u16 = (1 << 14) - 1; const REFERENCE_WORD_BITS: usize = u64::BITS as usize; @@ -45,7 +49,7 @@ const REFERENCE_WORD_BITS: usize = u64::BITS as usize; pub fn heat_memory_bytes(slot_count: usize) -> Option { let bitmap_bytes = slot_count .div_ceil(REFERENCE_WORD_BITS) - .checked_mul(std::mem::size_of::())?; + .checked_mul(size_of::())?; bitmap_bytes.checked_mul(2) } @@ -655,7 +659,7 @@ fn fingerprint(hash: u64) -> u16 { fn candidate_slots(hash: u64, slot_count: usize) -> [usize; INDEX_CANDIDATES] { let home = route_hash(hash.rotate_left(32), slot_count); - std::array::from_fn(|displacement| slot_from_home(home, displacement, slot_count)) + array::from_fn(|displacement| slot_from_home(home, displacement, slot_count)) } fn slot_from_home(home: usize, displacement: usize, slot_count: usize) -> usize { @@ -692,8 +696,8 @@ fn candidate_offset(displacement: usize, slot_count: usize) -> usize { #[cfg(test)] mod tests { use super::*; - use crate::index_storage::IndexPhysicalStats; - use crate::record_codec::hash_key; + use crate::region::index::storage::IndexPhysicalStats; + use crate::region::record::codec::hash_key; fn entry(region_id: u32, offset: u32) -> IndexEntry { IndexEntry { diff --git a/cache2/src/index.rs b/cache2/src/region/index/packed.rs similarity index 100% rename from cache2/src/index.rs rename to cache2/src/region/index/packed.rs diff --git a/cache2/src/fixtures/format_v1/index_page.golden b/cache2/src/region/index/storage/format_v1/index_page.golden similarity index 100% rename from cache2/src/fixtures/format_v1/index_page.golden rename to cache2/src/region/index/storage/format_v1/index_page.golden diff --git a/cache2/src/index_storage.rs b/cache2/src/region/index/storage/mod.rs similarity index 98% rename from cache2/src/index_storage.rs rename to cache2/src/region/index/storage/mod.rs index f8d148b..9b8f328 100644 --- a/cache2/src/index_storage.rs +++ b/cache2/src/region/index/storage/mod.rs @@ -28,6 +28,7 @@ use std::io::{self}; #[cfg(any(target_os = "linux", target_os = "macos"))] use std::os::fd::AsRawFd; use std::ptr; +use std::slice; use std::sync::Arc; use std::sync::RwLock; use std::sync::RwLockReadGuard; @@ -44,19 +45,20 @@ use self::page_format::put_u32; use self::page_format::put_u64; use self::page_format::read_u64; use self::page_format::validate_page_header; -use crate::index::INDEX_CANDIDATES; -use crate::index::IndexEntry; -use crate::index::MAX_INDEX_PARTITIONS; -use crate::index::PackedLocation; -use crate::index::PackedLocationError; -use crate::index::index_partition_for; -use crate::index::record_size_class_upper_bound; - -mod page_format; -pub use self::page_format::INDEX_IMAGE_PAGE_HEADER_SIZE; -pub use self::page_format::INDEX_IMAGE_PAGE_SIZE; -pub use self::page_format::INDEX_IMAGE_SLOT_SIZE; -pub use self::page_format::INDEX_IMAGE_SLOTS_PER_PAGE; +use crate::region::index::packed::INDEX_CANDIDATES; +use crate::region::index::packed::IndexEntry; +use crate::region::index::packed::MAX_INDEX_PARTITIONS; +use crate::region::index::packed::PackedLocation; +use crate::region::index::packed::PackedLocationError; +use crate::region::index::packed::index_partition_for; +use crate::region::index::packed::record_size_class_upper_bound; +use crate::region::index::storage::page_format::INDEX_IMAGE_PAGE_HEADER_SIZE; +use crate::region::index::storage::page_format::INDEX_IMAGE_PAGE_SIZE; +use crate::region::index::storage::page_format::INDEX_IMAGE_SLOT_SIZE; +use crate::region::index::storage::page_format::INDEX_IMAGE_SLOTS_PER_PAGE; +use crate::region::record::RECORD_ALIGNMENT; + +pub mod page_format; /// Upper bound for one underlying warm-image write. /// @@ -278,7 +280,7 @@ impl IndexSlot { entry, } => { let location = entry.location; - let offset_units = u64::from(location.offset() / crate::format::RECORD_ALIGNMENT); + let offset_units = u64::from(location.offset() / RECORD_ALIGNMENT); Self { encoded: u64::from(location.region_id()) | (offset_units << SLOT_OFFSET_SHIFT) @@ -311,7 +313,7 @@ impl IndexSlot { .ok_or(IndexSlotSemanticError::NonCanonicalMarker)?; let region_id = ((self.encoded >> SLOT_REGION_SHIFT) & SLOT_REGION_MASK) as u32; let offset_units = ((self.encoded >> SLOT_OFFSET_SHIFT) & SLOT_OFFSET_MASK) as u32; - let offset = offset_units * crate::format::RECORD_ALIGNMENT; + let offset = offset_units * RECORD_ALIGNMENT; let location = PackedLocation::new(region_id, offset, record_len) .map_err(IndexSlotSemanticError::InvalidLocation)?; Ok(IndexSlotState::Value { @@ -1057,9 +1059,8 @@ impl IndexStorageCore { .ok_or(IndexStorageError::SizeOverflow)?; // SAFETY: `offset` and the fixed page length are inside `image_len` by // construction, and the mapping remains alive for this borrow. - let page = unsafe { - std::slice::from_raw_parts(self.data_ptr().add(offset), INDEX_IMAGE_PAGE_SIZE) - }; + let page = + unsafe { slice::from_raw_parts(self.data_ptr().add(offset), INDEX_IMAGE_PAGE_SIZE) }; let page: &[u8; INDEX_IMAGE_PAGE_SIZE] = page .try_into() .expect("fixed mapped page has the Index Image page size"); @@ -1831,6 +1832,7 @@ unsafe impl Sync for Mapping {} #[cfg(test)] mod tests { + use std::env; use std::fs::OpenOptions; use std::io::Read; use std::io::Seek; @@ -1859,7 +1861,7 @@ mod tests { impl TestFile { fn create() -> Self { let id = NEXT_TEST_FILE.fetch_add(1, Ordering::Relaxed); - let path = std::env::temp_dir().join(format!( + let path = env::temp_dir().join(format!( "cache2-index-image-{}-{id}.tmp", std::process::id() )); @@ -1882,7 +1884,7 @@ mod tests { fn sample_slot(seed: u64) -> IndexSlot { let location = PackedLocation::new( (seed % 64) as u32, - ((seed % 128) * u64::from(crate::format::RECORD_ALIGNMENT)) as u32, + ((seed % 128) * u64::from(RECORD_ALIGNMENT)) as u32, 32, ) .unwrap(); @@ -2286,10 +2288,7 @@ mod tests { source .write_warm_image(&mut encoded, binding(0x1122_3344_5566_7788)) .unwrap(); - assert_golden( - &encoded, - include_str!("fixtures/format_v1/index_page.golden"), - ); + assert_golden(&encoded, include_str!("format_v1/index_page.golden")); } #[test] diff --git a/cache2/src/index_storage/page_format.rs b/cache2/src/region/index/storage/page_format.rs similarity index 96% rename from cache2/src/index_storage/page_format.rs rename to cache2/src/region/index/storage/page_format.rs index 89334f5..e6571a3 100644 --- a/cache2/src/index_storage/page_format.rs +++ b/cache2/src/region/index/storage/page_format.rs @@ -12,10 +12,10 @@ // See the License for the specific language governing permissions and // limitations under the License. -use super::CorruptPageReason; -use super::IndexImageBinding; -use super::IndexStorageError; use crate::checksum::Crc32c; +use crate::region::index::storage::CorruptPageReason; +use crate::region::index::storage::IndexImageBinding; +use crate::region::index::storage::IndexStorageError; pub const INDEX_IMAGE_PAGE_SIZE: usize = 4096; pub const INDEX_IMAGE_PAGE_HEADER_SIZE: usize = 64; @@ -197,8 +197,8 @@ pub fn validate_page_header( pub fn page_checksum(page: &[u8; INDEX_IMAGE_PAGE_SIZE]) -> u32 { let mut checksum = Crc32c::new(); checksum.update(&page[..PAGE_CHECKSUM_OFFSET]); - checksum.update(&[0_u8; std::mem::size_of::()]); - checksum.update(&page[PAGE_CHECKSUM_OFFSET + std::mem::size_of::()..]); + checksum.update(&[0_u8; size_of::()]); + checksum.update(&page[PAGE_CHECKSUM_OFFSET + size_of::()..]); checksum.finish() } diff --git a/cache2/src/region_manager.rs b/cache2/src/region/manager.rs similarity index 99% rename from cache2/src/region_manager.rs rename to cache2/src/region/manager.rs index 73b1eb7..1e45c90 100644 --- a/cache2/src/region_manager.rs +++ b/cache2/src/region/manager.rs @@ -21,15 +21,15 @@ use std::collections::VecDeque; -use crate::format::RECORD_ALIGNMENT; -use crate::io_backend::DIRECT_IO_ALIGNMENT; -use crate::recovery::PersistentId; -use crate::region_metadata::PartitionMetadataRecord; -use crate::region_metadata::RegionMetadata; -use crate::region_metadata::RegionMetadataError; -use crate::region_metadata::RegionMetadataRecord; -use crate::region_metadata::RegionMetadataRoot; -use crate::region_metadata::RegionMetadataState; +use crate::io::backend::DIRECT_IO_ALIGNMENT; +use crate::region::record::RECORD_ALIGNMENT; +use crate::region::recovery::PersistentId; +use crate::region::recovery::metadata::PartitionMetadataRecord; +use crate::region::recovery::metadata::RegionMetadata; +use crate::region::recovery::metadata::RegionMetadataError; +use crate::region::recovery::metadata::RegionMetadataRecord; +use crate::region::recovery::metadata::RegionMetadataRoot; +use crate::region::recovery::metadata::RegionMetadataState; use crate::snapshot::RegionSnapshot; const UNASSIGNED_REGION: u32 = u32::MAX; @@ -1280,8 +1280,8 @@ fn try_unassigned_queue( #[cfg(test)] mod tests { use super::*; - use crate::index_storage::INDEX_IMAGE_SLOTS_PER_PAGE; - use crate::index_storage::canonical_index_partition_ranges; + use crate::region::index::storage::canonical_index_partition_ranges; + use crate::region::index::storage::page_format::INDEX_IMAGE_SLOTS_PER_PAGE; fn id(byte: u8) -> PersistentId { PersistentId::from_bytes([byte; 16]).unwrap() diff --git a/cache2/src/region/mod.rs b/cache2/src/region/mod.rs index b86da0d..b295580 100644 --- a/cache2/src/region/mod.rs +++ b/cache2/src/region/mod.rs @@ -14,6 +14,7 @@ //! Steady-state Region authority and bounded request-path operations. +use std::fmt; use std::io; use std::ops::Range; use std::sync::Arc; @@ -24,60 +25,72 @@ use std::sync::atomic::AtomicU8; use std::sync::atomic::AtomicU64; use std::sync::atomic::Ordering; +use self::appender::submit_span; +use self::index::ReclaimIndexAction; +use self::index::RegionIndex; +use self::index::heat_memory_bytes; +use self::index::storage::IndexStorageError; +use self::index::storage::WARM_IMAGE_WRITE_BATCH_BYTES; +use self::index::storage::canonical_index_partition_ranges; +use self::manager::RegionManager; +use self::manager::RegionMutationError; +use self::manager::RegionReclaimReceipt; +use self::reader::PendingRead; +use self::reader::ReadCandidate; +use self::reader::ReadCompletion; +use self::reader::ReadPlan; +#[cfg(test)] +use self::reader::plan_read; +use self::reader::submit_read; +use self::record::RECORD_ALIGNMENT; +use self::record::RECORD_HEADER_SIZE; +use self::record::RecordHeader; +use self::recovery::DATA_REGION_AREA_OFFSET; +use self::recovery::recovery_image_index_len; +use self::staging::StageAppend; +use self::staging::StagedRecord; +use self::staging::StagedWrite; +use self::staging::StagingEncodeError; +use self::staging::StagingError; use crate::checksum::crc32c; -use crate::format::RECORD_ALIGNMENT; -use crate::format::RECORD_HEADER_SIZE; -use crate::format::RecordHeader; use crate::hashing::route_hash; +use crate::io::backend::DIRECT_IO_ALIGNMENT; +use crate::io::engine::IoBuffer; +use crate::io::engine::IoEngine; +use crate::io::engine::ReadSlot; +use crate::region::appender::RegionSpanCompletion; #[cfg(test)] -use crate::index::IndexEntry; -use crate::index::PackedLocation; -use crate::index_storage::INDEX_IMAGE_PAGE_SIZE; -use crate::index_storage::IndexStorageError; -use crate::index_storage::WARM_IMAGE_WRITE_BATCH_BYTES; -use crate::index_storage::canonical_index_partition_ranges; -use crate::io_engine::IoEngine; -use crate::io_engine::ReadSlot; -use crate::record_codec::RecordEncodeError; -use crate::record_codec::RecordPayload; -use crate::record_codec::encode_reinsert_into_hashed; -use crate::record_codec::encode_value_into_hashed; -#[cfg(test)] -use crate::record_codec::hash_key; -use crate::recovery::DATA_REGION_AREA_OFFSET; -use crate::recovery::recovery_image_index_len; -use crate::region_appender::submit_span; -use crate::region_index::ReclaimIndexAction; -use crate::region_index::RegionIndex; -use crate::region_index::heat_memory_bytes; -use crate::region_manager::RegionManager; -use crate::region_manager::RegionMutationError; -use crate::region_manager::RegionReclaimReceipt; -use crate::region_metadata::REGION_METADATA_PAGE_SIZE; -use crate::region_metadata::REGION_METADATA_PARTITIONS_PER_PAGE; -use crate::region_metadata::REGION_METADATA_REGIONS_PER_PAGE; -use crate::region_metadata::RegionMetadataError; -use crate::region_reader::PendingRead; -use crate::region_reader::ReadCandidate; -use crate::region_reader::ReadCompletion; -use crate::region_reader::ReadPlan; +use crate::region::index::packed::IndexEntry; +use crate::region::index::packed::PackedLocation; +use crate::region::index::storage::page_format::INDEX_IMAGE_PAGE_SIZE; +use crate::region::manager::RegionWriteSpan; +use crate::region::record::codec::RecordEncodeError; +use crate::region::record::codec::RecordPayload; +use crate::region::record::codec::encode_reinsert_into_hashed; +use crate::region::record::codec::encode_value_into_hashed; #[cfg(test)] -use crate::region_reader::plan_read; -use crate::region_reader::submit_read; -use crate::region_staging::RegionStaging; -use crate::region_staging::StageAppend; -use crate::region_staging::StagedRecord; -use crate::region_staging::StagedWrite; -use crate::region_staging::StagingEncodeError; -use crate::region_staging::StagingError; +use crate::region::record::codec::hash_key; +use crate::region::recovery::DataGeometry; +use crate::region::recovery::metadata::REGION_METADATA_PAGE_SIZE; +use crate::region::recovery::metadata::REGION_METADATA_PARTITIONS_PER_PAGE; +use crate::region::recovery::metadata::REGION_METADATA_REGIONS_PER_PAGE; +use crate::region::recovery::metadata::RegionMetadataError; +use crate::region::staging::RegionStaging; use crate::resources::BufferLease; use crate::snapshot::CacheIndexSnapshot; use crate::snapshot::RegionSnapshot; -mod file_backend; -pub use self::file_backend::FileRegionBackend; -pub use self::file_backend::RegionFiles; -pub use self::file_backend::SystemRegionFileSystem; +pub mod file_backend; +pub mod index; +pub mod manager; +pub mod record; +pub mod recovery; +pub mod runtime; +pub mod staging; +pub mod store; + +mod appender; +mod reader; const REGION_HEALTHY: u8 = 0; const REGION_MISS_ONLY: u8 = 1; @@ -111,7 +124,7 @@ impl RegionHealthLatch { } } - fn enter_miss_only_with_error(&self, reason: &'static str, error: &impl std::fmt::Display) { + fn enter_miss_only_with_error(&self, reason: &'static str, error: &impl fmt::Display) { if self.transition_to_miss_only() { log::warn!( target: "cache2::health", @@ -363,8 +376,7 @@ impl FileRegionCore { ..RegionReclaimStats::default() }; let alignment = u64::from(RECORD_ALIGNMENT); - let raw_budget = - (receipt.used_offset / 8).saturating_sub(crate::io_backend::DIRECT_IO_ALIGNMENT as u64); + let raw_budget = (receipt.used_offset / 8).saturating_sub(DIRECT_IO_ALIGNMENT as u64); let mut reinsert_budget = raw_budget - raw_budget % alignment; while offset < bytes.len() { let header_end = offset.checked_add(RECORD_HEADER_SIZE).ok_or_else(|| { @@ -518,7 +530,7 @@ impl FileRegionCore { self.health.enter_miss_only(); } - pub fn enter_miss_only_with_error(&self, reason: &'static str, error: &impl std::fmt::Display) { + pub fn enter_miss_only_with_error(&self, reason: &'static str, error: &impl fmt::Display) { self.health.enter_miss_only_with_error(reason, error); } @@ -603,7 +615,7 @@ impl FileRegionCore { fn read_value( &self, engine: &dyn IoEngine, - geometry: crate::recovery::DataGeometry, + geometry: DataGeometry, buffer: BufferLease, hash_seed: u64, key: &[u8], @@ -938,22 +950,21 @@ impl FileRegionCore { staging: &RegionStaging, engine: &dyn IoEngine, shard_id: usize, - ) -> io::Result> { + ) -> io::Result> { let shard_mutation = self.lock_shard_mutation(shard_id)?; let geometry_for = |manager: &RegionManager| { let region_count = u32::try_from(manager.regions().len()).map_err(|_| { self.health.enter_miss_only(); io::Error::new(io::ErrorKind::InvalidData, "Region count is too large") })?; - let data_file_len = crate::recovery::DataGeometry::expected_file_len( - manager.region_size(), - region_count, - ) - .ok_or_else(|| { - self.health.enter_miss_only(); - io::Error::new(io::ErrorKind::InvalidData, "data geometry overflow") - })?; - Ok::<_, io::Error>(crate::recovery::DataGeometry { + let data_file_len = + DataGeometry::expected_file_len(manager.region_size(), region_count).ok_or_else( + || { + self.health.enter_miss_only(); + io::Error::new(io::ErrorKind::InvalidData, "data geometry overflow") + }, + )?; + Ok::<_, io::Error>(DataGeometry { data_file_len, region_size: manager.region_size(), region_count, @@ -1043,7 +1054,7 @@ impl FileRegionCore { } }; let completion = flight.wait(engine); - let crate::region_appender::RegionSpanCompletion { + let RegionSpanCompletion { span, result, buffer, @@ -1157,8 +1168,8 @@ impl FileRegionCore { fn fail_staged_span( &self, staging: &RegionStaging, - span: crate::region_manager::RegionWriteSpan, - buffer: Option, + span: RegionWriteSpan, + buffer: Option, records: Vec, ) { self.health.enter_miss_only(); @@ -1189,7 +1200,7 @@ pub fn runtime_fixed_memory_bytes(index_slots: usize, region_count: u32) -> io:: ) })?; let index_page_state_bytes = (index_bytes / INDEX_IMAGE_PAGE_SIZE) - .checked_mul(std::mem::size_of::()) + .checked_mul(size_of::()) .ok_or_else(|| { io::Error::new( io::ErrorKind::InvalidInput, diff --git a/cache2/src/region_reader.rs b/cache2/src/region/reader.rs similarity index 94% rename from cache2/src/region_reader.rs rename to cache2/src/region/reader.rs index 0288d59..792aa77 100644 --- a/cache2/src/region_reader.rs +++ b/cache2/src/region/reader.rs @@ -22,21 +22,22 @@ use std::io; use std::ops::Range; - -use crate::format::RECORD_ALIGNMENT; -use crate::index::IndexEntry; -use crate::io_engine::BoundedIoRequest; -use crate::io_engine::IoBuffer; -use crate::io_engine::IoCompletion; -use crate::io_engine::IoDeadlineExceeded; -use crate::io_engine::IoEngine; -use crate::io_engine::IoOperation; -use crate::io_engine::OperationKind; -use crate::io_engine::ReadSlot; -use crate::io_engine::RequestId; -use crate::io_engine::submit_cache_read; -use crate::recovery::DATA_REGION_AREA_OFFSET; -use crate::recovery::DataGeometry; +use std::sync::Arc; + +use crate::io::engine::BoundedIoRequest; +use crate::io::engine::IoBuffer; +use crate::io::engine::IoCompletion; +use crate::io::engine::IoDeadlineExceeded; +use crate::io::engine::IoEngine; +use crate::io::engine::IoOperation; +use crate::io::engine::OperationKind; +use crate::io::engine::ReadSlot; +use crate::io::engine::RequestId; +use crate::io::engine::submit_cache_read; +use crate::region::index::packed::IndexEntry; +use crate::region::record::RECORD_ALIGNMENT; +use crate::region::recovery::DATA_REGION_AREA_OFFSET; +use crate::region::recovery::DataGeometry; use crate::resources::BufferLease; const _READ_ALIGNMENT: usize = 4096; @@ -99,7 +100,7 @@ impl PendingRead { pub async fn wait_async( self, - engine: std::sync::Arc, + engine: Arc, tokio_handle: &tokio::runtime::Handle, ) -> ReadCompletion { let Self { @@ -319,12 +320,12 @@ mod tests { use std::sync::Mutex; use super::*; - use crate::index::PackedLocation; - use crate::io_backend::IoBackend; - use crate::io_backend::SyncMode; - use crate::io_backend::SyncPoint; - use crate::io_backend::WritePoint; - use crate::io_engine::BackendIoEngine; + use crate::io::backend::IoBackend; + use crate::io::backend::SyncMode; + use crate::io::backend::SyncPoint; + use crate::io::backend::WritePoint; + use crate::io::engine::BackendIoEngine; + use crate::region::index::packed::PackedLocation; use crate::resources::ResourceController; use crate::resources::ResourceLimits; @@ -380,7 +381,7 @@ mod tests { } } - fn entry(location: crate::index::PackedLocation) -> IndexEntry { + fn entry(location: PackedLocation) -> IndexEntry { IndexEntry { location } } diff --git a/cache2/src/record_codec.rs b/cache2/src/region/record/codec.rs similarity index 95% rename from cache2/src/record_codec.rs rename to cache2/src/region/record/codec.rs index e5b4f03..2183cc4 100644 --- a/cache2/src/record_codec.rs +++ b/cache2/src/region/record/codec.rs @@ -24,15 +24,17 @@ use std::fmt; use hashcrew::xxhash::xxh3_64_with_seed; use crate::checksum::Crc32c; -use crate::format::MAX_KEY_SIZE; -use crate::format::RECORD_ALIGNMENT; -use crate::format::RECORD_HEADER_SIZE; -use crate::format::RecordHeader; -use crate::index::IndexEntry; -use crate::index::MAX_RECORD_LEN; -use crate::index::PackedLocation; -use crate::index::PackedLocationError; -use crate::region_manager::RegionAppendReservation; +#[cfg(test)] +use crate::io::backend::DIRECT_IO_ALIGNMENT; +use crate::region::index::packed::IndexEntry; +use crate::region::index::packed::MAX_RECORD_LEN; +use crate::region::index::packed::PackedLocation; +use crate::region::index::packed::PackedLocationError; +use crate::region::manager::RegionAppendReservation; +use crate::region::record::MAX_KEY_SIZE; +use crate::region::record::RECORD_ALIGNMENT; +use crate::region::record::RECORD_HEADER_SIZE; +use crate::region::record::RecordHeader; #[derive(Clone, Copy, Debug, Eq, PartialEq)] pub enum RecordEncodeError { @@ -301,7 +303,7 @@ mod tests { required_record_bytes(key_len, value_len).unwrap() as usize, expected ); - assert!(!expected.is_multiple_of(crate::io_backend::DIRECT_IO_ALIGNMENT)); + assert!(!expected.is_multiple_of(DIRECT_IO_ALIGNMENT)); } #[test] diff --git a/cache2/src/fixtures/format_v1/value_record.golden b/cache2/src/region/record/format_v1/value_record.golden similarity index 100% rename from cache2/src/fixtures/format_v1/value_record.golden rename to cache2/src/region/record/format_v1/value_record.golden diff --git a/cache2/src/format.rs b/cache2/src/region/record/mod.rs similarity index 98% rename from cache2/src/format.rs rename to cache2/src/region/record/mod.rs index 12e5ef8..e238eea 100644 --- a/cache2/src/format.rs +++ b/cache2/src/region/record/mod.rs @@ -20,6 +20,8 @@ use crate::checksum::Crc32c; use crate::checksum::crc32c; +pub mod codec; + pub const RECORD_FORMAT_VERSION: u16 = 1; pub const RECORD_HEADER_SIZE: usize = 48; @@ -213,10 +215,7 @@ mod tests { let mut encoded = vec![0_u8; record_len as usize]; encoded[..RECORD_HEADER_SIZE].copy_from_slice(&header.encode()); encoded[RECORD_HEADER_SIZE..RECORD_HEADER_SIZE + payload.len()].copy_from_slice(&payload); - let golden = assert_golden( - &encoded, - include_str!("fixtures/format_v1/value_record.golden"), - ); + let golden = assert_golden(&encoded, include_str!("format_v1/value_record.golden")); assert_eq!( RecordHeader::decode(&golden[..RECORD_HEADER_SIZE]), Some(header) diff --git a/cache2/src/fixtures/format_v1/clean_state.golden b/cache2/src/region/recovery/format_v1/clean_state.golden similarity index 100% rename from cache2/src/fixtures/format_v1/clean_state.golden rename to cache2/src/region/recovery/format_v1/clean_state.golden diff --git a/cache2/src/fixtures/format_v1/data_superblock.golden b/cache2/src/region/recovery/format_v1/data_superblock.golden similarity index 100% rename from cache2/src/fixtures/format_v1/data_superblock.golden rename to cache2/src/region/recovery/format_v1/data_superblock.golden diff --git a/cache2/src/fixtures/format_v1/recovery_image_header.golden b/cache2/src/region/recovery/format_v1/recovery_image_header.golden similarity index 100% rename from cache2/src/fixtures/format_v1/recovery_image_header.golden rename to cache2/src/region/recovery/format_v1/recovery_image_header.golden diff --git a/cache2/src/fixtures/format_v1/region_metadata.golden b/cache2/src/region/recovery/format_v1/region_metadata.golden similarity index 100% rename from cache2/src/fixtures/format_v1/region_metadata.golden rename to cache2/src/region/recovery/format_v1/region_metadata.golden diff --git a/cache2/src/region_metadata.rs b/cache2/src/region/recovery/metadata.rs similarity index 94% rename from cache2/src/region_metadata.rs rename to cache2/src/region/recovery/metadata.rs index 3a2d357..01a80f9 100644 --- a/cache2/src/region_metadata.rs +++ b/cache2/src/region/recovery/metadata.rs @@ -19,20 +19,21 @@ //! lazy-validated; this section contains only O(regions + index partitions) state. use std::fmt; +use std::mem; use crate::checksum::Crc32c; -use crate::index::MAX_INDEX_PARTITIONS; -use crate::index::MAX_PACKED_REGION_COUNT; -use crate::index::MAX_PACKED_REGION_SIZE; -use crate::index_storage::INDEX_IMAGE_PAGE_SIZE; -use crate::index_storage::INDEX_IMAGE_SLOTS_PER_PAGE; -use crate::index_storage::IndexStorageError; -use crate::index_storage::canonical_index_partition_ranges; -use crate::index_storage::validated_index_partition_ranges; -use crate::recovery::DataSuperblock; -use crate::recovery::PersistentId; -use crate::recovery::RECOVERY_PAGE_SIZE; -use crate::recovery::RecoveryImageHeader; +use crate::region::index::packed::MAX_INDEX_PARTITIONS; +use crate::region::index::packed::MAX_PACKED_REGION_COUNT; +use crate::region::index::packed::MAX_PACKED_REGION_SIZE; +use crate::region::index::storage::IndexStorageError; +use crate::region::index::storage::canonical_index_partition_ranges; +use crate::region::index::storage::page_format::INDEX_IMAGE_PAGE_SIZE; +use crate::region::index::storage::page_format::INDEX_IMAGE_SLOTS_PER_PAGE; +use crate::region::index::storage::validated_index_partition_ranges; +use crate::region::recovery::DataSuperblock; +use crate::region::recovery::PersistentId; +use crate::region::recovery::RECOVERY_PAGE_SIZE; +use crate::region::recovery::RecoveryImageHeader; pub const REGION_METADATA_PAGE_SIZE: usize = RECOVERY_PAGE_SIZE; const REGION_METADATA_PAGE_HEADER_SIZE: usize = 64; @@ -232,10 +233,8 @@ impl fmt::Display for RegionMetadataError { impl std::error::Error for RegionMetadataError {} -type Result = std::result::Result; - impl RegionMetadata { - pub fn encoded_len(&self) -> Result { + pub fn encoded_len(&self) -> Result { encoded_len_for_counts(self.root.region_count, self.root.partition_count) } @@ -262,7 +261,7 @@ impl RegionMetadata { && encoded_len == image.region_table_len } - pub fn encode(&self) -> Result> { + pub fn encode(&self) -> Result, RegionMetadataError> { self.validate()?; let layout = MetadataLayout::new(self.root.region_count, self.root.partition_count)?; let encoded_len = usize::try_from(layout.encoded_len) @@ -321,7 +320,7 @@ impl RegionMetadata { } #[cfg(test)] - pub fn decode(input: &[u8]) -> Result { + pub fn decode(input: &[u8]) -> Result { let metadata = Self::decode_pages(input)?; metadata.validate()?; Ok(metadata) @@ -329,14 +328,14 @@ impl RegionMetadata { /// Decodes an owned image and releases its encoded pages before allocating /// the queue-validation workspaces used by [`Self::validate`]. - pub fn decode_owned(input: Vec) -> Result { + pub fn decode_owned(input: Vec) -> Result { let metadata = Self::decode_pages(&input)?; drop(input); metadata.validate()?; Ok(metadata) } - fn decode_pages(input: &[u8]) -> Result { + fn decode_pages(input: &[u8]) -> Result { if input.len() < REGION_METADATA_PAGE_SIZE || !input.len().is_multiple_of(REGION_METADATA_PAGE_SIZE) { @@ -428,7 +427,7 @@ impl RegionMetadata { }) } - pub fn validate(&self) -> Result<()> { + pub fn validate(&self) -> Result<(), RegionMetadataError> { let layout = MetadataLayout::new(self.root.region_count, self.root.partition_count)?; validate_root_directory(self.root, layout)?; if self.regions.len() != self.root.region_count as usize { @@ -448,7 +447,7 @@ impl RegionMetadata { /// Shrinking seals excess Active Regions at the back of the sealed FIFO. /// Growing activates Regions from the back of the free FIFO so its existing /// rotation order remains stable. No index or Region data needs rewriting. - pub fn rebind_append_shards(&mut self, shard_count: u32) -> Result<()> { + pub fn rebind_append_shards(&mut self, shard_count: u32) -> Result<(), RegionMetadataError> { let old_shard_count = self.root.shard_count; if shard_count == old_shard_count { return Ok(()); @@ -529,7 +528,7 @@ struct MetadataLayout { } impl MetadataLayout { - fn new(region_count: u32, partition_count: u32) -> Result { + fn new(region_count: u32, partition_count: u32) -> Result { if region_count == 0 || partition_count == 0 { return Err(RegionMetadataError::InvalidField("record_count")); } @@ -561,11 +560,14 @@ impl MetadataLayout { } } -fn encoded_len_for_counts(region_count: u32, partition_count: u32) -> Result { +fn encoded_len_for_counts( + region_count: u32, + partition_count: u32, +) -> Result { Ok(MetadataLayout::new(region_count, partition_count)?.encoded_len) } -fn pages_for_records(records: u64, per_page: u64) -> Result { +fn pages_for_records(records: u64, per_page: u64) -> Result { let pages = records .checked_add(per_page - 1) .ok_or(RegionMetadataError::ArithmeticOverflow)? @@ -573,7 +575,10 @@ fn pages_for_records(records: u64, per_page: u64) -> Result { u32::try_from(pages).map_err(|_| RegionMetadataError::ArithmeticOverflow) } -fn validate_root_directory(root: RegionMetadataRoot, layout: MetadataLayout) -> Result<()> { +fn validate_root_directory( + root: RegionMetadataRoot, + layout: MetadataLayout, +) -> Result<(), RegionMetadataError> { if root.index_slots < 8 { return Err(RegionMetadataError::InvalidField("root")); } @@ -618,7 +623,10 @@ fn validate_root_directory(root: RegionMetadataRoot, layout: MetadataLayout) -> Ok(()) } -fn validate_encoded_root_directory(input: &[u8], layout: MetadataLayout) -> Result<()> { +fn validate_encoded_root_directory( + input: &[u8], + layout: MetadataLayout, +) -> Result<(), RegionMetadataError> { if get_u32(input, ROOT_REGION_FIRST_PAGE_OFFSET)? != layout.region_first_page || get_u32(input, ROOT_REGION_PAGE_COUNT_OFFSET)? != layout.region_page_count || get_u32(input, ROOT_PARTITION_FIRST_PAGE_OFFSET)? != layout.partition_first_page @@ -629,7 +637,10 @@ fn validate_encoded_root_directory(input: &[u8], layout: MetadataLayout) -> Resu Ok(()) } -fn validate_regions(root: RegionMetadataRoot, regions: &[RegionMetadataRecord]) -> Result<()> { +fn validate_regions( + root: RegionMetadataRoot, + regions: &[RegionMetadataRecord], +) -> Result<(), RegionMetadataError> { let mut free_seen = zeroed_bytes(root.free_region_count as usize)?; let mut active_seen = zeroed_bytes(root.active_region_count as usize)?; let mut sealed_seen = zeroed_bytes(root.sealed_region_count as usize)?; @@ -643,7 +654,7 @@ fn validate_regions(root: RegionMetadataRoot, regions: &[RegionMetadataRecord]) RegionMetadataState::Sealed => (&mut sealed_seen, root.sealed_region_count), }; if region.queue_ordinal >= state_count - || std::mem::replace(&mut seen[region.queue_ordinal as usize], 1) != 0 + || mem::replace(&mut seen[region.queue_ordinal as usize], 1) != 0 { return Err(RegionMetadataError::InvalidField("region_queue_ordinal")); } @@ -683,7 +694,7 @@ fn validate_regions(root: RegionMetadataRoot, regions: &[RegionMetadataRecord]) fn validate_partitions( root: RegionMetadataRoot, partitions: &[PartitionMetadataRecord], -) -> Result<()> { +) -> Result<(), RegionMetadataError> { let index_slots = usize::try_from(root.index_slots).map_err(|_| RegionMetadataError::ArithmeticOverflow)?; let canonical = @@ -733,14 +744,14 @@ fn index_layout_metadata_error(error: IndexStorageError) -> RegionMetadataError } } -fn minimum_bytes_fit(count: u64, bytes: u64) -> Result { +fn minimum_bytes_fit(count: u64, bytes: u64) -> Result { Ok(count .checked_mul(MIN_ENCODED_RECORD_SIZE) .ok_or(RegionMetadataError::ArithmeticOverflow)? <= bytes) } -fn zeroed_bytes(len: usize) -> Result> { +fn zeroed_bytes(len: usize) -> Result, RegionMetadataError> { let mut output = Vec::new(); output .try_reserve_exact(len) @@ -785,7 +796,7 @@ fn encode_page_envelope(page: &mut [u8], envelope: PageEnvelope) { put_u32(page, PAGE_RESERVED_OFFSET, 0); } -fn decode_page_envelope(page: &[u8]) -> Result { +fn decode_page_envelope(page: &[u8]) -> Result { if page.len() != REGION_METADATA_PAGE_SIZE { return Err(RegionMetadataError::InvalidLength); } @@ -831,7 +842,7 @@ fn validate_envelope_shape( page_index: u32, first_record: u32, record_count: u32, -) -> Result<()> { +) -> Result<(), RegionMetadataError> { if envelope.kind != kind || usize::from(envelope.record_size) != record_size || envelope.image_generation == 0 @@ -869,7 +880,7 @@ fn encode_record_pages( image_generation: u64, records: &[T], encode_record: fn(&T, &mut [u8]), -) -> Result<()> { +) -> Result<(), RegionMetadataError> { for (page_in_section, records) in records.chunks(records_per_page).enumerate() { let first_record = page_in_section .checked_mul(records_per_page) @@ -913,8 +924,8 @@ fn decode_record_pages( image_identity: PersistentId, image_generation: u64, record_count: usize, - decode_record: fn(&[u8]) -> Result, -) -> Result> { + decode_record: fn(&[u8]) -> Result, +) -> Result, RegionMetadataError> { let mut output = Vec::new(); output .try_reserve_exact(record_count) @@ -949,7 +960,7 @@ fn decode_record_pages( Ok(output) } -fn require_zero_padding(page: &[u8], payload_len: usize) -> Result<()> { +fn require_zero_padding(page: &[u8], payload_len: usize) -> Result<(), RegionMetadataError> { let end = REGION_METADATA_PAGE_HEADER_SIZE .checked_add(payload_len) .ok_or(RegionMetadataError::ArithmeticOverflow)?; @@ -1024,7 +1035,7 @@ fn encode_root(root: &RegionMetadataRoot, layout: MetadataLayout, output: &mut [ put_u32(output, ROOT_RESERVED_OFFSET, 0); } -fn decode_root(input: &[u8]) -> Result { +fn decode_root(input: &[u8]) -> Result { if input.len() != REGION_METADATA_ROOT_SIZE || get_u32(input, ROOT_RESERVED32_OFFSET)? != 0 || get_u64(input, ROOT_RESERVED_EPOCH_OFFSET)? != 0 @@ -1071,7 +1082,7 @@ fn encode_region(region: &RegionMetadataRecord, output: &mut [u8]) { output[REGION_STATE_OFFSET] = region.state as u8; } -fn decode_region(input: &[u8]) -> Result { +fn decode_region(input: &[u8]) -> Result { if input.len() != REGION_METADATA_REGION_SIZE { return Err(RegionMetadataError::InvalidField("region_encoding")); } @@ -1104,7 +1115,7 @@ fn encode_partition(partition: &PartitionMetadataRecord, output: &mut [u8]) { put_u32(output, PARTITION_RESERVED_OFFSET, 0); } -fn decode_partition(input: &[u8]) -> Result { +fn decode_partition(input: &[u8]) -> Result { if input.len() != REGION_METADATA_PARTITION_SIZE || get_u32(input, PARTITION_RESERVED_OFFSET)? != 0 { @@ -1117,7 +1128,7 @@ fn decode_partition(input: &[u8]) -> Result { }) } -fn page(input: &[u8], page_index: usize) -> Result<&[u8]> { +fn page(input: &[u8], page_index: usize) -> Result<&[u8], RegionMetadataError> { let start = page_index .checked_mul(REGION_METADATA_PAGE_SIZE) .ok_or(RegionMetadataError::ArithmeticOverflow)?; @@ -1129,7 +1140,7 @@ fn page(input: &[u8], page_index: usize) -> Result<&[u8]> { .ok_or(RegionMetadataError::InvalidLength) } -fn page_mut(input: &mut [u8], page_index: usize) -> Result<&mut [u8]> { +fn page_mut(input: &mut [u8], page_index: usize) -> Result<&mut [u8], RegionMetadataError> { let start = page_index .checked_mul(REGION_METADATA_PAGE_SIZE) .ok_or(RegionMetadataError::ArithmeticOverflow)?; @@ -1151,7 +1162,7 @@ fn page_payload_mut(page: &mut [u8], record: usize, record_size: usize) -> &mut &mut page[start..start + record_size] } -fn get_id(input: &[u8], offset: usize) -> Result { +fn get_id(input: &[u8], offset: usize) -> Result { let bytes: [u8; 16] = input .get(offset..offset + 16) .ok_or(RegionMetadataError::InvalidLength)? @@ -1164,7 +1175,7 @@ fn put_id(output: &mut [u8], offset: usize, id: PersistentId) { output[offset..offset + 16].copy_from_slice(&id.to_bytes()); } -fn get_u16(input: &[u8], offset: usize) -> Result { +fn get_u16(input: &[u8], offset: usize) -> Result { let bytes = input .get(offset..offset + 2) .ok_or(RegionMetadataError::InvalidLength)? @@ -1173,7 +1184,7 @@ fn get_u16(input: &[u8], offset: usize) -> Result { Ok(u16::from_le_bytes(bytes)) } -fn get_u32(input: &[u8], offset: usize) -> Result { +fn get_u32(input: &[u8], offset: usize) -> Result { let bytes = input .get(offset..offset + 4) .ok_or(RegionMetadataError::InvalidLength)? @@ -1182,7 +1193,7 @@ fn get_u32(input: &[u8], offset: usize) -> Result { Ok(u32::from_le_bytes(bytes)) } -fn get_u64(input: &[u8], offset: usize) -> Result { +fn get_u64(input: &[u8], offset: usize) -> Result { let bytes = input .get(offset..offset + 8) .ok_or(RegionMetadataError::InvalidLength)? @@ -1353,10 +1364,7 @@ mod tests { fn complete_metadata_matches_committed_golden_bytes() { let expected = sample(); let encoded = expected.encode().unwrap(); - let golden = assert_golden( - &encoded, - include_str!("fixtures/format_v1/region_metadata.golden"), - ); + let golden = assert_golden(&encoded, include_str!("format_v1/region_metadata.golden")); assert_eq!(RegionMetadata::decode(&golden).unwrap(), expected); } diff --git a/cache2/src/recovery.rs b/cache2/src/region/recovery/mod.rs similarity index 98% rename from cache2/src/recovery.rs rename to cache2/src/region/recovery/mod.rs index 3116a32..ac40b42 100644 --- a/cache2/src/recovery.rs +++ b/cache2/src/region/recovery/mod.rs @@ -22,12 +22,14 @@ use crate::checksum::Crc32c; use crate::checksum::crc32c; -use crate::format::RECORD_ALIGNMENT; -use crate::format::RECORD_FORMAT_VERSION; -use crate::index::MAX_PACKED_REGION_COUNT; -use crate::index::MAX_PACKED_REGION_SIZE; -use crate::index_storage::INDEX_IMAGE_PAGE_SIZE; -use crate::index_storage::INDEX_IMAGE_SLOTS_PER_PAGE; +use crate::region::index::packed::MAX_PACKED_REGION_COUNT; +use crate::region::index::packed::MAX_PACKED_REGION_SIZE; +use crate::region::index::storage::page_format::INDEX_IMAGE_PAGE_SIZE; +use crate::region::index::storage::page_format::INDEX_IMAGE_SLOTS_PER_PAGE; +use crate::region::record::RECORD_ALIGNMENT; +use crate::region::record::RECORD_FORMAT_VERSION; + +pub mod metadata; const RECOVERY_FORMAT_VERSION: u16 = 1; pub const RECOVERY_PAGE_SIZE: usize = 4 * 1024; @@ -1009,7 +1011,7 @@ mod tests { let data = data_superblock(); let data_golden = assert_golden( &data.encode().unwrap(), - include_str!("fixtures/format_v1/data_superblock.golden"), + include_str!("format_v1/data_superblock.golden"), ); assert_eq!( DataSuperblock::probe(&data_golden), @@ -1022,7 +1024,7 @@ mod tests { let clean = record(19, RecoveryState::Clean); let clean_golden = assert_golden( &clean.encode().unwrap(), - include_str!("fixtures/format_v1/clean_state.golden"), + include_str!("format_v1/clean_state.golden"), ); assert_eq!(StateRecord::decode(&clean_golden), Some(clean)); } @@ -1033,7 +1035,7 @@ mod tests { let header = image_header(); let image_golden = assert_golden( &header.encode().unwrap(), - include_str!("fixtures/format_v1/recovery_image_header.golden"), + include_str!("format_v1/recovery_image_header.golden"), ); assert_eq!( RecoveryImageHeader::probe(&image_golden), diff --git a/cache2/src/region_runtime/metrics.rs b/cache2/src/region/runtime/metrics.rs similarity index 96% rename from cache2/src/region_runtime/metrics.rs rename to cache2/src/region/runtime/metrics.rs index 68988bd..1d4adfc 100644 --- a/cache2/src/region_runtime/metrics.rs +++ b/cache2/src/region/runtime/metrics.rs @@ -13,14 +13,17 @@ // limitations under the License. use std::io; +use std::sync::atomic::AtomicU8; use std::sync::atomic::AtomicU64; use std::sync::atomic::Ordering; use std::time::Duration; -use super::LIFECYCLE_DRAINING; -use super::LIFECYCLE_FAILED; use crate::hashing::route_hash; use crate::memory::MemoryMetricsSnapshot; +use crate::region::RegionReclaimStats; +use crate::region::runtime::LIFECYCLE_DRAINING; +use crate::region::runtime::LIFECYCLE_FAILED; +use crate::region::runtime::LIFECYCLE_RUNNING; use crate::resources::ManagedMemorySnapshot; use crate::snapshot::CacheHealth; use crate::snapshot::CacheIoSnapshot; @@ -31,7 +34,7 @@ static NEXT_METRICS_EPOCH: AtomicU64 = AtomicU64::new(1); pub struct RuntimeMetrics { metrics_epoch: u64, - pub lifecycle: std::sync::atomic::AtomicU8, + pub lifecycle: AtomicU8, activity: Box<[ActivityMetrics]>, l2_read_overloads: AtomicU64, l2_read_wait_ns: AtomicU64, @@ -94,7 +97,7 @@ impl RuntimeMetrics { activity.resize_with(shard_count, ActivityMetrics::new); Ok(Self { metrics_epoch: NEXT_METRICS_EPOCH.fetch_add(1, Ordering::Relaxed), - lifecycle: std::sync::atomic::AtomicU8::new(super::LIFECYCLE_RUNNING), + lifecycle: AtomicU8::new(LIFECYCLE_RUNNING), activity: activity.into_boxed_slice(), l2_read_overloads: AtomicU64::new(0), l2_read_wait_ns: AtomicU64::new(0), @@ -143,7 +146,7 @@ impl RuntimeMetrics { self.l2_read_wait_ns.fetch_add(nanos, Ordering::Relaxed); } - pub fn record_reclaim(&self, stats: crate::region::RegionReclaimStats) { + pub fn record_reclaim(&self, stats: RegionReclaimStats) { Self::increment(&self.reclaimed_regions); self.reclaimed_bytes .fetch_add(stats.bytes_read, Ordering::Relaxed); diff --git a/cache2/src/region_runtime.rs b/cache2/src/region/runtime/mod.rs similarity index 96% rename from cache2/src/region_runtime.rs rename to cache2/src/region/runtime/mod.rs index 2f0e6e0..182155c 100644 --- a/cache2/src/region_runtime.rs +++ b/cache2/src/region/runtime/mod.rs @@ -21,9 +21,13 @@ //! a durability sync; CLEAN remains the only steady-state durability boundary. use std::io; +use std::mem; +use std::panic; +use std::panic::AssertUnwindSafe; use std::sync::Arc; use std::sync::Condvar; use std::sync::Mutex; +use std::sync::MutexGuard; use std::sync::atomic::AtomicU8; use std::sync::atomic::AtomicUsize; use std::sync::atomic::Ordering; @@ -36,53 +40,62 @@ use asyncband::semaphore::Semaphore; use asyncband::watch; use self::metrics::RuntimeMetrics; +use crate::IoEngineConfig; use crate::config::CacheConfig; -use crate::config::IoMode; -use crate::config::IoPoolTopology; -#[cfg(test)] -use crate::config::ReadAdmission; -use crate::config::RuntimeOptions; use crate::config::l1_entry_capacity; -use crate::config::read_io_wait_capacity; -use crate::config::read_io_wait_timeout; use crate::config::reserved_memory_bytes; +use crate::config::runtime::IoMode; +use crate::config::runtime::IoPoolTopology; +#[cfg(test)] +use crate::config::runtime::ReadAdmission; +use crate::config::runtime::RuntimeOptions; +use crate::config::runtime::read_io_wait_capacity; +use crate::config::runtime::read_io_wait_timeout; use crate::config::storage_geometry; -use crate::format::MAX_KEY_SIZE; use crate::hashing::route_hash; -#[cfg(test)] -use crate::index_storage::INDEX_IMAGE_PAGE_SIZE; -#[cfg(test)] -use crate::index_storage::INDEX_IMAGE_SLOTS_PER_PAGE; -use crate::io_backend::RuntimeFileSet; -use crate::io_engine::IoBuffer; -use crate::io_engine::IoEngine; -use crate::io_engine::IoOperation; -use crate::io_engine::ReadSlot; -use crate::io_engine::ReadSlotWaiter; -use crate::io_engine::build_file_engine; -use crate::io_engine::submit_cache_io; +use crate::io::backend::RuntimeFileSet; +use crate::io::engine::IoBuffer; +use crate::io::engine::IoEngine; +use crate::io::engine::IoOperation; +use crate::io::engine::ReadSlot; +use crate::io::engine::ReadSlotWaiter; +use crate::io::engine::build_file_engine; +use crate::io::engine::submit_cache_io; use crate::memory::MemoryLookup; #[cfg(test)] use crate::memory::MemoryMetricsSnapshot; use crate::memory::MemoryReadToken; use crate::memory::MemoryStore; use crate::memory::MemoryValue; -use crate::record_codec::hash_key; -use crate::record_codec::required_record_bytes; -#[cfg(test)] -use crate::recovery::DataGeometry; -use crate::recovery::DataSuperblock; use crate::region::FileRegionCore; use crate::region::RegionStageValue; use crate::region::RegionValueRead; #[cfg(test)] +use crate::region::index::packed::IndexEntry; +#[cfg(test)] +use crate::region::index::packed::PackedLocation; +#[cfg(test)] +use crate::region::index::storage::page_format::INDEX_IMAGE_PAGE_SIZE; +#[cfg(test)] +use crate::region::index::storage::page_format::INDEX_IMAGE_SLOTS_PER_PAGE; +use crate::region::reader::PendingRead; +#[cfg(test)] +use crate::region::reader::ReadCandidate; +use crate::region::reader::ReadCompletion; +use crate::region::reader::ReadPlan; +use crate::region::reader::plan_read; +use crate::region::record::MAX_KEY_SIZE; +#[cfg(test)] +use crate::region::record::RECORD_HEADER_SIZE; +use crate::region::record::codec::hash_key; +use crate::region::record::codec::required_record_bytes; +#[cfg(test)] +use crate::region::recovery::DataGeometry; +use crate::region::recovery::DataSuperblock; +#[cfg(test)] use crate::region::runtime_fixed_memory_bytes; -use crate::region_reader::PendingRead; -use crate::region_reader::ReadCompletion; -use crate::region_reader::ReadPlan; -use crate::region_reader::plan_read; -use crate::region_staging::RegionStaging; -use crate::region_staging::StagingError; +use crate::region::staging::RegionStaging; +use crate::region::staging::StagingError; use crate::resources::BufferLease; use crate::resources::CACHE_THREAD_STACK_BYTES; #[cfg(test)] @@ -95,8 +108,7 @@ use crate::snapshot::CacheIoSnapshot; use crate::snapshot::CacheSnapshot; use crate::snapshot::DetailedCacheSnapshot; -mod metrics; -pub use self::metrics::ActivityMetrics; +pub mod metrics; const WRITE_FLUSH_DELAY: Duration = Duration::from_millis(1); const _RETRY_AGE: Duration = Duration::from_micros(50); @@ -718,7 +730,7 @@ impl ShardControl { self.async_changed.send_replace(()); } - fn lock(&self) -> io::Result> { + fn lock(&self) -> io::Result> { self.state.lock().map_err(|_| poisoned_runtime_error()) } } @@ -1287,7 +1299,7 @@ impl RegionDataPlane { #[cfg(test)] pub fn poison_shard_for_test(&self, shard_id: usize) { let shard = self.shared.shards.get(shard_id).expect("test shard exists"); - let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { + let result = panic::catch_unwind(AssertUnwindSafe(|| { let _state = shard.state.lock().unwrap(); panic!("poison shard gate"); })); @@ -1535,7 +1547,7 @@ fn build_engine_pool( engines .try_reserve_exact(engine_count) .map_err(|_| io::Error::new(io::ErrorKind::OutOfMemory, "cannot allocate I/O workers"))?; - let posix_workers = if matches!(config.io_engine, crate::config::IoEngine::Posix(_)) { + let posix_workers = if matches!(config.io_engine, IoEngineConfig::Posix(_)) { topology.max_in_flight } else { 1 @@ -1561,7 +1573,7 @@ fn build_engine_pool( fn shard_worker(shared: Arc, shard_id: usize) { let control = Arc::clone(&shared.shards[shard_id]); - let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { + let result = panic::catch_unwind(AssertUnwindSafe(|| { shard_worker_result(&shared, shard_id, &control) })); let error = match result { @@ -1637,7 +1649,7 @@ fn reclaim_worker( worker_count: usize, ) { let mut buffer = Some(buffer); - let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { + let result = panic::catch_unwind(AssertUnwindSafe(|| { reclaim_worker_result(&shared, &mut buffer, worker_id, worker_count) })); let error = match result { @@ -1913,7 +1925,7 @@ fn wait_for_shard_work( if let Some(failure) = &state.failure { return Err(failure.to_error()); } - let flags = std::mem::take(&mut state.wake_flags); + let flags = mem::take(&mut state.wake_flags); let drain_generation = if state.drain_requested > state.drain_completed { state.drain_requested } else { @@ -2065,7 +2077,7 @@ fn stop_running(mut owner: RunningOwner) -> io::Result { // has no trustworthy future fence and remains process-lifetime state. if unfenced { for engine in owner.shared.engines() { - std::mem::forget(Arc::clone(engine)); + mem::forget(Arc::clone(engine)); } } else { for engine in owner.shared.engines() { @@ -2097,7 +2109,7 @@ fn reap_engine_after_target_fence(engine: &Arc) { // The original owner is still alive while this fallback clone is // created, so a failed thread spawn cannot synchronously run the // engine's blocking Drop path. - std::mem::forget(Arc::clone(engine)); + mem::forget(Arc::clone(engine)); } } @@ -2145,6 +2157,7 @@ fn invalid_runtime_config(message: &'static str) -> io::Error { #[cfg(test)] mod tests { + use std::env; use std::sync::Barrier; use std::sync::atomic::AtomicBool; use std::sync::atomic::AtomicU64; @@ -2153,9 +2166,9 @@ mod tests { use std::task::Waker; use super::*; - use crate::io_backend::FileBackend; - use crate::io_backend::IoBackend; - use crate::io_engine::BackendIoEngine; + use crate::io::backend::FileBackend; + use crate::io::backend::IoBackend; + use crate::io::engine::BackendIoEngine; static LANE_TEST_ID: AtomicU64 = AtomicU64::new(1); @@ -2174,7 +2187,7 @@ mod tests { #[test] fn read_lane_uses_one_bounded_alternate_on_primary_pressure() { let id = LANE_TEST_ID.fetch_add(1, Ordering::Relaxed); - let path = std::env::temp_dir().join(format!( + let path = env::temp_dir().join(format!( "cache2-read-lane-{}-{id}.cache", std::process::id() )); @@ -2216,7 +2229,7 @@ mod tests { #[test] fn hot_read_route_rotates_pressure_fallback_across_all_lanes() { let id = LANE_TEST_ID.fetch_add(1, Ordering::Relaxed); - let path = std::env::temp_dir().join(format!( + let path = env::temp_dir().join(format!( "cache2-read-lane-rotation-{}-{id}.cache", std::process::id() )); @@ -2450,18 +2463,18 @@ mod tests { #[test] fn completion_timeouts_follow_read_wait_mode() { - use crate::config::IoEngine; - use crate::config::PosixIoConfig; - use crate::index::IndexEntry; - use crate::index::PackedLocation; - use crate::recovery::DATA_REGION_AREA_OFFSET; - use crate::recovery::PersistentId; - use crate::region::FileRegionBackend; - use crate::region::RegionFiles; - use crate::region_store::RegionStore; + use crate::config::runtime::IoEngineConfig; + use crate::config::runtime::PosixIoConfig; + use crate::region::file_backend::FileRegionBackend; + use crate::region::file_backend::RegionFiles; + use crate::region::index::packed::IndexEntry; + use crate::region::index::packed::PackedLocation; + use crate::region::recovery::DATA_REGION_AREA_OFFSET; + use crate::region::recovery::PersistentId; + use crate::region::store::RegionStore; let id = LANE_TEST_ID.fetch_add(1, Ordering::Relaxed); - let path = std::env::temp_dir().join(format!( + let path = env::temp_dir().join(format!( "cache2-completion-timeout-{}-{id}", std::process::id() )); @@ -2484,7 +2497,7 @@ mod tests { }; for wait in [Duration::ZERO, Duration::from_millis(1)] { let config = RuntimeOptions { - io_engine: IoEngine::Posix(PosixIoConfig::new(1, 1, 1)), + io_engine: IoEngineConfig::Posix(PosixIoConfig::new(1, 1, 1)), append_shards: 1, l1_capacity_bytes: 0, statistics: true, @@ -2552,17 +2565,17 @@ mod tests { region_size: 512 * 1024, region_count: 10, }; - let value_len = geometry.region_size as usize - crate::format::RECORD_HEADER_SIZE; + let value_len = geometry.region_size as usize - RECORD_HEADER_SIZE; let record_len = required_record_bytes(0, value_len).unwrap(); assert_eq!(u64::from(record_len), geometry.region_size); - let entry = crate::index::IndexEntry { - location: crate::index::PackedLocation::new(0, 0, record_len).unwrap(), + let entry = IndexEntry { + location: PackedLocation::new(0, 0, record_len).unwrap(), }; assert_eq!( plan_read( geometry, 1, - crate::region_reader::ReadCandidate { + ReadCandidate { entry, region_generation: 1, }, diff --git a/cache2/src/region_runtime/shutdown_tests.rs b/cache2/src/region/runtime/shutdown_tests.rs similarity index 89% rename from cache2/src/region_runtime/shutdown_tests.rs rename to cache2/src/region/runtime/shutdown_tests.rs index 83d9451..c8c0a50 100644 --- a/cache2/src/region_runtime/shutdown_tests.rs +++ b/cache2/src/region/runtime/shutdown_tests.rs @@ -12,20 +12,23 @@ // See the License for the specific language governing permissions and // limitations under the License. +use std::env; use std::sync::atomic::AtomicBool; +use std::sync::mpsc; use super::*; -use crate::io_backend::IoBackend; -use crate::io_backend::SyncMode; -use crate::io_backend::SyncPoint; -use crate::io_backend::WritePoint; -use crate::io_engine::BackendIoEngine; -use crate::io_engine::CompletionState; -use crate::io_engine::EngineIoSnapshot; -use crate::io_engine::IoRequest; -use crate::io_engine::ReadSlotWaiter; -use crate::io_engine::RequestId; -use crate::io_engine::SubmitError; +use crate::IoEngineConfig; +use crate::io::backend::IoBackend; +use crate::io::backend::SyncMode; +use crate::io::backend::SyncPoint; +use crate::io::backend::WritePoint; +use crate::io::engine::BackendIoEngine; +use crate::io::engine::CompletionState; +use crate::io::engine::EngineIoSnapshot; +use crate::io::engine::IoRequest; +use crate::io::engine::ReadSlotWaiter; +use crate::io::engine::RequestId; +use crate::io::engine::SubmitError; #[derive(Default)] struct BlockedReadState { @@ -202,12 +205,12 @@ fn submitted_read_must_not_pin_close() { } fn assert_close_does_not_wait_for_read(submit_before_close: bool) { - use crate::config::PosixIoConfig; - use crate::recovery::PersistentId; - use crate::region::FileRegionBackend; - use crate::region::RegionFiles; - use crate::region_store::RegionStore; - let root = std::env::temp_dir().join(format!( + use crate::config::runtime::PosixIoConfig; + use crate::region::file_backend::FileRegionBackend; + use crate::region::file_backend::RegionFiles; + use crate::region::recovery::PersistentId; + use crate::region::store::RegionStore; + let root = env::temp_dir().join(format!( "cache2-close-race-{}-{submit_before_close}", std::process::id() )); @@ -228,7 +231,7 @@ fn assert_close_does_not_wait_for_read(submit_before_close: bool) { let config = RuntimeOptions { append_shards: 1, l1_capacity_bytes: 0, - io_engine: crate::config::IoEngine::Posix(PosixIoConfig::new(1, 1, 1)), + io_engine: IoEngineConfig::Posix(PosixIoConfig::new(1, 1, 1)), ..RuntimeOptions::default() }; let mut store = RegionStore::open( @@ -257,7 +260,7 @@ fn assert_close_does_not_wait_for_read(submit_before_close: bool) { shared.reclaim_engines = Box::new([]); shared.shards = Box::new([]); let shared = Arc::clone(&plane.shared); - let (tx, rx) = std::sync::mpsc::channel(); + let (tx, rx) = mpsc::channel(); let thread = std::thread::spawn(move || { let result = stop_running(RunningOwner { shared, diff --git a/cache2/src/region_staging.rs b/cache2/src/region/staging.rs similarity index 98% rename from cache2/src/region_staging.rs rename to cache2/src/region/staging.rs index b2ac60b..1563a3b 100644 --- a/cache2/src/region_staging.rs +++ b/cache2/src/region/staging.rs @@ -17,22 +17,23 @@ //! Region manager receipts are the only span authority. use std::fmt; +use std::mem; use std::sync::Mutex; use std::sync::MutexGuard; -use crate::format::RECORD_ALIGNMENT; -use crate::format::RECORD_HEADER_SIZE; -use crate::format::RecordHeader; -use crate::index::IndexEntry; -use crate::index::MAX_RECORD_LEN; -use crate::index::PackedLocation; -use crate::io_backend::DIRECT_IO_ALIGNMENT; -use crate::io_engine::IoBuffer; -use crate::recovery::DATA_REGION_AREA_OFFSET; -use crate::recovery::RECOVERY_PAGE_SIZE; -use crate::region_manager::RegionAppendReservation; -use crate::region_manager::RegionPaddingReceipt; -use crate::region_manager::RegionWriteSpan; +use crate::io::backend::DIRECT_IO_ALIGNMENT; +use crate::io::engine::IoBuffer; +use crate::region::index::packed::IndexEntry; +use crate::region::index::packed::MAX_RECORD_LEN; +use crate::region::index::packed::PackedLocation; +use crate::region::manager::RegionAppendReservation; +use crate::region::manager::RegionPaddingReceipt; +use crate::region::manager::RegionWriteSpan; +use crate::region::record::RECORD_ALIGNMENT; +use crate::region::record::RECORD_HEADER_SIZE; +use crate::region::record::RecordHeader; +use crate::region::recovery::DATA_REGION_AREA_OFFSET; +use crate::region::recovery::RECOVERY_PAGE_SIZE; use crate::resources::BUFFER_ALIGNMENT; use crate::resources::BufferLease; use crate::resources::ResourceBuildError; @@ -259,7 +260,7 @@ impl RegionStaging { pub fn reservation_bytes(shard_count: usize, chunk_bytes: usize) -> Option { let buffers_per_shard = chunk_bytes.checked_mul(2)?; let records_per_shard = MAX_STAGING_RECORDS - .checked_mul(std::mem::size_of::())? + .checked_mul(size_of::())? .checked_mul(2)?; buffers_per_shard .checked_add(records_per_shard)? @@ -713,7 +714,7 @@ impl RegionStaging { state.failed = true; StagingError::Invariant("staging lost its second record vector") })?; - let records = std::mem::replace(&mut state.fill.records, replacement_records); + let records = mem::replace(&mut state.fill.records, replacement_records); state.fill.reset(replacement_buffer); state.submitted = Some(span); drop(state); @@ -904,7 +905,7 @@ mod tests { use std::time::Duration; use super::*; - use crate::index::PackedLocation; + use crate::region::index::packed::PackedLocation; use crate::resources::ResourceLimits; fn resources(memory_limit_bytes: usize) -> ResourceController { @@ -975,13 +976,13 @@ mod tests { #[test] fn seal_moves_the_aligned_fill_lease_and_keeps_filling_the_second_buffer() { - assert_eq!(std::mem::size_of::(), 32); + assert_eq!(size_of::(), 32); let resources = resources(4 * 1024 * 1024); let staging = RegionStaging::try_new(1, 4096, 64 * 1024, &resources).unwrap(); assert_eq!(staging.chunk_bytes(), 4096); assert_eq!( resources.managed_memory_snapshot().current_bytes, - 2 * 4096 + 2 * MAX_STAGING_RECORDS * std::mem::size_of::() + 2 * 4096 + 2 * MAX_STAGING_RECORDS * size_of::() ); let (first, first_record) = reservation(0, 64, 11); diff --git a/cache2/src/region_store.rs b/cache2/src/region/store.rs similarity index 99% rename from cache2/src/region_store.rs rename to cache2/src/region/store.rs index f67b641..37bfe87 100644 --- a/cache2/src/region_store.rs +++ b/cache2/src/region/store.rs @@ -25,7 +25,7 @@ use std::io; -use crate::index_storage::validated_index_partition_ranges; +use crate::region::index::storage::validated_index_partition_ranges; use crate::snapshot::StartupMode; /// Result of inspecting the latest valid state record. diff --git a/cache2/src/resources.rs b/cache2/src/resources.rs index 22fb429..da47262 100644 --- a/cache2/src/resources.rs +++ b/cache2/src/resources.rs @@ -23,6 +23,7 @@ use std::alloc::alloc; use std::alloc::dealloc; use std::fmt; use std::ptr::NonNull; +use std::slice; use std::sync::Arc; use std::sync::atomic::AtomicUsize; use std::sync::atomic::Ordering; @@ -207,7 +208,7 @@ impl BufferLease { } // SAFETY: the allocation holds `initialized` initialized bytes and the // returned shared slice cannot mutate the exclusively leased buffer. - Ok(unsafe { std::slice::from_raw_parts(buffer.ptr.as_ptr(), length) }) + Ok(unsafe { slice::from_raw_parts(buffer.ptr.as_ptr(), length) }) } pub fn prepared_mut(&mut self, length: usize) -> Result<&mut [u8], ()> { @@ -306,7 +307,7 @@ impl AlignedBuffer { debug_assert!(length <= self.initialized); // SAFETY: the allocation holds `initialized` initialized bytes, this // mutable borrow is exclusive, and `length <= initialized`. - unsafe { std::slice::from_raw_parts_mut(self.ptr.as_ptr(), length) } + unsafe { slice::from_raw_parts_mut(self.ptr.as_ptr(), length) } } fn deallocate(&mut self) { diff --git a/tests-integration/tests/cache.rs b/tests-integration/tests/cache.rs index 8547b11..039f99f 100644 --- a/tests-integration/tests/cache.rs +++ b/tests-integration/tests/cache.rs @@ -12,10 +12,15 @@ // See the License for the specific language governing permissions and // limitations under the License. +use std::env; +use std::fs; +use std::fs::OpenOptions; +use std::io; use std::io::Read; use std::io::Seek; use std::io::SeekFrom; use std::io::Write; +use std::path::Path; use std::path::PathBuf; use std::sync::Arc; use std::sync::Barrier; @@ -23,19 +28,23 @@ use std::sync::atomic::AtomicBool; use std::sync::atomic::AtomicU64; use std::sync::atomic::AtomicUsize; use std::sync::atomic::Ordering; -use std::thread; use std::time::Duration; use std::time::Instant; use cache2::Cache; use cache2::CacheConfig; use cache2::CacheHealth; +use cache2::CacheIoSnapshot; use cache2::CacheTier; +use cache2::DetailedCacheSnapshot; +use cache2::Error; use cache2::ErrorKind; use cache2::ErrorOperation; -use cache2::IoEngine; +use cache2::IoEngineConfig; #[cfg(not(target_os = "linux"))] use cache2::IoMode; +#[cfg(not(target_os = "linux"))] +use cache2::IoUringConfig; use cache2::L1EvictionPolicy; use cache2::PosixIoConfig; use cache2::ReadAdmission; @@ -58,7 +67,7 @@ fn test_storage() -> StorageLayout { fn test_runtime_options(workers: usize, append_shards: u32) -> RuntimeOptions { RuntimeOptions { - io_engine: IoEngine::Posix(PosixIoConfig::new(workers, workers, 1)), + io_engine: IoEngineConfig::Posix(PosixIoConfig::new(workers, workers, 1)), append_shards, l1_capacity_bytes: 4 * 1024 * 1024, managed_memory_limit_bytes: 32 * 1024 * 1024, @@ -91,8 +100,7 @@ struct TestCache { impl TestCache { fn new(name: &str) -> Self { let id = NEXT_FILE.fetch_add(1, Ordering::Relaxed); - let data = - std::env::temp_dir().join(format!("cache2-{name}-{}-{id}.cache", std::process::id())); + let data = env::temp_dir().join(format!("cache2-{name}-{}-{id}.cache", std::process::id())); Self { data } } @@ -120,17 +128,17 @@ impl Drop for TestCache { self.sidecar(".image"), self.sidecar(".image.next"), ] { - let _ = std::fs::remove_file(path); + let _ = fs::remove_file(path); } } } -fn rewrite_page_version(path: &std::path::Path, offset: u64, version: u16) { +fn rewrite_page_version(path: &Path, offset: u64, version: u16) { const PAGE_BYTES: usize = 4096; const VERSION_OFFSET: usize = 8; const CRC_OFFSET: usize = PAGE_BYTES - 4; - let mut file = std::fs::OpenOptions::new() + let mut file = OpenOptions::new() .read(true) .write(true) .open(path) @@ -147,7 +155,7 @@ fn rewrite_page_version(path: &std::path::Path, offset: u64, version: u16) { file.sync_all().unwrap(); } -fn eventually_admitted(mut put: impl FnMut() -> cache2::Result) -> T { +fn eventually_admitted(mut put: impl FnMut() -> Result) -> T { let deadline = Instant::now() + Duration::from_secs(2); loop { match put() { @@ -164,7 +172,7 @@ fn eventually_admitted(mut put: impl FnMut() -> cache2::Result) -> T { } } -async fn completed_reclaim_snapshot(cache: &cache2::Cache) -> cache2::DetailedCacheSnapshot { +async fn completed_reclaim_snapshot(cache: &Cache) -> DetailedCacheSnapshot { for ordinal in 0_u64..128 { eventually_admitted(|| cache.put(ordinal.to_le_bytes(), vec![ordinal as u8; 8 * 1024])); } @@ -193,7 +201,7 @@ async fn completed_reclaim_snapshot(cache: &cache2::Cache) -> cache2::DetailedCa return detailed; } assert!(Instant::now() < deadline, "reclaim did not make progress"); - thread::yield_now(); + std::thread::yield_now(); } } @@ -312,7 +320,7 @@ async fn warm_close_fences_concurrent_arc_mutations() { let writer_cache = Arc::clone(&cache); let ready = Arc::new(Barrier::new(2)); let writer_ready = Arc::clone(&ready); - let writer = thread::spawn(move || { + let writer = std::thread::spawn(move || { writer_ready.wait(); let mut accepted = Vec::new(); for ordinal in 0_u64..256 { @@ -322,7 +330,7 @@ async fn warm_close_fences_concurrent_arc_mutations() { accepted.push(ordinal); break; } - Err(error) if error.kind() == ErrorKind::Overloaded => thread::yield_now(), + Err(error) if error.kind() == ErrorKind::Overloaded => std::thread::yield_now(), Err(error) if error.kind() == ErrorKind::Unavailable => return accepted, Err(error) => panic!("concurrent cache write failed: {error}"), } @@ -359,7 +367,7 @@ async fn concurrent_open_reports_structured_busy_error() { let error = Cache::open(&files.data, test_config(1)).await.unwrap_err(); assert_eq!(error.kind(), ErrorKind::Busy); assert_eq!(error.operation(), ErrorOperation::Open); - assert_eq!(error.io_kind(), std::io::ErrorKind::WouldBlock); + assert_eq!(error.io_kind(), io::ErrorKind::WouldBlock); cache.close_fast().await.unwrap(); } @@ -504,7 +512,7 @@ async fn l1_bypass_may_remain_stale_after_region_completion() { fn unavailable_io_engine_is_rejected_before_file_creation() { let files = TestCache::new("unavailable-io-engine"); let runtime = RuntimeOptions { - io_engine: IoEngine::IoUring(cache2::IoUringConfig::default()), + io_engine: IoEngineConfig::IoUring(IoUringConfig::default()), write_flush_threshold_bytes: 128 * 1024, statistics: false, ..test_runtime_options(1, 2) @@ -513,7 +521,7 @@ fn unavailable_io_engine_is_rejected_before_file_creation() { let error = CacheConfig::new(test_storage(), runtime).unwrap_err(); assert_eq!(error.kind(), ErrorKind::Unsupported); assert_eq!(error.operation(), ErrorOperation::BuildConfig); - assert_eq!(error.io_kind(), std::io::ErrorKind::Unsupported); + assert_eq!(error.io_kind(), io::ErrorKind::Unsupported); files.assert_absent(); } @@ -531,7 +539,7 @@ fn unavailable_direct_io_is_rejected_before_file_creation() { let error = CacheConfig::new(test_storage(), runtime).unwrap_err(); assert_eq!(error.kind(), ErrorKind::Unsupported); assert_eq!(error.operation(), ErrorOperation::BuildConfig); - assert_eq!(error.io_kind(), std::io::ErrorKind::Unsupported); + assert_eq!(error.io_kind(), io::ErrorKind::Unsupported); files.assert_absent(); } @@ -561,7 +569,7 @@ async fn runtime_options_can_change_across_a_warm_reopen() { cache.close_warm().await.unwrap(); let retuned = RuntimeOptions { - io_engine: IoEngine::Posix(PosixIoConfig::new(7, 2, 2)), + io_engine: IoEngineConfig::Posix(PosixIoConfig::new(7, 2, 2)), l1_capacity_bytes: 2 * 1024 * 1024, l1_eviction_policy: L1EvictionPolicy::S3Fifo, l1_shards: 7, @@ -722,7 +730,7 @@ async fn concurrent_mixed_mutations_never_return_wrong_key_or_future_values() { let hits = AtomicU64::new(0); let runtime = tokio::runtime::Handle::current(); - thread::scope(|scope| { + std::thread::scope(|scope| { for writer in 0..WRITERS { let cache = &cache; let keys = &keys; @@ -731,7 +739,7 @@ async fn concurrent_mixed_mutations_never_return_wrong_key_or_future_values() { let start = &start; scope.spawn(move || { while !start.load(Ordering::Acquire) { - thread::yield_now(); + std::thread::yield_now(); } let mut value = vec![0_u8; *VALUE_SIZES.iter().max().unwrap()]; for ordinal in 0..WRITES_PER_CLIENT { @@ -766,7 +774,7 @@ async fn concurrent_mixed_mutations_never_return_wrong_key_or_future_values() { let runtime = runtime.clone(); scope.spawn(move || { while !start.load(Ordering::Acquire) { - thread::yield_now(); + std::thread::yield_now(); } let mut ordinal = reader; while writers_left.load(Ordering::Acquire) != 0 { @@ -818,7 +826,7 @@ async fn public_key_and_record_size_limits_are_enforced() { let error = cache.put(&oversized_key, b"value").unwrap_err(); assert_eq!(error.kind(), ErrorKind::InvalidInput); assert_eq!(error.operation(), ErrorOperation::Put); - assert_eq!(error.io_kind(), std::io::ErrorKind::InvalidInput); + assert_eq!(error.io_kind(), io::ErrorKind::InvalidInput); let error = cache.put(b"too-large", &oversized_value).unwrap_err(); assert_eq!(error.kind(), ErrorKind::InvalidInput); @@ -864,8 +872,8 @@ async fn cold_start_removes_stale_recovery_files() { let files = TestCache::new("stale-recovery-files"); let image = files.sidecar(".image"); let temporary = files.sidecar(".image.next"); - std::fs::write(&image, b"stale image").unwrap(); - std::fs::write(&temporary, b"stale temporary image").unwrap(); + fs::write(&image, b"stale image").unwrap(); + fs::write(&temporary, b"stale temporary image").unwrap(); let cache = Cache::open(&files.data, test_config(2)).await.unwrap(); assert!(!image.exists()); @@ -924,7 +932,7 @@ async fn reported_peak_disk_bytes_covers_atomic_warm_publication() { let image = files.sidecar(".image"); let temporary = files.sidecar(".image.next"); - std::fs::copy(&image, &temporary).unwrap(); + fs::copy(&image, &temporary).unwrap(); let logical_bytes = [ files.data.clone(), files.sidecar(".state"), @@ -932,7 +940,7 @@ async fn reported_peak_disk_bytes_covers_atomic_warm_publication() { temporary, ] .into_iter() - .map(|path| std::fs::metadata(path).unwrap().len()) + .map(|path| fs::metadata(path).unwrap().len()) .sum::(); assert_eq!(logical_bytes, peak_disk_bytes); @@ -1068,7 +1076,7 @@ async fn cache_snapshot_reports_tier_activity_and_resets_on_open() { assert_eq!(fresh.l1_hits, 0); assert_eq!(fresh.l2_hits, 0); assert_ne!(fresh.metrics_epoch, before_close.metrics_epoch); - assert_eq!(fresh.io, cache2::CacheIoSnapshot::default()); + assert_eq!(fresh.io, CacheIoSnapshot::default()); assert_eq!( reopened.get("key").await.unwrap().unwrap().tier(), CacheTier::L2 @@ -1148,7 +1156,7 @@ async fn read_io_failure_is_counted_and_latches_miss_only() { cache.put("key", vec![9_u8; 16 * 1024]).unwrap(); cache.drain().await.unwrap(); - std::fs::OpenOptions::new() + OpenOptions::new() .write(true) .open(&files.data) .unwrap() diff --git a/tests-integration/tests/config.rs b/tests-integration/tests/config.rs index 65b45b2..3f38db1 100644 --- a/tests-integration/tests/config.rs +++ b/tests-integration/tests/config.rs @@ -17,7 +17,7 @@ use std::time::Duration; use cache2::CacheConfig; use cache2::ErrorKind; use cache2::ErrorOperation; -use cache2::IoEngine; +use cache2::IoEngineConfig; use cache2::L1EvictionPolicy; use cache2::PosixIoConfig; use cache2::ReadAdmission; @@ -100,7 +100,7 @@ fn large_layout_memory_floor_includes_each_l1_policy() { let runtime = RuntimeOptions { l1_capacity_bytes: 10 * GIB, managed_memory_limit_bytes: 15 * GIB, - io_engine: IoEngine::Posix(PosixIoConfig::new(4, 4, 2)), + io_engine: IoEngineConfig::Posix(PosixIoConfig::new(4, 4, 2)), l1_shards: 64, l1_eviction_policy: policy, ..RuntimeOptions::default() @@ -144,7 +144,7 @@ fn automatic_wait_capacity_uses_the_selected_engine() { let config = CacheConfig::new( storage.clone(), RuntimeOptions { - io_engine: IoEngine::Posix(PosixIoConfig::new(workers, 1, 1)), + io_engine: IoEngineConfig::Posix(PosixIoConfig::new(workers, 1, 1)), ..options.clone() }, ) @@ -204,23 +204,23 @@ fn invalid_runtime_options_are_rejected_when_building_configuration() { ..config }), ("zero-reclaim-workers", |config| RuntimeOptions { - io_engine: IoEngine::Posix(PosixIoConfig::new(1, 1, 0)), + io_engine: IoEngineConfig::Posix(PosixIoConfig::new(1, 1, 0)), ..config }), ("too-many-reclaim-workers", |config| RuntimeOptions { - io_engine: IoEngine::Posix(PosixIoConfig::new(1, 1, 3)), + io_engine: IoEngineConfig::Posix(PosixIoConfig::new(1, 1, 3)), ..config }), ("zero-read-workers", |config| RuntimeOptions { - io_engine: IoEngine::Posix(PosixIoConfig::new(0, 1, 1)), + io_engine: IoEngineConfig::Posix(PosixIoConfig::new(0, 1, 1)), ..config }), ("zero-write-workers", |config| RuntimeOptions { - io_engine: IoEngine::Posix(PosixIoConfig::new(1, 0, 1)), + io_engine: IoEngineConfig::Posix(PosixIoConfig::new(1, 0, 1)), ..config }), ("too-many-read-workers", |config| RuntimeOptions { - io_engine: IoEngine::Posix(PosixIoConfig::new(4097, 1, 1)), + io_engine: IoEngineConfig::Posix(PosixIoConfig::new(4097, 1, 1)), ..config }), ("zero-read-wait-capacity", |config| RuntimeOptions { @@ -238,7 +238,7 @@ fn invalid_runtime_options_are_rejected_when_building_configuration() { ..config }), ("too-many-write-workers", |config| RuntimeOptions { - io_engine: IoEngine::Posix(PosixIoConfig::new(1, 4097, 1)), + io_engine: IoEngineConfig::Posix(PosixIoConfig::new(1, 4097, 1)), ..config }), ("excessive-read-wait", |config| RuntimeOptions { @@ -254,7 +254,7 @@ fn invalid_runtime_options_are_rejected_when_building_configuration() { ..config }), ("fixed-footprint-exceeds-budget", |config| RuntimeOptions { - io_engine: IoEngine::Posix(PosixIoConfig::new(2, 2, 1)), + io_engine: IoEngineConfig::Posix(PosixIoConfig::new(2, 2, 1)), l1_capacity_bytes: 0, managed_memory_limit_bytes: 2 * 1024 * 1024, write_flush_threshold_bytes: 128 * 1024, diff --git a/tests-integration/tests/error.rs b/tests-integration/tests/error.rs index a09221b..14955d9 100644 --- a/tests-integration/tests/error.rs +++ b/tests-integration/tests/error.rs @@ -12,8 +12,9 @@ // See the License for the specific language governing permissions and // limitations under the License. -use std::error::Error as _; +use std::io; +use cache2::Error; use cache2::ErrorKind; use cache2::ErrorOperation; use cache2::StorageOptions; @@ -24,9 +25,9 @@ fn storage_construction_errors_expose_structured_context() { assert_eq!(error.kind(), ErrorKind::InvalidInput); assert_eq!(error.operation(), ErrorOperation::BuildStorage); - assert_eq!(error.io_kind(), std::io::ErrorKind::InvalidInput); + assert_eq!(error.io_kind(), io::ErrorKind::InvalidInput); assert!(error.raw_os_error().is_none()); - assert!(error.source().is_some()); + assert!(std::error::Error::source(&error).is_some()); assert!(error.to_string().contains("build_storage")); } @@ -34,14 +35,14 @@ fn storage_construction_errors_expose_structured_context() { fn default_io_conversion_preserves_the_original_error() { let error = StorageOptions::new(1).build().unwrap_err(); let message = error.as_io_error().to_string(); - let error = std::io::Error::from(error); + let error = io::Error::from(error); - assert_eq!(error.kind(), std::io::ErrorKind::InvalidInput); + assert_eq!(error.kind(), io::ErrorKind::InvalidInput); assert_eq!(error.to_string(), message); assert!( error .get_ref() - .and_then(|source| source.downcast_ref::()) + .and_then(|source| source.downcast_ref::()) .is_none() ); } @@ -53,10 +54,10 @@ fn contextual_io_conversion_keeps_structured_source() { .unwrap_err() .into_io_error_with_context(); - assert_eq!(error.kind(), std::io::ErrorKind::InvalidInput); + assert_eq!(error.kind(), io::ErrorKind::InvalidInput); let source = error .get_ref() - .and_then(|source| source.downcast_ref::()) + .and_then(|source| source.downcast_ref::()) .expect("structured error remains in the source chain"); assert_eq!(source.operation(), ErrorOperation::BuildStorage); } diff --git a/xtask/src/main.rs b/xtask/src/main.rs index 9301cdb..a19fcba 100644 --- a/xtask/src/main.rs +++ b/xtask/src/main.rs @@ -16,7 +16,6 @@ use std::env; use std::ffi::OsStr; use std::ffi::OsString; use std::path::Path; -use std::process::Command as StdCommand; use cargo_metadata::Metadata; use cargo_metadata::MetadataCommand; @@ -87,7 +86,7 @@ impl CommandBench { run(self.command()); } - fn command(self) -> StdCommand { + fn command(self) -> std::process::Command { let mut command = cargo(); command.args(["bench", "--package", "benchmarks"]); command.args(self.cargo_args); @@ -233,15 +232,15 @@ impl CommandTest { } } -fn cargo() -> StdCommand { +fn cargo() -> std::process::Command { let executable = env::var_os("CARGO").unwrap_or_else(|| OsString::from("cargo")); - let mut command = StdCommand::new(executable); + let mut command = std::process::Command::new(executable); command.current_dir(Path::new(env!("CARGO_WORKSPACE_DIR"))); command } -fn nightly_cargo() -> StdCommand { - let mut command = StdCommand::new("rustup"); +fn nightly_cargo() -> std::process::Command { + let mut command = std::process::Command::new("rustup"); command .args(["run", "nightly", "cargo"]) .current_dir(Path::new(env!("CARGO_WORKSPACE_DIR"))); @@ -263,14 +262,14 @@ where I: IntoIterator, S: AsRef, { - let mut command = StdCommand::new(executable); + let mut command = std::process::Command::new(executable); command .current_dir(Path::new(env!("CARGO_WORKSPACE_DIR"))) .args(args); run(command); } -fn run(mut command: StdCommand) { +fn run(mut command: std::process::Command) { println!("{command:?}"); match command.status() { Ok(status) if status.success() => {}