Add generation, tensor, and async engine usage guides

README.md

@@ -118,6 +118,9 @@ cargo run --release --bin bench            # matmul/gelu/MLP numbers on your mac
 
 Both measured and reproducible — see [docs/benchmarks.md](docs/benchmarks.md) (real M2 numbers, tiled vs naive matmul) and [docs/correctness.md](docs/correctness.md) (per-op CPU↔Metal deviations + the GPT-2 end-to-end check). For common setup and platform questions, see the [troubleshooting guide](docs/troubleshooting.md); browse the [documentation index](docs/README.md) for the complete guide list. Contributors can start with [CONTRIBUTING.md](CONTRIBUTING.md).
 
+For hands-on use, read the [generation guide](docs/generation.md),
+[tensor API guide](docs/tensors.md), and [async engine guide](docs/engine.md).
+
 ## License
 
 MIT OR Apache-2.0

docs/README.md

@@ -8,3 +8,6 @@ Start with the [test command reference](test-matrix.md) to choose the smallest v
 - [Issue templates](../.github/ISSUE_TEMPLATE/) — structured bug reports and enhancement proposals.
 - [Test command reference](test-matrix.md) — formatting, compile, lint, and test commands for each task.
 - [Troubleshooting](troubleshooting.md) — platform, model-asset, and test setup notes.
+- [Generation guide](generation.md) — GPT-2 setup, command-line controls, output, and context behavior.
+- [Tensor API guide](tensors.md) — Shapes, fallible allocation, validation, parity, and borrowed views.
+- [Async engine guide](engine.md) — Request submission, queue bounds, cancellation, shutdown, and errors.

docs/engine.md

@@ -0,0 +1,46 @@
+# Async inference engine guide
+
+The engine serves MLP requests through a bounded Tokio channel. Create a
+submitter while a Tokio runtime is active, then await each request result:
+
+```rust
+use std::sync::Arc;
+use batch_forge::engine::spawn;
+use batch_forge::model::{CpuBackend, Mlp, ModelError};
+use batch_forge::tensor::Tensor;
+
+async fn infer_once(model: Arc<Mlp>, input: Tensor) -> Result<Tensor, ModelError> {
+    let client = spawn(Arc::new(CpuBackend), model, 8);
+    client.infer(1, input).await
+}
+```
+
+The input is owned by the submitted request. The request ID is used for
+tracing; it is not a deduplication key or a guarantee of result persistence.
+
+## Queue capacity and execution
+
+`queue_depth` sets the bounded channel capacity; zero is normalized to one
+slot. When the queue is full, sending waits for room. This bounds queued
+requests, but not the number of caller futures retaining inputs while waiting
+to submit.
+
+The manager runs one forward pass at a time. The forward call is synchronous
+inside its Tokio task, so CPU work occupies a runtime worker during inference.
+Async channels do not fuse requests into a batch or make the numerical work
+itself asynchronous.
+
+## Cancellation, shutdown, and errors
+
+If a response receiver is dropped while its request is queued, the manager
+skips that request before computing. A forward pass that has already started
+is not interrupted by this queue check.
+
+Dropping all submitter handles closes the queue. The manager drains available
+requests and exits after receiving channel closure. `infer` returns
+`ModelError::EngineClosed` when sending or receiving can no longer complete;
+model-validation errors are returned through the same result channel.
+
+The submitter does not retry automatically. A caller should distinguish an
+invalid input from a closed engine before deciding whether another attempt is
+appropriate.

docs/generation.md

@@ -0,0 +1,130 @@
+# GPT-2 generation guide
+
+The `generate` binary runs GPT-2 small (124M). The default `batch_forge` binary
+runs exported MLP checkpoints. Invoke the generation binary explicitly:
+
+```bash
+python3 python/fetch_gpt2.py
+cargo run --release --bin generate -- --backend cpu --prompt "Hello" --greedy
+```
+
+Run commands from the repository root. Downloads are setup work; generation
+runs in Rust and does not require a Python runtime.
+
+## Choose an asset directory
+
+Keep `model.safetensors`, `vocab.json`, and `merges.txt` together. The download
+script's `--out` directory and the generator's `--model-dir` must refer to the
+same location. Quote paths that contain spaces:
+
+```bash
+python3 python/fetch_gpt2.py --out "/path/to/gpt2 assets"
+cargo run --release --bin generate -- --model-dir "/path/to/gpt2 assets" --backend cpu
+```
+
+Existing nonempty assets are skipped by the download script. Its `--force`
+option replaces them after a completed transfer; `--timeout SECONDS` sets the
+socket timeout. Failed transfers leave an existing final asset intact.
+
+## Inspect options without downloading a model
+
+```bash
+cargo run --bin generate -- --help
+cargo run --bin generate -- --version
+```
+
+Help and version return before loading model assets. Unknown arguments,
+missing values, invalid integers, and invalid temperatures return argument-error
+exit status 2. Asset and execution errors return failure. The default MLP
+binary has its own `--help`; its flags differ from the generator's flags.
+
+## Select the compute backend
+
+`--backend cpu` selects the portable Rust reference. On macOS, the generator
+defaults to `metal`; elsewhere it defaults to `cpu`. An explicit Metal request
+fails on a non-macOS platform or when Metal initialization fails. It does not
+silently select a different backend.
+
+```bash
+cargo run --release --bin generate -- --backend cpu --prompt "Once upon a time"
+cargo run --release --bin generate -- --backend metal --prompt "Once upon a time"
+```
+
+See [platform troubleshooting](troubleshooting.md) for Metal requirements.
+
+## Redirect generated text
+
+Generation writes text to stdout and setup, backend, and timing information
+to stderr. Redirect them independently:
+
+```bash
+cargo run --release --bin generate -- --backend cpu --prompt "Hello" > answer.txt 2> run.log
+```
+
+The output stream ends with a newline. Characters that span multiple tokenizer
+tokens are emitted after their UTF-8 bytes are complete. Closing an output pipe
+early stops the token loop and treats a broken pipe as normal termination;
+other output errors report failure.
+
+## Empty prompts and end-of-text
+
+An empty CLI prompt is seeded internally with GPT-2's end-of-text token. This
+provides the first input token required by the model without printing that seed.
+The loop stops when it samples end-of-text and does not print the marker.
+
+```bash
+cargo run --release --bin generate -- --backend cpu --prompt "" --max-new 20
+```
+
+`--max-new` must be a positive integer. Timing statistics count sampled tokens,
+including a sampled stop token, so the token count is not a character count.
+
+## Temperature and greedy decoding
+
+`--temperature 0` selects greedy argmax. A positive temperature enables
+sampling; the value must be finite and non-negative. `--greedy` forces zero
+temperature regardless of where the temperature flag appears:
+
+```bash
+cargo run --release --bin generate -- --backend cpu --greedy --temperature 0.8 --prompt "Hello"
+```
+
+Greedy ties retain the earliest token ID. NaN, infinity, and negative CLI
+temperatures are rejected before asset loading.
+
+## Top-k candidate filtering
+
+`--top-k K` restricts sampling to the K largest logits. Zero disables this
+filter, and a value at least as large as the vocabulary retains all candidates.
+With positive temperature, `--top-k 1` leaves only the highest candidate.
+
+Equal-logit cutoff ties prefer smaller token IDs, including signed-zero ties.
+Retained candidates use a stable token-ID order when drawing from their
+probabilities. A greedy run uses argmax directly and ignores top-k filtering.
+
+## Reproduce a sampling run
+
+Choose a seed explicitly when comparing runs:
+
+```bash
+cargo run --release --bin generate -- --backend cpu --seed 42 --temperature 0.8 --top-k 40 --prompt "Hello"
+```
+
+The generation loop retains its RNG across tokens. Record the seed, backend,
+model and tokenizer assets, prompt, sampling flags, and code revision. The same
+seed alone does not guarantee identical output across different assets,
+backends, or revisions: numerical logits and sampling implementation changes
+can affect the result.
+
+## Context limits and loading errors
+
+The default GPT-2 small configuration has a 1024-token context. Generation
+forwards the most recent 1024 tokens when a sequence grows beyond that limit,
+and uses positions within that forwarded slice. Each step recomputes its
+sequence; the loop does not use a KV cache.
+
+Loading rejects checkpoint shapes that differ from the configured model,
+structurally invalid tensors, and mismatched tokenizer vocabulary size. Keep
+weights and tokenizer files from the same model. For a report, include the
+failing command, error text, backend/platform, and asset source; see
+[troubleshooting](troubleshooting.md) and [correctness](correctness.md).

docs/tensors.md

@@ -0,0 +1,87 @@
+# Tensor API guide
+
+`Tensor` is an owned row-major F32 buffer. Its shape records dimensions; its
+data vector records elements. `Tensor::new` checks that these agree:
+
+```rust
+use batch_forge::tensor::{Tensor, TensorError};
+
+fn matrix() -> Result<Tensor, TensorError> {
+    Tensor::new(vec![1.0, 2.0, 3.0, 4.0], vec![2, 2])
+}
+```
+
+`dims2()` checks rank and returns `(rows, cols)`. A 2×2 tensor contains four
+F32 elements; byte-backed views count bytes separately.
+
+## Scalars and empty dimensions
+
+An empty shape `vec![]` describes a scalar containing one element:
+
+```rust
+let scalar = Tensor::new(vec![3.0], vec![])?;
+```
+
+A shape containing a zero dimension describes zero elements. For example,
+`vec![2, 0, 3]` requires an empty data vector. A zero dimension makes the count
+zero regardless of its position in the shape. These are shape semantics;
+model layers and reduction operators can impose additional positive-width
+requirements.
+
+## Fallible zero allocation
+
+Use `try_zeros` when dimensions come from a caller or checkpoint:
+
+```rust
+fn empty_batch() -> Result<Tensor, TensorError> {
+    Tensor::try_zeros(vec![0, 768])
+}
+```
+
+An overflowing element count returns `TensorError::BufferOverflow`. A failed
+buffer reservation returns `TensorError::AllocationFailed`. The convenience
+`zeros` constructor panics on those failures. The fallible API handles errors
+reported by buffer reservation; it is not a process-wide memory limit.
+
+## Validate after changing public fields
+
+`shape` and `data` are public. If a caller edits either after construction,
+call `tensor.validate()` before using that structure:
+
+```rust
+let mut tensor = Tensor::new(vec![1.0, 2.0], vec![2])?;
+tensor.shape = vec![1, 2];
+tensor.validate()?;
+```
+
+Validation checks structural consistency and shape overflow. It does not
+require every F32 value to be finite. MLP inference revalidates its public
+layer tensors and the supplied input before forwarding.
+
+## Interpret parity differences
+
+`max_abs_diff` returns the largest absolute difference for valid, equal-shaped
+buffers. It returns infinity for shape mismatch, invalid buffers, or a
+non-finite element difference; even two equally truncated tensors fail.
+
+```rust
+let difference = actual.max_abs_diff(&reference);
+let passes = difference.is_finite() && difference <= 1e-3;
+```
+
+Choose a tolerance appropriate to the operation and test. A matching numeric
+buffer is evidence about that comparison, not proof that a whole model or
+asset set has been verified. See [correctness](correctness.md) for the test
+boundaries used by this repository.
+
+## Borrowed views and portable F32 materialization
+
+`TensorView::new` checks shape, dtype, and byte-buffer size while borrowing the
+bytes. `to_tensor_f32()` copies F32 values by reading little-endian bytes and
+works with an unaligned source offset. It rejects a non-F32 dtype and
+revalidates public fields before conversion.
+
+`as_slice::<T>()` attempts a byte reinterpretation with alignment and length
+checks. It does not verify that `T` matches the semantic dtype or convert byte
+order. Callers must check those conditions themselves; use F32 materialization
+when a portable owned tensor is needed.

docs/troubleshooting.md

@@ -68,3 +68,11 @@ The default MLP binary returns failure for missing checkpoints, invalid input
 buffers, failed inference requests, and failed numerical checks. A reference
 file selected with --verify must contain both input and output; a
 synthetic fallback is not used to claim verification.
+
+## Detailed usage references
+
+See the [generation guide](generation.md) for backend, sampling, output, and
+context-window behavior. The [tensor guide](tensors.md) explains shape and
+buffer errors; the [engine guide](engine.md) explains queue bounds and shutdown
+behavior. Use the [test command reference](test-matrix.md) to choose a local
+check for a reproduction.

src/engine.rs

@@ -2,7 +2,8 @@
 //!
 //! A `tokio` task owns the model and backend and serves inference requests off
 //! an `mpsc` queue, replying on a per-request `oneshot` channel. This is the
-//! non-blocking request/response architecture; true continuous batching (fusing
+//! asynchronous request/response queue; forward computation is synchronous
+//! inside the manager task. True continuous batching (fusing
 //! queued requests into one dispatch) is future work — see the README roadmap.
 //!
 //! The engine is backend-agnostic (`Arc<dyn Backend>`), so it runs on the CPU

src/tensor.rs

@@ -90,7 +90,9 @@ impl<'data> TensorView<'data> {
         self.shape.iter().product()
     }
 
-    /// Safely casts the underlying byte buffer to a typed slice if the dtype matches.
+    /// Attempts to reinterpret bytes as a Pod slice, checking alignment and length.
+    /// This does not check semantic dtype or convert byte order. Use
+    /// [`TensorView::to_tensor_f32`] for portable, owned F32 materialization.
     pub fn as_slice<T: Pod>(&self) -> Option<&[T]> {
         bytemuck::try_cast_slice(self.data).ok()
     }