diff --git a/docs/deployment/adapter.md b/docs/deployment/adapter.md index 5082d8f0..b3d91c31 100644 --- a/docs/deployment/adapter.md +++ b/docs/deployment/adapter.md @@ -1,7 +1,7 @@ --- sidebar_position: 3 title: FUSE Adapter -description: Mount a CTE-backed virtual filesystem using FUSE — no LD_PRELOAD required. +description: Mount a CTE-backed virtual filesystem using FUSE on Linux, macOS, and Windows — no LD_PRELOAD required. --- # FUSE Adapter @@ -21,50 +21,149 @@ No in-memory metadata structures are needed. All state — file contents, sizes, --- +## Platform Support + +`clio_cte_fuse` runs on all three desktop platforms. The FUSE **callbacks and the entire CTE data path below them are identical everywhere** — only the kernel backend and the mount mechanics differ. + +| | Linux | macOS | Windows | +|---|---|---|---| +| **Backend** | libfuse3 | [macFUSE](https://macfuse.github.io) 5 (ships libfuse3) | [WinFsp](https://winfsp.dev) 2.0 | +| **Ships in the pip wheel** | Yes | **No** — source build required | Yes | +| **Mountpoint** | Any directory | A directory (kext) or under `/Volumes` (FSKit) | A drive letter (`Z:`) or a directory | +| **Unmount** | `fusermount3 -u` | `umount` / `diskutil unmount force` | Stop the daemon process | +| **Discovery** | `pkg-config fuse3` | `pkg-config fuse3` | `WINFSP_ROOT` (no `.pc` file) | +| **CI coverage** | Build + unit + ops + live mount + xfstests conformance | Build + unit + ops enforced; mount smoke is a non-blocking probe | Build + unit + live WinFsp mount smoke | + +:::note macOS is a source build +The macOS wheel does not ship `clio_cte_fuse`. Build from source against macFUSE — see the macOS tab below. (Older notes claiming macOS "has no FUSE3 API" predate macFUSE 5, which does ship libfuse3.) + +The `clio_cte_fuse` **console script is declared in every wheel, including macOS**, so the command exists on `PATH` there even though the binary behind it does not. Running it prints `Error: clio_cte_fuse binary not found at ` and exits 1 — that is the expected symptom on a pip-installed macOS host, not a broken install. + +macOS wheels are also published for **Apple Silicon only** (`macosx_14_0_arm64`), and there is no sdist, so `pip install iowarp-core` has nothing to resolve on an Intel Mac. +::: + +--- + ## Installation -### Option 1: install.sh (recommended) +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; -The simplest way to build IOWarp with FUSE support: + + + +**1. Install libfuse3.** It is a system dependency and is deliberately *not* bundled in the wheel: + +```bash +sudo apt install libfuse3-dev fuse3 # Ubuntu / Debian +sudo dnf install fuse3-devel fuse3 # RHEL / Fedora +``` + +If you only ever run a prebuilt binary, the runtime packages (`fuse3 libfuse3-3` / `fuse3 fuse3-libs`) are enough; the `-dev` / `-devel` packages are needed to *build* the adapter. + +**2a. Prebuilt (pip).** The Linux wheel already ships `clio_cte_fuse`: ```bash -# Install FUSE3 system dependency first -sudo apt install libfuse3-dev fuse3 # Ubuntu / Debian -sudo dnf install fuse3-devel fuse3 # RHEL / Fedora +pip install iowarp-core +clio_cte_fuse --help +``` + +**2b. From source.** The `release-fuse` preset is a Release build with the FUSE adapter enabled: -# Build and install IOWarp with FUSE enabled +```bash bash install.sh release-fuse +# or, configuring manually: +cmake --preset release-fuse +cmake --build build -j"$(nproc)" ``` -The `release-fuse` preset builds a Release-mode IOWarp with the FUSE adapter, ADIOS2 adapter, and all standard components enabled. +To enable FUSE on any other preset, add `-DCLIO_CTE_ENABLE_FUSE_ADAPTER=ON`. + +**3. Verify.** `/dev/fuse` must exist and be accessible: + +```bash +ls -l /dev/fuse +fusermount3 --version +``` -### Option 2: CMake manual build + + -If you prefer to configure manually: +**1. Install macFUSE.** macFUSE 5 ships libfuse3 — headers, dylib, and a `fuse3` pkg-config file — under `/usr/local`: ```bash -# Using the preset -cmake --preset release-fuse -cmake --build build -j$(nproc) +brew install --cask macfuse +``` + +Installing the cask needs no kernel-extension approval. Only *mounting* through the kext backend does — see the mount step below. + +**2. Build from source.** The macOS wheel does not include the adapter, so this step is required: -# Or enable FUSE on any existing preset -cmake -DCLIO_CTE_ENABLE_FUSE_ADAPTER=ON .. -cmake --build . --target clio_cte_fuse -j$(nproc) +```bash +cmake --preset release-mac-fuse +cmake --build build-mac-fuse -j"$(sysctl -n hw.ncpu)" ``` -This produces the `clio_cte_fuse` binary. The adapter links against `clio_cte_core_client` and `libfuse3` — it does **not** require MPI or ELF interception. +`release-mac-fuse` is guarded by a `hostSystemName == Darwin` condition and turns off `io_uring` (Linux-only). To enable FUSE on a different preset, add `-DCLIO_CTE_ENABLE_FUSE_ADAPTER=ON`. + +**3. If configure does not find FUSE.** Discovery goes through pkg-config, and macFUSE's `.pc` lives outside the default search path on some setups. Make it visible: -### Prerequisites +```bash +export PKG_CONFIG_PATH="/usr/local/lib/pkgconfig:/opt/homebrew/lib/pkgconfig:$PKG_CONFIG_PATH" +``` -Ensure FUSE3 and `/dev/fuse` are available: +The adapter is skipped — not failed — when the backend is missing, so a silent absence of `bin/clio_cte_fuse` is the symptom. Check for it explicitly: ```bash -# Verify installation -ls -l /dev/fuse -fusermount3 --version +test -x build-mac-fuse/bin/clio_cte_fuse && echo "FUSE adapter built" +``` + + + + +**1. Install WinFsp.** WinFsp provides a FUSE3-compatible header (`inc/fuse3/fuse.h`) and an import library. Install it from [winfsp.dev](https://winfsp.dev), or silently: + +```powershell +$msi = "$env:TEMP\winfsp.msi" +Invoke-WebRequest -Uri "https://github.com/winfsp/winfsp/releases/download/v2.0/winfsp-2.0.23075.msi" -OutFile $msi +# ADDLOCAL=ALL pulls in the Developer feature (inc/fuse3 + lib) needed to BUILD. +Start-Process msiexec.exe -ArgumentList "/i `"$msi`" /qn ADDLOCAL=ALL" -Wait +``` + +The MSI installs the kernel driver too, so mounting works without a reboot. To only *run* a prebuilt binary, the default (runtime-only) install is enough — the Developer feature is a build-time requirement. + +**2a. Prebuilt (pip).** The Windows wheel ships `clio_cte_fuse.exe`: + +```powershell +pip install iowarp-core +clio_cte_fuse --help +``` + +The console script prepends WinFsp's `bin` directory to `PATH` itself, so `winfsp-x64.dll` resolves without a system-wide `PATH` edit. If WinFsp is missing, the script exits with an explanatory error rather than a bare DLL-load failure. + +**2b. From source.** There is no dedicated Windows FUSE preset — enable the option on `windows-release`: + +```powershell +cmake --preset windows-release -DCLIO_CTE_ENABLE_FUSE_ADAPTER=ON +cmake --build build --config Release -j $env:NUMBER_OF_PROCESSORS +``` + +WinFsp ships no pkg-config file, so it is located by path rather than by `pkg-config`. The default is `%ProgramFiles(x86)%\WinFsp`; override it if you installed elsewhere: + +```powershell +cmake -B build -A x64 -DCLIO_CTE_ENABLE_FUSE_ADAPTER=ON -DWINFSP_ROOT="D:\WinFsp" +``` + +**3. Verify.** A missing backend skips the adapter rather than failing the build, so check the binary exists: + +```powershell +Test-Path build\bin\clio_cte_fuse.exe ``` -The CLIO Runtime must also be installed. See [Configuration](./configuration.md) for details. + + + +The adapter links against `clio_cte_filesystem_client`, `clio_cte_core_client`, and the platform's FUSE backend — it does **not** require MPI or ELF interception. The CLIO Runtime must also be installed; see [Configuration](./configuration.md). --- @@ -85,6 +184,11 @@ clio_run start ### 2. Mount the FUSE filesystem +The daemon **create-or-binds** the filesystem pool and the CTE pool underneath it, so no separate `compose` step is required. To size storage tiers explicitly instead of taking the defaults, compose a CTE pool first with `clio_run compose start my_cte.yaml`. + + + + ```bash mkdir -p /mnt/cte @@ -92,14 +196,58 @@ mkdir -p /mnt/cte CLIO_WITH_RUNTIME=0 clio_cte_fuse /mnt/cte -f ``` +Any directory works as a mountpoint. + + + + +macFUSE offers two kernel backends, and which one you can use decides where the mountpoint may live. + +**Kext backend (default).** Full-featured, but the kernel extension needs one-time user approval: System Settings → Privacy & Security → *Allow* the developer, then reboot. This cannot be automated, which is why CI does not use it. + +```bash +mkdir -p ~/cte-mnt +CLIO_WITH_RUNTIME=0 clio_cte_fuse ~/cte-mnt -f +``` + +**FSKit backend (kext-free).** Requires **macFUSE 5.1+ on macOS 15.4+**, and the mountpoint **must be under `/Volumes`**: + +```bash +sudo mkdir -p /Volumes/cte-mnt +sudo chown "$(whoami)" /Volumes/cte-mnt + +CLIO_WITH_RUNTIME=0 clio_cte_fuse /Volumes/cte-mnt -f -o backend=fskit +``` + +:::caution macOS mounting is the least-proven path +The macOS build and its unit + in-process operation suites are enforced in CI, but the live mount smoke test is a **non-blocking probe** — FSKit extension-approval behavior on CI runner images is still unproven. A first mount attempt that hangs rather than failing is a known shape of this problem; bound it with a timeout rather than waiting indefinitely. +::: + + + + +WinFsp mounts drive letters natively, which is the usual choice: + +```powershell +$env:CLIO_WITH_RUNTIME = "0" +clio_cte_fuse Z: -f +``` + +A host directory also works in place of `Z:`. Pick a drive letter that is actually free — `Get-PSDrive -PSProvider FileSystem` lists the ones in use. + + + + +Every argument after the mountpoint is handed to `fuse_main`, so the standard libfuse options apply: + | Flag | Description | |------|-------------| -| `-f` | Run in the foreground (recommended for debugging). Omit to daemonize. | +| `-f` | Run in the foreground (recommended for debugging, and what CI exercises). Omit to daemonize. | | `-d` | Debug mode — prints every FUSE callback to stderr. | | `-o allow_other` | Allow other users to access the mount (requires `user_allow_other` in `/etc/fuse.conf`). | | `-s` | Single-threaded mode. By default FUSE is multi-threaded. | -Set `CLIO_WITH_RUNTIME=0` so the FUSE daemon connects as a pure client to the existing runtime instead of trying to start its own. +Set `CLIO_WITH_RUNTIME=0` so the FUSE daemon attaches to the runtime you already started instead of spawning its own embedded one. Without it the daemon comes up on a private runtime and **the data will not be visible to other clients**. ### 3. Use it @@ -123,10 +271,58 @@ rm /mnt/cte/greeting.txt ### 4. Unmount + + + ```bash fusermount3 -u /mnt/cte +# older systems: fusermount -u /mnt/cte +``` + + + + +```bash +umount ~/cte-mnt +# if the volume is busy or wedged: +diskutil unmount force ~/cte-mnt ``` + + + +There is no `fusermount` on Windows — stopping the daemon makes WinFsp unmount the volume: + +```powershell +Stop-Process -Name clio_cte_fuse -Force +``` + +Or press `Ctrl+C` in the foreground (`-f`) window. + + + + +--- + +## Running Under Apptainer (Linux HPC) + +Apptainer's `--fusemount` opens `/dev/fuse` and passes the FUSE binary a pre-opened file descriptor as the last argument, `/dev/fd/`. libfuse 3's high-level argv parser rejects that token (it is a libfuse2-era convention), and Apptainer strips the mountpoint from `argv`, so there is no plain `fuse_main` invocation that works. + +`clio_cte_fuse` detects a trailing `/dev/fd/` and takes a separate path: it binds the descriptor to a mountpoint itself and drives the protocol with `fuse_session_custom_io()`. Two things this requires: + +1. **`CLIO_CTE_FUSE_MOUNTPOINT` must be set** before the binary is exec'd. Apptainer communicates the mountpoint through neither `argv` nor the environment, so the binary cannot discover it and exits with an error rather than guessing. +2. **`CAP_SYS_ADMIN` in the current user namespace** — unprivileged Apptainer (no setuid starter) does not call `mount(2)` itself for a user-supplied binary; it only hands over the descriptor. Apptainer's userns mapping normally provides the capability. + +```bash +export CLIO_CTE_FUSE_MOUNTPOINT=/mnt/cte +``` + +:::warning The pip wheel cannot do this +The custom-io path needs **libfuse 3.14+ headers at build time**. The manylinux images used to build Linux wheels ship libfuse 3.10.2, so the path is compiled out of the published wheel — running it in `--fusemount` mode prints a "needs 3.14+ headers at build time" error and exits. Normal mounting is unaffected. **Build from source against libfuse 3.14+ if you need the Apptainer path.** +::: + +This is Linux-only. Windows and macOS always take the ordinary `fuse_main` mount-and-serve route. + --- ## Quick Start Scripts @@ -146,6 +342,28 @@ cd context-transfer-engine/test/integration/fuse-manual ./stop.sh ``` +### End-to-end mount check + +The scripts CI uses to validate a real mount are the fastest way to confirm a fresh deployment works. They start the runtime, compose a CTE pool, mount, write + read back + verify a file, then tear everything down: + +```bash +# Linux and macOS +CI/fuse_mount_smoke.sh + +# macOS with the kext-free FSKit backend +sudo mkdir -p /Volumes/cte_smoke && sudo chown "$(whoami)" /Volumes/cte_smoke +CLIO_SMOKE_MOUNT_POINT=/Volumes/cte_smoke \ +CLIO_SMOKE_FUSE_OPTS="-o backend=fskit" \ + CI/fuse_mount_smoke.sh +``` + +```powershell +# Windows — picks a free drive letter automatically +pwsh CI/fuse_mount_smoke.ps1 -BuildDir +``` + +`` is a CMake binary directory whose `bin/` holds `clio_run` and `clio_cte_fuse`. Both scripts fail fast with an explicit message if the adapter binary is missing, which is the usual sign that the FUSE backend was not detected at configure time. + --- ## Configuration @@ -183,7 +401,9 @@ With this config, files written to the FUSE mount are automatically placed acros ## Docker -When running inside a Docker container, the container needs FUSE device access: +A FUSE mount created inside a container lives in that container's mount namespace — it is not visible on the host. On macOS and Windows the container also runs inside a Linux VM, so the mount is doubly unreachable from the host filesystem. If you want the mount usable from the host on those platforms, run the FUSE daemon natively (as above) and let containers reach the runtime over the network instead. + +When mounting inside a Linux container, the container needs FUSE device access: ```bash docker run --cap-add SYS_ADMIN --device /dev/fuse \ @@ -238,17 +458,29 @@ The 1 MB default page size minimizes the number of CTE blob operations per write | `unlink` | Deletes tag with `AsyncDelTag` | | `mkdir` | Creates sentinel tag (path + `/`) so the directory is immediately visible | | `rmdir` | Deletes sentinel tag; fails with `ENOTEMPTY` if children exist | -| `truncate` | Updates cached size (CTE does not yet support blob truncation) | -| `utimens` | Accepted silently (CTE manages timestamps internally) | - -### Not yet supported +| `truncate` | `AsyncTruncate` on the filesystem chimod | +| `utimens` | Sets atime/mtime; honors `UTIME_NOW` (resolved server-side, sharing the tag clock) and `UTIME_OMIT` | +| `rename` | `AsyncRename`; honors `RENAME_NOREPLACE` | +| `chmod` / `chown` | `AsyncChmod` / `AsyncChown` — POSIX mode bits and ownership are stored | +| `symlink` / `readlink` | Target string stored in a reserved marker blob under the link's tag | +| `link` | Hard link — both names bind to the same CTE tag, so they share all data and inode | +| `statfs` | Reports real capacity via `GetCapacity` | +| `setxattr` / `getxattr` / `listxattr` / `removexattr` | Extended attributes, stored per tag | +| `fsync` / `flush` | No-ops returning success — writes are already write-through, so there is nothing buffered to flush | +| `fallocate` | Linux only. `FALLOC_FL_KEEP_SIZE` and `FALLOC_FL_ZERO_RANGE`; punch/collapse/insert return `EOPNOTSUPP` | + +### Not supported | Operation | Notes | |-----------|-------| -| `rename` | Would require CTE tag rename support | -| `chmod` / `chown` | CTE does not store POSIX permission metadata | -| `symlink` / `link` | Not in scope | -| `statfs` | Not implemented | +| `RENAME_EXCHANGE` / `RENAME_WHITEOUT` | Return `EINVAL` so callers fall back cleanly; would need chimod-level atomic swap / whiteout | +| `fallocate` punch / collapse / insert | Layout-changing; return `EOPNOTSUPP` | + +### Caching semantics + +The `init` callback deliberately disables the kernel's attribute, entry, and negative caches (`attr_timeout = entry_timeout = negative_timeout = 0`). Metadata can change without *this* FUSE process being the one that changed it, and there is no upcall to invalidate a stale entry — so every `getattr`/lookup goes to the chimod, which is the source of truth. Without this, `ln a b; stat a` would report `a`'s stale cached link count. + +The kernel page cache for file *data* stays enabled (`direct_io = 0`). This is what makes `mmap` work: the high-level FUSE API has no `.mmap` callback, so the kernel faults mapped pages through `read` and flushes dirty pages through `write`. Turning `direct_io` on would bypass the page cache and make every `mmap` fail with `ENODEV`. --- @@ -256,6 +488,7 @@ The 1 MB default page size minimizes the number of CTE blob operations per write | | POSIX Adapter | STDIO Adapter | FUSE Adapter | |---|---|---|---| +| **Platforms** | **Linux only** | **Linux only** | Linux, macOS, Windows | | **Mechanism** | `LD_PRELOAD` | `LD_PRELOAD` | Kernel VFS mount | | **Requires preloading** | Yes | Yes | No | | **Requires recompilation** | No | No | No | @@ -265,3 +498,41 @@ The 1 MB default page size minimizes the number of CTE blob operations per write | **Performance overhead** | Low (direct SHM) | Low (direct SHM) | Moderate (kernel ↔ userspace copies) | The FUSE adapter trades some performance for universal compatibility — any program that can open a file path can use it, regardless of language or link-time dependencies. + +The POSIX, STDIO, and HDF5 VFD adapters are built on the glibc ELF/`dlsym` interceptor, which has **no macOS or Windows port**. On those platforms FUSE is the only transparent-interception option. (The HDF5 VOL connector is portable and works on all three.) + +--- + +## Troubleshooting + +### `clio_cte_fuse` was not built + +The adapter is **skipped, not failed**, when its backend is not found at configure time — so the symptom is a missing binary, not a build error. Re-run CMake and look for the warning: + +- **Linux** — `FUSE3 not found`. Install `libfuse3-dev` / `fuse3-devel`. +- **macOS** — the `fuse3` `.pc` file is not on the pkg-config path. Add `/usr/local/lib/pkgconfig` (and `/opt/homebrew/lib/pkgconfig` on Apple Silicon) to `PKG_CONFIG_PATH`. +- **Windows** — `WinFsp not found`. Reinstall the MSI with `ADDLOCAL=ALL` so the Developer feature (`inc/fuse3` + `lib`) is present, or pass `-DWINFSP_ROOT=`. + +### Data written through the mount is invisible to other clients + +`CLIO_WITH_RUNTIME` was unset or non-zero, so the daemon started its own private embedded runtime. Set `CLIO_WITH_RUNTIME=0` and mount again. + +### The mount point does not appear + +Give the daemon a few seconds — the CI smoke tests poll for up to 20 s. If the daemon exits first, run it with `-f` and read stderr; `-d` adds a trace of every FUSE callback. On Windows, confirm the drive letter you chose is actually free. + +### macOS: mount hangs or is refused + +The kext backend needs one-time approval in System Settings → Privacy & Security followed by a reboot. If you cannot approve a kext (managed machines, CI), use the FSKit backend instead: `-o backend=fskit`, macFUSE 5.1+ on macOS 15.4+, mountpoint under `/Volumes`. A hung mount is a known shape here — bound the attempt with a timeout rather than waiting on it. + +### Windows: `winfsp-x64.dll` could not be found + +The MSI does not add WinFsp's `bin` to the system `PATH`. The pip console script handles this itself; a directly-invoked binary does not. Prepend it manually: + +```powershell +$env:PATH = "${env:ProgramFiles(x86)}\WinFsp\bin;$env:PATH" +``` + +### `mmap` fails with `ENODEV` + +Something has enabled `direct_io`, which bypasses the page cache the kernel needs to serve mapped pages. The adapter leaves it off by default; do not pass `-o direct_io`. diff --git a/docs/deployment/configuration.md b/docs/deployment/configuration.md index af8beb7a..b9724131 100644 --- a/docs/deployment/configuration.md +++ b/docs/deployment/configuration.md @@ -16,11 +16,21 @@ The configuration file is located via (in priority order, first hit wins): | Source | Priority | Description | |--------|----------|-------------| -| `CLIO_SERVER_CONF` env var | **1st** | Checked first. | -| `~/.clio/clio.yaml` | **2nd** | Per-user default. Seeded at install time. | -| Built-in defaults | **3rd** | Compiled-in fallback. | - -A handful of legacy paths (`~/.clio/chimaera.yaml`, `~/.chimaera/clio.yaml`, `~/.chimaera/chimaera.yaml`) are also accepted for backward compat — see [Deprecation Notes](../deprecation-notes) for the full lookup order. +| `CLIO_SERVER_CONF` env var | **1st** | Checked first. Any path you like. | +| `~/.clio/clio.yaml` | **2nd** | Per-user default. Seeded at install time from `context-runtime/config/clio_default.yaml`. | +| Built-in defaults | **3rd** | Compiled-in fallback (default port, default workers, **empty compose** — so no storage tiers). | + +:::caution +An **empty** config file is reported as a load failure and logged loudly rather than +being accepted silently. It would otherwise parse as a YAML null, every section would +miss, and the runtime would come up on the built-in defaults — an empty compose section, +so no storage tiers, with the resulting failures (e.g. `PutBlob` out-of-space on the very +first write) surfacing far downstream. The runtime still falls back to defaults, but you +get a warning naming the file. +::: + +Legacy `~/.chimaera/` paths are **no longer** consulted — see +[Deprecation Notes](../deprecation-notes). ```bash # Use the installed default @@ -45,6 +55,13 @@ Size values throughout the file accept: `B`, `KB`, `MB`, `GB`, `TB` (case-insens | `wait_for_restart` | `30` | Seconds to wait for peer nodes during startup. | | `wait_for_restart_poll_period` | `1` | Seconds between connection retry attempts during startup. | +Two address knobs have **no YAML key** and are set through the environment: + +| Variable | Default | Description | +|----------|---------|-------------| +| `CLIO_SERVER_ADDR` | `127.0.0.1` | Address clients dial to reach the runtime. | +| `CLIO_BIND_ADDR` | `0.0.0.0` | Address the runtime's listener sockets bind to. Pin it to `127.0.0.1` on developer machines to avoid per-binary host firewall prompts. | + ```yaml networking: port: 9413 @@ -92,10 +109,15 @@ Log routing: | Parameter | Default | Description | |-----------|---------|-------------| -| `num_threads` | `4` | Worker threads for task execution. | +| `num_threads` | `4` | Worker threads for task execution. Overridden by `CLIO_NUM_THREADS`. | | `queue_depth` | `1024` | Task queue depth per worker. | | `local_sched` | `"default"` | Local task scheduler algorithm. | | `first_busy_wait` | `10000` | Microseconds of busy-waiting before a worker sleeps when idle (10 ms). | +| `learning_rate` | `0.2` | SGD learning rate for the task load-prediction model used by the scheduler. | +| `task_progress_interval_ms` | `5000` | Interval for the periodic cross-node task-validity check. Overridden by `CLIO_TASK_PROGRESS_INTERVAL_MS`. | +| `conf_dir` | `/tmp/clio` | Directory for persistent runtime state written by the daemon. | +| `main_segment_size` | `0` (auto) | Size of the main task-data segment (`Future` + task payload allocations). Accepts a byte count or a size string (`"512m"`, `"1g"`). Overridden by `CLIO_MAIN_SEGMENT_SIZE`. | +| `metadata_segment_size` | `0` (auto) | Size of the runtime-wide metadata segment backing CTE's shared-memory tag/blob maps. Auto default is the host's RAM capacity; the reservation is lazy, so only touched pages cost anything. | ```yaml runtime: @@ -103,10 +125,64 @@ runtime: queue_depth: 1024 local_sched: "default" first_busy_wait: 10000 + learning_rate: 0.2 + # task_progress_interval_ms: 5000 + # main_segment_size: "1g" + # metadata_segment_size: "8g" ``` **Recommendation**: Set `num_threads` to the number of CPU cores on the node. +:::caution Segment sizing on constrained hosts +`main_segment_size` dominates the daemon's commit charge, and +`metadata_segment_size` dominates its shared-memory live-set exposure — +both matter far more than actual data volume in memory-limited containers +and on Windows. Creation-time safety clamps apply (half the cgroup-aware +budget on Linux, a 1 GB file cap elsewhere). Note that if `/dev/shm` is +smaller than the live set, touching past the tmpfs limit raises `SIGBUS`, +not `ENOMEM`. +::: + +--- + +## GPU Orchestrator (`gpu`) + +Only meaningful in a CUDA/ROCm/SYCL build. + +| Parameter | Default | Description | +|-----------|---------|-------------| +| `blocks` | `1` | GPU blocks (task queue partitions). Overridden by `CLIO_GPU_BLOCKS`. | +| `threads_per_block` | `32` | Threads per block. Overridden by `CLIO_GPU_THREADS`. | +| `queue_depth` | `16` | Tasks per GPU queue. | + +```yaml +gpu: + blocks: 1 + threads_per_block: 32 + queue_depth: 16 +``` + +--- + +## Failure Detection (`swim`) + +SWIM gossip-based failure detection across cluster nodes. + +| Parameter | Default | Description | +|-----------|---------|-------------| +| `enabled` | `true` | Enable SWIM failure detection. | +| `direct_probe_timeout_sec` | `30.0` | Timeout for a direct probe of a peer. | +| `indirect_probe_timeout_sec` | `15.0` | Timeout for an indirect (proxied) probe. | +| `suspicion_timeout_sec` | `60.0` | How long a node stays *suspect* before being declared dead. | + +```yaml +swim: + enabled: true + direct_probe_timeout_sec: 30.0 + indirect_probe_timeout_sec: 15.0 + suspicion_timeout_sec: 60.0 +``` + --- ## Compose Section @@ -183,10 +259,13 @@ Array of storage targets. At least one entry is required when CTE is enabled. | Parameter | Required | Description | |-----------|----------|-------------| -| `path` | Yes | `ram::` for DRAM storage, or a filesystem path for disk. | -| `bdev_type` | Yes | `"ram"` for memory-backed, `"file"` for filesystem-backed. | -| `capacity_limit` | Yes | Maximum capacity (e.g., `"512MB"`, `"200GB"`). | -| `score` | No | Placement priority (0.0–1.0). Higher = preferred. `-1.0` = automatic scoring. | +| `path` | Yes | `ram::` for DRAM storage, or a filesystem path for disk. Supports `${HOME}` expansion. | +| `bdev_type` | Yes | `file`, `ram`, `hbm`, `pinned`, or `noop`. | +| `capacity_limit` | Yes | Maximum capacity (e.g., `"512MB"`, `"200GB"`). `0` / `"0g"` = 80% of total system DRAM. For file tiers this is a cap, not an upfront allocation — the file grows lazily in 1 GB units. | +| `score` | No | Placement priority (0.0–1.0). Higher = preferred. `-1.0` (default) = automatic scoring. | +| `persistence_level` | No | `"volatile"` (default), `"temporary"`, or `"long_term"`. | +| `existing_pool_id` | No | Bind this target to an already-composed bdev pool instead of creating one. Skips `path` / `capacity_limit` validation — routing is purely by pool id. | +| `existing_pool_module` | No | Module name backing `existing_pool_id`, when it is not a plain `clio_bdev`. | ```yaml storage: @@ -201,20 +280,67 @@ storage: bdev_type: file capacity_limit: 200GB score: 0.9 + persistence_level: temporary # HDD tier - path: /mnt/hdd/cte bdev_type: file capacity_limit: 2TB score: 0.3 + persistence_level: long_term ``` +:::caution `persistence_level` is load-bearing +Placement filters (`Context::min_persistence_level_`) and durable replicas +(`REPLICA_PERSISTENT`, used by the +[replication ChiMod](../sdk/context-transfer-engine/chimod-chain#replication-chimod-clio_cte_replication)) +key off this field. A tier left at the default `volatile` can **never** +satisfy a persistence request, so a config with no `temporary` or +`long_term` tier gets no durable copies. +::: + ### Data Placement Engine (`dpe`) | Parameter | Default | Description | |-----------|---------|-------------| | `dpe_type` | `"max_bw"` | Placement algorithm: `"max_bw"`, `"round_robin"`, `"random"`. | +### Data Organizer + +The data organizer is CTE's internal, periodically-driven reorganization +engine. When enabled, the runtime spawns periodic `DynamicReorganize` tasks +that rescore blobs and move them between tiers — no external extension or +explicit `ReorganizeBlob` calls required. The `frecency` organizer promotes +recently/frequently accessed blobs toward fast tiers and demotes cold blobs +toward slow tiers. + +| Parameter | Default | Description | +|-----------|---------|-------------| +| `organizer` | `"none"` | Organization policy: `"none"` (disabled) or `"frecency"`. | +| `organizer_tasks` | `1` | Number of periodic task replicas; each organizes a disjoint hash partition of the blob space. | +| `organizer_period_ms` | `5000` | Interval between organizer invocations (ms). | + +```yaml + - mod_name: clio_cte_core + # ... + organizer: frecency + organizer_tasks: 2 + organizer_period_ms: 5000 +``` + +### Bdev Performance Stats Persistence + +Each bdev persists its measured performance statistics (latency/bandwidth +and the learned wall-clock model) across sessions so performance does not +need to be re-estimated on every startup. This also feeds the CTE's I/O +emulation mode (`Context::emulate_` on PutBlob/GetBlob), which models the +duration of an operation from these stats instead of performing the I/O. + +| Environment variable | Default | Description | +|-----------|---------|-------------| +| `CLIO_BDEV_STATS_DIR` | `/.clio/bdev_perf` | Directory holding per-bdev `.perf` stats files. | +| `CLIO_BDEV_PERSIST_STATS` | `1` | Set to `0` to disable persistence (start from a cold model). | + ### Targets (`targets`) | Parameter | Default | Description | @@ -230,37 +356,244 @@ All fields are optional and override compile-time defaults. | Parameter | Default | Description | |-----------|---------|-------------| | `stat_targets_period_ms` | `50` | Periodic StatTargets interval (ms). | +| `target_stat_interval_ms` | `50` | Interval for per-target statistics collection (ms). | | `max_concurrent_operations` | `64` | Max concurrent I/O operations. | | `score_threshold` | `0.7` | Score above which blobs are reorganized. | | `score_difference_threshold` | `0.05` | Min score delta to trigger reorganization. | | `flush_metadata_period_ms` | `5000` | Metadata flush interval (ms). | | `flush_data_period_ms` | `10000` | Data flush interval (ms). | | `flush_data_min_persistence` | `1` | Min persistence level (1 = temp-nonvolatile). | +| `metadata_log_path` | *(none)* | Write-ahead log + snapshot path for tag/blob metadata. Supports `${HOME}` expansion. | | `transaction_log_capacity` | `"32MB"` | Write-ahead log capacity. | +:::danger Metadata durability is opt-in +**`metadata_log_path` is required for data to survive a runtime reboot.** +Without it, a persistent tier's bytes survive but nothing remembers which +blob they belong to. The shipped default config sets it to +`${HOME}/.clio/cte_metadata_log`. +::: + +```yaml +performance: + metadata_log_path: "${HOME}/.clio/cte_metadata_log" + transaction_log_capacity: "32MB" +``` + +### GPU Metadata Cache (`gpu_metadata_cache`) + +Optional; mirrors tag/blob metadata into GPU memory so device-side kernels +can resolve blobs without a host round trip. + +| Parameter | Default | Description | +|-----------|---------|-------------| +| `enabled` | `false` | Enable the GPU-resident metadata mirror. | +| `capacity` | — | Size of the mirror (size string). | +| `max_blobs` | — | Maximum blob entries. | +| `max_tags` | — | Maximum tag entries. | + +--- + +## CTE Interposition Chain ChiMods + +Caching, replication, compression, and semantic-search indexing are separate +ChiMods that stack over the CTE core via `next_pool_id`. Each speaks the +core's own task interface, so a `clio::cte::core::Client` pointed at the top +of the chain works unchanged. + +``` +cache(563.0) → indexer(564.0) → [compressor(562.0) →] replication(561.0) → core(512.0) +``` + +| Module | Pool | Purpose | Key parameters | +|--------|------|---------|----------------| +| `clio_cte_cache` | `563.0` | Node-local untransformed copies (locality, zero-IPC SHM reads) | `next_pool_id`, `min_score` | +| `clio_cte_indexer` | `564.0` | BM25 term index serving `SemanticSearch` | `next_pool_id`, `index_log_path`, `index_sweep_period_ms`, `index_wal_compact_bytes`, `tag_re`, `blob_re` | +| `clio_cte_compressor` | `562.0` | Transparent compression (needs `CLIO_CTE_ENABLE_COMPRESS=ON`) | `next_pool_id`, `tracking_enabled` | +| `clio_cte_replication` | `561.0` | Fixed set of persistent replicas (reliability) | `next_pool_id`, `num_replicas`, `cache_score`, `replica_score`, `replicate_period_ms` | +| `clio_cte_filesystem` | `560.0` | POSIX-style filesystem the FUSE / POSIX adapters drive | `next_pool_id` | + +:::warning Ordering +Pools are created in file order and a pool cannot be created before the pool +its `next_pool_id` names. List them core → replication → compressor → +indexer → cache → filesystem. +::: + +:::note The CTE core no longer implements semantic search +`Method::kSemanticSearch` is served by `clio_cte_indexer`. Without an +indexer pool in the chain there is nothing to answer the query. +::: + +See [Cache / Replication / Indexing ChiMods](../sdk/context-transfer-engine/chimod-chain) +for the full behavior, parameter semantics, and module verbs. + --- ## CAE Module Parameters (`clio_cae_core`) +CAE is the assimilation / discovery entrypoint: CEE calls `ParseOmni` here, +and CAE forwards the data-path tasks it owns (`GetOrCreateTag`, `PutBlob`, +`GetBlob`, `SemanticSearch`) on to CTE at `next_pool_id`. + | Parameter | Required | Description | |-----------|----------|-------------| | `pool_name` | Yes | User-defined pool name. | | `pool_query` | Yes | Routing policy (`local`, `dynamic`, `broadcast`). | -| `pool_id` | Yes | Unique pool ID. Default CAE pool ID is `"400.0"`. | +| `pool_id` | Yes | Unique pool ID. Canonical CAE pool ID is `"400.0"`. | +| `next_pool_id` | No | CTE pool the data path is forwarded to (`"512.0"`). | +| `label_endpoint` | No | Ollama-compatible server URL for transparent LLM labeling. | +| `label_prompts` | No | Named prompt templates. | +| `label_matches` | No | Rules (`tag_re`, `blob_re`, `model`, `prompt`, `context_length`) selecting which blobs get labeled. | + +```yaml +- mod_name: clio_cae_core + pool_name: cae_main + pool_query: local + pool_id: "400.0" + next_pool_id: "512.0" +``` + +### Transparent LLM labeling (optional) + +When configured, `PutBlob` calls `model` on `label_endpoint` for every blob +whose tag and name match a rule, and stores the response as +`{blob_name}_label` in the same tag. Leave it out for a pure-passthrough CAE. ```yaml - mod_name: clio_cae_core - pool_name: clio_cae_core_pool + pool_name: cae_main pool_query: local pool_id: "400.0" + next_pool_id: "512.0" + label_endpoint: "http://127.0.0.1:11434" + label_prompts: + summarize: "Summarize the following text in one short sentence." + label_matches: + - tag_re: ".*\\.txt$" + blob_re: ".*" + model: "gemma3:1b" + prompt: "summarize" + context_length: 4096 ``` +:::danger Never put CAE in front of CTE at pool 512.0 +CAE only mirrors the four data-path method ids above. The rest of its method +ids **collide** with CTE's (CAE `kParseOmni` == CTE `kRegisterTarget` == 10), +so a CTE client hitting CAE for `RegisterTarget` / `TagQuery` / `BlobQuery` +is dispatched to the wrong handler and crashes. CAE gets its own pool +(`400.0`) and forwards; it does not interpose. +::: + --- ## Complete Examples +### The Shipped Default (`~/.clio/clio.yaml`) + +This is what `make install`, the pip wheel, and the `iowarp/deploy-cpu` +image seed. It brings up a DRAM bdev, the CTE core with a DRAM tier and a +lazily-grown persistent disk tier, CAE, and the full interposition chain. + +```yaml +networking: + port: 9413 + neighborhood_size: 32 + wait_for_restart: 30 + wait_for_restart_poll_period: 1 + +runtime: + num_threads: 4 + queue_depth: 1024 + local_sched: "default" + first_busy_wait: 10000 + learning_rate: 0.2 + +compose: + # === Block Device (DRAM) — required === + - mod_name: clio_bdev + pool_name: "ram::chi_default_bdev" + pool_query: local + pool_id: "301.0" + bdev_type: ram + capacity: "0g" # 0g = 80% of total system DRAM + + # === Context Assimilation Engine — its own pool, forwards to CTE === + - mod_name: clio_cae_core + pool_name: cae_main + pool_query: local + pool_id: "400.0" + next_pool_id: "512.0" + + # === Context Transfer Engine core === + - mod_name: clio_cte_core + pool_name: cte_main + pool_query: local + pool_id: "512.0" + storage: + - path: "ram::cte_ram_tier1" + bdev_type: "ram" + capacity_limit: "0g" + score: 1.0 + # Persistent tier the replication chimod keeps durable copies on. + - path: "${HOME}/.clio/cte_disk_tier.dat" + bdev_type: "file" + capacity_limit: "10GB" + score: 0.2 + persistence_level: "temporary" + performance: + metadata_log_path: "${HOME}/.clio/cte_metadata_log" + transaction_log_capacity: "32MB" + dpe: + dpe_type: "max_bw" + targets: + neighborhood: 1 + default_target_timeout_ms: 30000 + poll_period_ms: 5000 + + # === Interposition chain: replication → indexer → cache === + - mod_name: clio_cte_replication + pool_name: clio_cte_replication + pool_query: local + pool_id: "561.0" + next_pool_id: "512.0" + num_replicas: 1 + cache_score: 1.0 + replica_score: 0.2 + + # Optional; needs a build with CLIO_CTE_ENABLE_COMPRESS=ON. Enabling it + # also means re-pointing the indexer's next_pool_id at 562.0. + # - mod_name: clio_cte_compressor + # pool_name: clio_cte_compressor + # pool_query: local + # pool_id: "562.0" + # next_pool_id: "561.0" + + - mod_name: clio_cte_indexer + pool_name: clio_cte_indexer + pool_query: local + pool_id: "564.0" + next_pool_id: "561.0" + index_log_path: "${HOME}/.clio/cte_indexer_index" + + - mod_name: clio_cte_cache + pool_name: clio_cte_cache + pool_query: local + pool_id: "563.0" + next_pool_id: "564.0" + min_score: 0.5 + + # === Context Filesystem — what the FUSE / POSIX adapters drive === + - mod_name: clio_cte_filesystem + pool_name: clio_cte_filesystem + pool_query: local + pool_id: "560.0" + next_pool_id: "563.0" # chain top +``` + ### Minimal Single-Node +Only `clio_bdev` is strictly required. Trim to just the core when you want +storage without any policy layers: + ```yaml networking: port: 9413 @@ -289,6 +622,10 @@ compose: dpe_type: max_bw ``` +This config has no persistent tier and no `metadata_log_path`, so nothing +survives a restart, and no interposer, so `SemanticSearch` has no index to +answer from. + ### Multi-Tier RAM + NVMe + HDD ```yaml @@ -373,7 +710,13 @@ compose: ## Docker Deployment -IOWarp uses `memfd_create()` for shared memory on Linux, so no special `/dev/shm` configuration is needed. Only `mem_limit` matters for resource control. +IOWarp uses `memfd_create()` for shared memory on Linux, so no special +`/dev/shm` configuration is needed. Only `mem_limit` matters for resource +control. + +The `iowarp/deploy-cpu` image already ships the default config at +`/home/iowarp/.clio/clio.yaml` and runs as the `iowarp` user, so the image +works with no volumes at all. Mount over that path to override it: ```yaml # docker-compose.yml @@ -384,6 +727,8 @@ services: hostname: iowarp volumes: - ./clio.yaml:/home/iowarp/.clio/clio.yaml:ro + environment: + - CTP_LOG_LEVEL=info ports: - "9413:9413" mem_limit: 8g @@ -391,4 +736,151 @@ services: restart: unless-stopped ``` -For multi-node Docker deployments, mount a shared hostfile and set the `networking.hostfile` path accordingly. See [HPC Cluster](./hpc-cluster) for details. +Or mount the config anywhere and point `CLIO_SERVER_CONF` at it, which +sidesteps the container's home directory entirely: + +```yaml +services: + iowarp: + image: iowarp/deploy-cpu:latest + container_name: iowarp + hostname: iowarp + volumes: + - ./clio.yaml:/etc/iowarp/clio.yaml:ro + environment: + - CLIO_SERVER_CONF=/etc/iowarp/clio.yaml + ports: + - "9413:9413" + mem_limit: 8g + command: ["clio_run", "start"] + restart: unless-stopped +``` + +:::tip Persisting data across container restarts +The default config writes the persistent tier, metadata log, and index to +`${HOME}/.clio/`. Add a volume for that directory or those files land in the +container's writable layer and vanish with `docker compose down`: + +```yaml + volumes: + - ./clio.yaml:/home/iowarp/.clio/clio.yaml:ro + - iowarp-state:/home/iowarp/.clio + +volumes: + iowarp-state: +``` +::: + +For multi-node Docker deployments, mount a shared hostfile and set the +`networking.hostfile` path accordingly. See [HPC Cluster](./hpc-cluster) for +details. + +--- + +## Environment Variables + +Environment variables are read at process startup. Where a variable overlaps +a YAML key, the **environment wins** — it is applied after the config file is +parsed, so a deployment can retune without editing a config. + +:::note +Every variable uses the `CLIO_` prefix (or `CTP_` for transport-primitive +concerns). The old `CHI_` prefix is **no longer recognized** — see +[Deprecation Notes](../deprecation-notes). +::: + +### Configuration and startup + +| Variable | Default | Description | +|----------|---------|-------------| +| `CLIO_SERVER_CONF` | *(none)* | Path to the YAML config. Highest priority in the lookup order. | +| `CLIO_PORT` | `9413` | RPC listener port. Overrides `networking.port`. | +| `CLIO_SERVER_ADDR` | `127.0.0.1` | Address clients dial to reach the runtime. | +| `CLIO_BIND_ADDR` | `0.0.0.0` | Address the runtime's listener sockets bind to. | +| `CLIO_EPHEMERAL` | `0` | `1` starts the runtime **bare** — the `compose` section is skipped and pools are created explicitly. Equivalent to `clio_run start --ephemeral`. | +| `CLIO_NUM_THREADS` | *(from YAML)* | Worker-thread count. Last word after the config file — useful for forcing a single worker to test whether a failure depends on cross-thread task migration. | +| `CLIO_MAIN_SEGMENT_SIZE` | *(auto)* | Bounds the main task-data segment. Byte count or size string (`"512m"`); `0` restores the auto default. | +| `CLIO_TASK_PROGRESS_INTERVAL_MS` | `5000` | Cross-node task-validity check interval. | +| `CLIO_REPO_PATH` | *(none)* | Where the runtime looks for ChiMod shared libraries. | +| `CLIO_MEMFD_DIR` | `/tmp/clio_$USER` | Per-user directory holding the shared-memory segment files. | + +### Client and transport + +| Variable | Default | Description | +|----------|---------|-------------| +| `CLIO_IPC_MODE` | *(auto)* | Force a client transport: `SHM`, `IPC` (Unix domain socket), or `TCP`. When unset the client **probes for the fastest usable one** in that order — SHM if the local runtime's main segment exists, then IPC if the server bound a Unix socket, else TCP. Setting this bypasses the probe entirely. | +| `CLIO_WITH_RUNTIME` | — | Whether a client process should co-locate a runtime. | +| `CLIO_WAIT_SERVER` | `30` | Seconds to wait for a local runtime during client init. `0` = fail immediately, `-1` = wait forever. | +| `CLIO_CLIENT_RETRY_TIMEOUT` | `60` | Seconds a client retries a request against a restarted runtime. `0` = fail immediately, `-1` = retry forever. | +| `CLIO_CLIENT_TRY_NEW_SERVERS` | `0` | Non-zero lets a client fail over to other hosts from the hostfile when its server is unreachable. | +| `CLIO_NUM_CONTAINERS` | — | Containers per pool (parallelism within a pool). | +| `CLIO_FORCE_NET` | `0` | Route every task whose `PoolQuery` is not explicitly `Local()` through the network path, even single-node. Benchmarking / testing aid: it makes client-side fast paths stand down. | +| `CLIO_ZMQ_IO_THREADS` | `8` | ZeroMQ I/O threads. The default scales comfortably to ~512 nodes; raise it beyond that. | +| `CLIO_ZMQ_LOCAL_IPC` | *(on for macOS)* | Run the local client↔runtime ROUTER/DEALER over `ipc://` instead of TCP. The cross-node ROUTER is untouched, so multi-node TCP is unaffected. | +| `CLIO_LBM_THALLIUM_PROTOCOL` | — | Thallium/Mercury protocol string (e.g. `ofi+verbs`). | +| `CLIO_LBM_THALLIUM_RPC_THREADS` | — | Thallium RPC handler threads. | + +### Shared-memory ingest tuning + +| Variable | Default | Description | +|----------|---------|-------------| +| `CLIO_SHM_IN_SHARDS` | `0` (= worker count) | Parallel inbound SHM rings. Each worker drains its own shard, so more shards spread ingest across the pool with no extra threads. `1` restores a single ingest ring. | +| `CLIO_SHM_ASYNC_SEND` | `0` | Defer the SHM response send to a background thread. **Off by default** — it costs ~3× latency on latency-bound workloads. | +| `CLIO_SHM_CLIENT_SPIN_US` | `50` | Waiter spin-before-park budget, in microseconds. | + +### CTE + +| Variable | Default | Description | +|----------|---------|-------------| +| `CLIO_CTE_POOL` | `512.0` | `major.minor` — bind the process-wide CTE client singleton to an interposing pool (e.g. `563.0` for the chain top). | +| `CLIO_CTE_SHM_TAG_CAPACITY` | `65536` | Tag slots in the SHM metadata mirror. **Resident**, ~80 B per slot. | +| `CLIO_CTE_SHM_BLOB_CAPACITY` | `262144` | Blob slots in the SHM metadata mirror. **Resident**, ~376 B per slot (the defaults are ~100 MB of blob table). Capacity is fixed at creation — entries beyond it are simply not cached and those blobs keep using RPC. Sizing for 1M blobs wants ~2M slots (~0.79 GB), since the load factor caps useful occupancy at 7/8. | +| `CLIO_CTE_BATCHING` | `0` | Opt in to task-merge batching for `PutBlob` / `GetBlob`. | +| `CLIO_INDEXER_PASSIVE` | *(unset)* | Set to disable all indexer maintenance (forward-only interposition). Production triage kill switch. | +| `CLIO_CTE_FUSE_MOUNTPOINT` | *(none)* | Required when the FUSE binary is exec'd with a pre-opened `/dev/fuse` fd (e.g. Apptainer `--fusemount`), which does not communicate the mountpoint any other way. | + +### Context Filesystem (CFS) adapters + +| Variable | Default | Description | +|----------|---------|-------------| +| `CLIO_CFS_ASYNC_WRITES` | `1` | Allow `write(2)` to return before the runtime has the bytes. `0` restores blocking writes. | +| `CLIO_CFS_WRITE_WINDOW_BYTES` | — | Staging bytes allowed in flight before a write blocks on the oldest. Back-pressure, never a failure. | +| `CLIO_CFS_WRITE_WINDOW_COUNT` | `256` | In-flight write count before back-pressure. | +| `CLIO_CFS_SHM_FILE_CAPACITY` | `65536` | Path entries in the filesystem SHM attribute mirror. **Resident**, ~112 B each (~7 MB at the default). Paths beyond capacity keep using RPC. | + +### Block devices + +| Variable | Default | Description | +|----------|---------|-------------| +| `CLIO_BDEV_STATS_DIR` | `~/.clio/bdev_perf` | Directory holding per-bdev `.perf` files (measured latency/bandwidth plus the learned wall-clock model). | +| `CLIO_BDEV_PERSIST_STATS` | `1` | `0` disables persistence, so each start begins from a cold model. | + +### ADIOS2 adapter startup (large-scale MPI) + +At 512+ nodes the local daemon is busy serving cross-node SWIM probes and can +take many seconds to drain its accept queue. These control how the ADIOS2 +engine's ranks stagger and retry their client init so they do not stampede it. + +| Variable | Default | Description | +|----------|---------|-------------| +| `IOWARP_PPN` | `12` | Ranks per node. Used to derive each rank's node-local index; only local ranks contend for a given daemon. | +| `CLIO_INIT_STAGGER_MS` | `250` | Per-local-rank stagger step. At the defaults, 12 ranks spread over 3 s. | +| `CLIO_INIT_ATTEMPTS` | `60` | Client-init retry attempts. | +| `CLIO_INIT_SLEEP_MS` | `3000` | **Mean** backoff between attempts; the actual sleep is uniform over `[0.5×, 1.5×]` with a per-rank seed, so same-node ranks do not all retry on the same second. Default budget: 60 × ~3 s ≈ 3 minutes. | + +### Task scheduling + +| Variable | Default | Description | +|----------|---------|-------------| +| `CLIO_TASK_BATCHING` | `1` | Worker-loop task batching. `0` restores the pre-batching dequeue loop — the escape hatch for bisecting a regression to this phase rather than to a container's policy. | +| `CLIO_GPU_BLOCKS` / `CLIO_GPU_THREADS` | `1` / `32` | GPU orchestrator partitioning. Override `gpu.blocks` / `gpu.threads_per_block`. | + +### Logging + +| Variable | Default | Description | +|----------|---------|-------------| +| `CTP_LOG_LEVEL` | `info` | `debug`, `info`, `success`, `warning`, `error`, `fatal` (case-insensitive; numeric values also accepted). | +| `CTP_LOG_OUT` | *(console only)* | Path to a log file. Messages are also written there, without ANSI color codes. | + +See [Logging](#logging-environment-variables) above for the compile-time +threshold caveat and stream routing. diff --git a/docs/deployment/hpc-cluster.md b/docs/deployment/hpc-cluster.md index 93a2aec4..b483d2b1 100644 --- a/docs/deployment/hpc-cluster.md +++ b/docs/deployment/hpc-cluster.md @@ -77,7 +77,7 @@ export CLIO_IPC_MODE=TCP |----------|---------|-------------| | `CLIO_WITH_RUNTIME` | *(unset)* | When set to `1`, starts the runtime server in-process. When `0`, client-only mode. | -This variable is read by `CHIMAERA_INIT()`. If unset, the value of the `default_with_runtime` argument passed to `CHIMAERA_INIT()` is used instead. +This variable is read by `CLIO_RUNTIME_INIT()`. If unset, the value of the `default_with_runtime` argument passed to `CLIO_RUNTIME_INIT()` is used instead. --- diff --git a/docs/deployment/monitoring.md b/docs/deployment/monitoring.md index b21d925d..21b0e3f4 100644 --- a/docs/deployment/monitoring.md +++ b/docs/deployment/monitoring.md @@ -202,7 +202,7 @@ The dashboard reads the same config file as the runtime, using the same search o | `CLIO_SERVER_CONF` environment variable | **1st** | | `~/.clio/clio.yaml` | **2nd** | -Legacy paths (`~/.clio/chimaera.yaml`, `~/.chimaera/clio.yaml`, `~/.chimaera/chimaera.yaml`) and the legacy env var (`CHI_SERVER_CONF`) are also accepted. See [Deprecation Notes](../deprecation-notes) for the full list, and [Configuration](./configuration) for the file format. +Legacy paths (`~/.chimaera/…`) and the legacy `CHI_SERVER_CONF` env var are **no longer** accepted. See [Deprecation Notes](../deprecation-notes), and [Configuration](./configuration) for the file format. ### Connection lifecycle diff --git a/docs/deprecation-notes.md b/docs/deprecation-notes.md index c7f9c91f..432a2234 100644 --- a/docs/deprecation-notes.md +++ b/docs/deprecation-notes.md @@ -1,37 +1,38 @@ --- sidebar_position: 9 title: Deprecation Notes -description: Legacy names, config paths, env vars, and CLI invocations that still work as aliases of their canonical replacements. +description: Legacy names that have been removed, the ones that still work, and how to migrate a downstream project. --- # Deprecation Notes -This page lists everything in IOWarp that has been renamed or relocated but -**still works under its old name** as a backward-compatibility alias. If you -have existing scripts, pipelines, container images, or downstream C++ code -written against the legacy names, you do not need to migrate immediately: -every legacy form below resolves to the new canonical form at runtime. +IOWarp was renamed from `chimaera` / `hermes_shm` to `clio` / `clio_ctp`. +For a period, every legacy name kept working as a compatibility alias. -The rest of the documentation uses the **new canonical** name in every -example. This page is the single place to look up "what was the old name?" -or "what should I update to?". +:::danger The compatibility shims have been removed +The `CHI_*` env vars, `` and `` header shims, the +`hshm::` / `hipc::` namespace aliases, the `HSHM_*` and `CHI_*` macro +`#define`s, the `~/.chimaera/` config paths, and the `chimaera` CLI symlink +**no longer exist**. Code or scripts still written against them will fail to +compile or will silently fall back to defaults. -:::tip TL;DR -Old names still work. Migrate at your own pace. No removal is planned at -this time; an explicit deprecation announcement will precede any future -removal. +The [migration sweep](#migrating-a-downstream-project) below is now +mandatory rather than optional. ::: +The rest of the documentation uses the canonical name in every example. This +page is where you look up "what was the old name?" and "what should I update +to?". + --- -## CLI binaries +## What still works | Canonical | Legacy alias | How the alias works | |-----------|--------------|---------------------| -| `clio_run` | `chimaera` | `chimaera` is installed as a symlink to `clio_run` in the same `bin/` dir. The binary adapts its usage text via `argv[0]` so each name prints the right help. | -| `clio_cae` | `clio_cae_omni` | `clio_cae_omni` is installed as a symlink to `clio_cae`. | +| `clio_cae` | `clio_cae_omni` | `clio_cae_omni` is installed as a symlink to `clio_cae` (a copy on Windows). | -### Flat vs nested subcommand forms +### Flat vs nested `clio_run` subcommands `clio_run` accepts both a flat and a nested subcommand form: @@ -43,31 +44,49 @@ removal. | `clio_run refresh` | `clio_run repo refresh` | Both forms resolve to the same handler. The flat form is shorter and is the -form used in docs and examples. +form used in docs and examples. The newer subcommands — `compose`, `config`, +`migrate`, `monitor` — have **no** nested form. --- -## Configuration files and directories +## What has been removed -| Canonical | Legacy alias | How the alias works | -|-----------|--------------|---------------------| -| `~/.clio/clio.yaml` | `~/.clio/chimaera.yaml`, `~/.chimaera/clio.yaml`, `~/.chimaera/chimaera.yaml` | The runtime checks all four paths in priority order; first hit wins. `make install` and the pip wheel seed **both** `~/.clio/clio.yaml` and `~/.chimaera/chimaera.yaml` with identical content. | -| `clio_repo.yaml` | `chimaera_repo.yaml` | The CMake repo parser checks `clio_repo.yaml` first, then falls back to the legacy name. | -| `clio_mod.yaml` | `chimaera_mod.yaml` | Same — parser checks `clio_mod.yaml` first. | +### CLI binaries -See [Configuration Reference](./deployment/configuration) for the full -priority order including the env-var override. +| Canonical | Removed alias | +|-----------|---------------| +| `clio_run` | `chimaera` — the symlink is no longer installed. | ---- +### Configuration files and directories + +The runtime now looks **only** at `$CLIO_SERVER_CONF` and then +`~/.clio/clio.yaml`. These paths are no longer consulted: + +- `~/.clio/chimaera.yaml` +- `~/.chimaera/clio.yaml` +- `~/.chimaera/chimaera.yaml` + +Installers seed `~/.clio/clio.yaml` only. If your config still lives under +`~/.chimaera/`, move it or point `CLIO_SERVER_CONF` at it — otherwise the +runtime starts on the **built-in defaults**, which have an empty `compose` +section and therefore no storage tiers. + +Repository and module manifests are likewise `clio_repo.yaml` and +`clio_mod.yaml` only; the `chimaera_repo.yaml` / `chimaera_mod.yaml` +fallbacks are gone. The file contents did not change — only the filename. -## Environment variables +### Environment variables -Every runtime env var migrated from a `CHI_` prefix to a `CLIO_` prefix. -Both prefixes work; the runtime uses an internal `GetCompat()` helper that -reads `CLIO_` first and falls back to `CHI_`. +Every runtime env var moved from a `CHI_` prefix to a `CLIO_` prefix. The +`GetCompat()` helper that used to fall back to `CHI_` now reads +`CLIO_` and nothing else. -| Canonical | Legacy alias | -|-----------|--------------| +This is the failure mode to watch for: an unset variable is not an error, so +a launch script still exporting `CHI_SERVER_CONF` or `CHI_PORT` will start +the runtime successfully **on the wrong configuration**. Rename them: + +| Canonical | Removed alias | +|-----------|---------------| | `CLIO_SERVER_CONF` | `CHI_SERVER_CONF` | | `CLIO_IPC_MODE` | `CHI_IPC_MODE` | | `CLIO_WITH_RUNTIME` | `CHI_WITH_RUNTIME` | @@ -81,91 +100,57 @@ reads `CLIO_` first and falls back to `CHI_`. | `CLIO_LBM_THALLIUM_PROTOCOL`, `CLIO_LBM_THALLIUM_RPC_THREADS`, `CLIO_LBM_ZMQ_STATS` | `CHI_LBM_*` | | `CLIO_MEMFD_DIR`, `CLIO_TEST_DATA_DIR`, `CLIO_WAIT_SERVER`, `CLIO_ZMQ_IO_THREADS` | `CHI_*` (same suffix) | -Any new `CLIO_` env var you find in the docs has a `CHI_` legacy form; -they're documented by their canonical name only. - ---- - -## C++ headers +The rule is mechanical: `CHI_` → `CLIO_`. See the +[Configuration Reference](./deployment/configuration#environment-variables) +for the full current list. -The runtime header tree moved from `` to ``, -and the transport-primitives layer moved from `` to -``. The umbrella headers also renamed: +### C++ headers -| Canonical | Legacy alias | -|-----------|--------------| +| Canonical | Removed alias | +|-----------|---------------| | `` | `` | -| `` (whole tree) | `` | +| `` (whole tree) | `` | | `` | `` | -| `` (whole tree) | `` | +| `` (whole tree) | `` | -Every legacy path is a one-line forwarder shim that `#include`s the -canonical path, so any TU built against either form sees the same symbols. +The forwarder shims have been deleted; the legacy trees are not installed. ---- - -## C++ macros and namespaces +### C++ macros and namespaces -### Init / finalize - -| Canonical | Legacy alias | -|-----------|--------------| +| Canonical | Removed alias | +|-----------|---------------| | `CLIO_RUNTIME_INIT(mode, with_runtime)` | `CHIMAERA_INIT(mode, with_runtime)` | | `CLIO_RUNTIME_FINALIZE()` | `CHIMAERA_FINALIZE()` | - -### Singletons and accessors - -| Canonical | Legacy alias | -|-----------|--------------| | `CLIO_IPC`, `CLIO_ADMIN`, `CLIO_POOL_MANAGER`, `CLIO_CONFIG_MANAGER`, `CLIO_MODULE_MANAGER`, `CLIO_WORK_ORCHESTRATOR`, `CLIO_CUR_WORKER` | `CHI_*` (same suffix) | - -### Module / task macros - -| Canonical | Legacy alias | -|-----------|--------------| | `CLIO_CHIMOD_CC(...)`, `CLIO_TASK_CC(...)` | `CHI_CHIMOD_CC(...)`, `CHI_TASK_CC(...)` | | `CLIO_TASK_BODY_BEGIN`, `CLIO_TASK_BODY_END`, `CLIO_CO_AWAIT`, `CLIO_CO_RETURN` | `CHI_*` (same suffix) | | `CLIO_QUEUE_ALLOC_T`, `CLIO_TASK_ALLOC_T`, `CLIO_PRIV_ALLOC[_T]`, `CLIO_PRIV_SHARED_ALLOC[_T]` | `CHI_*` (same suffix) | - -Each `CLIO_*` form is a `#define` to the matching `CHI_*` macro, placed in -``. - -### Transport-primitive namespaces and macros - -| Canonical | Legacy alias | -|-----------|--------------| -| `ctp::` namespace | `hshm::` (`namespace hshm = ctp;`) | +| `ctp::` namespace | `hshm::` | | `ctp::ipc::` namespace | `hshm::ipc::`, `hipc::` | | `ctp::thread::`, `ctp::lbm::`, … | `hshm::thread::`, `hshm::lbm::`, … | -| 89 `CTP_*` macros (`CTP_CROSS_FUN`, `CTP_INLINE`, `CTP_GPU_FUN`, `CTP_MALLOC`, …) | matching `HSHM_*` forms (one `#define HSHM_X CTP_X` per macro) | +| `CTP_*` macros (`CTP_CROSS_FUN`, `CTP_INLINE`, `CTP_GPU_FUN`, `CTP_MALLOC`, …) | matching `HSHM_*` forms | -The compat header `` is auto-included by the -umbrella ``, so any TU that pulls the umbrella sees -all aliases. - ---- +The compat header `` no longer exists. -## What is **not** renamed +### Namespaces, enums, and CMake targets -These intentionally kept their `chimaera` / `hermes_shm` form to keep the -public ABI surface and dynamic-loader paths stable: +The `chi::` namespace, the `ChimaeraMode` enum, and the `chimaera_*` CMake +target and library filenames were previously kept for ABI and +dynamic-loader stability. They have since been renamed too: -- The C++ namespace `chi::` (still the canonical runtime namespace). -- CMake target names and installed shared-library filenames: - `chimaera_cxx`, `libchimaera_cxx.so`, `chimaera_admin_runtime`, - `libchimaera_admin_runtime.so`, … -- The `ChimaeraMode` enum class (`chi::ChimaeraMode::kClient` / `kServer`). -- Identifiers that have `chimaera` as a fragment, e.g. the `chi::Chimaera` - class itself and any internal `chimaera_manager` symbol. +| Canonical | Removed | +|-----------|---------| +| `clio::run::` namespace (plus `clio::run::priv::`, `clio::run::ipc::`) | `chi::` | +| `clio::run::RuntimeMode::kClient` / `kServer` | `ChimaeraMode::kClient` / `kServer` | +| `clio_run_cxx`, `clio_admin_client`, … | `chimaera_cxx`, `chimaera_admin_runtime`, … | -These can be done in a follow-up pass if there's demand; for now they're -load-bearing for everything that links against the runtime. +Downstream CMake must `find_package` / link the `clio_*` target names. --- ## Migrating a downstream project -You do not have to migrate. If you want to anyway, the mechanical sweep is: +The mechanical sweep: ```bash # Header paths @@ -193,22 +178,25 @@ grep -rl '\bhshm::\|\bhipc::\|\bHSHM_' \ -e 's/\bhipc::/ctp::ipc::/g' \ -e 's/\bHSHM_/CTP_/g' -# Env vars in launch scripts: just rename CHI_ -> CLIO_. +# Runtime namespace and mode enum +grep -rl '\bchi::\|\bChimaeraMode\b' \ + --include='*.cc' --include='*.h' --include='*.cpp' \ + | xargs sed -i -E \ + -e 's/\bchi::/clio::run::/g' \ + -e 's/\bChimaeraMode\b/RuntimeMode/g' + +# Env vars in launch scripts: rename CHI_ -> CLIO_. +grep -rl '\bCHI_[A-Z_]' --include='*.sh' --include='*.bash' --include='*.yaml' \ + | xargs sed -i -E 's/\bCHI_([A-Z_]+)\b/CLIO_\1/g' # Config YAMLs (in your repo, not the runtime's): # rename chimaera_repo.yaml -> clio_repo.yaml and # rename chimaera_mod.yaml -> clio_mod.yaml. No content changes needed. -``` -After the sweep your project will build against the canonical names and -remain compatible with both the current IOWarp Core release and any future -release that keeps the compat shims. - ---- - -## Removal timeline +# Per-user config, if you still have one under the legacy directory: +# mkdir -p ~/.clio && mv ~/.chimaera/chimaera.yaml ~/.clio/clio.yaml +``` -**None planned.** The shims, aliases, dual-name parsers, and symlinks are -intended to be permanent. If a specific alias is ever scheduled for removal, -the deprecation will be announced here with at least one full release of -overlap before the alias is dropped. +Then update your `CMakeLists.txt` to `find_package` / link the `clio_*` +target names, and grep your launch scripts one more time for a stray +`CHI_` — that is the failure that does not announce itself. diff --git a/docs/getting-started/installation.mdx b/docs/getting-started/installation.mdx index 23938908..cbf1357a 100644 --- a/docs/getting-started/installation.mdx +++ b/docs/getting-started/installation.mdx @@ -133,7 +133,26 @@ docker run -d -p 9413:9413 --memory=8g --name iowarp iowarp/deploy-cpu:latest cl ### Using Docker Compose -Create a `docker-compose.yml`: +The image already ships the default config at +`/home/iowarp/.clio/clio.yaml` and runs as the `iowarp` user, so the shortest +working `docker-compose.yml` is just: + +```yaml +services: + iowarp: + image: iowarp/deploy-cpu:latest + container_name: iowarp + hostname: iowarp + ports: + - "9413:9413" + mem_limit: 8g + command: ["clio_run", "start"] + restart: unless-stopped +``` + +In practice you want two more things: your own config, and a volume so the +persistent tier, metadata log, and search index survive `docker compose down` +(the default config writes all three under `~/.clio/`). ```yaml services: @@ -142,18 +161,31 @@ services: container_name: iowarp hostname: iowarp volumes: - - ./clio.yaml:/home/iowarp/.clio/clio.yaml:ro + - ./clio.yaml:/etc/iowarp/clio.yaml:ro + - iowarp-state:/home/iowarp/.clio + environment: + - CLIO_SERVER_CONF=/etc/iowarp/clio.yaml + - CTP_LOG_LEVEL=info ports: - "9413:9413" mem_limit: 8g command: ["clio_run", "start"] restart: unless-stopped + +volumes: + iowarp-state: ``` +Mounting the config at `/etc/iowarp/clio.yaml` and pointing +`CLIO_SERVER_CONF` at it keeps the config mount and the state volume from +fighting over the same directory. + Start the service: ```bash docker compose up -d +docker compose logs -f # "SpawnWorkerThreads" means the runtime is up +docker compose down ``` :::info diff --git a/docs/getting-started/quick-start.mdx b/docs/getting-started/quick-start.mdx index e7bc2856..c06745cf 100644 --- a/docs/getting-started/quick-start.mdx +++ b/docs/getting-started/quick-start.mdx @@ -63,13 +63,29 @@ You can edit it directly or point the runtime at a custom file with overwritten on reinstall. The default configuration starts **4 worker threads** on **port 9413** and -composes three modules automatically: +composes these modules automatically: -| Module | Purpose | -|--------|---------| -| `clio_bdev` | 512 MB RAM block device | -| `clio_cte_core` | Context Transfer Engine with a 512 MB RAM cache | -| `clio_cae_core` | Context Assimilation Engine | +| Module | Pool | Purpose | +|--------|------|---------| +| `clio_bdev` | `301.0` | DRAM block device (`0g` = 80% of total system DRAM) | +| `clio_cae_core` | `400.0` | Context Assimilation Engine — forwards its data path to CTE | +| `clio_cte_core` | `512.0` | Context Transfer Engine: a DRAM tier plus a 10 GB persistent disk tier at `~/.clio/cte_disk_tier.dat`, with metadata logging to `~/.clio/cte_metadata_log` | +| `clio_cte_replication` | `561.0` | One persistent replica of every blob on the disk tier | +| `clio_cte_indexer` | `564.0` | BM25 index that serves semantic search | +| `clio_cte_cache` | `563.0` | Node-local raw copies (the zero-IPC read fast path) | +| `clio_cte_filesystem` | `560.0` | POSIX-style filesystem the FUSE / POSIX adapters drive | + +The last four form the **interposition chain** — each speaks the CTE core's +own task interface and stacks over the next via `next_pool_id`: + +``` +cache(563.0) → indexer(564.0) → replication(561.0) → core(512.0) +``` + +Because they share the core's vocabulary, none of this changes how you call +CTE. Every layer is optional: delete its `compose` entry and re-point the +entry above it. See +[Cache / Replication / Indexing ChiMods](../sdk/context-transfer-engine/chimod-chain). ## 3. Start the Runtime @@ -162,10 +178,17 @@ Done! | Variable | Description | |----------|-------------| | `CLIO_SERVER_CONF` | Path to YAML configuration file (highest priority) | -| `CLIO_IPC_MODE` | Transport: `SHM` (shared memory), `TCP` (default), `IPC` (Unix socket) | +| `CLIO_IPC_MODE` | Force a transport: `SHM`, `IPC` (Unix socket), or `TCP`. Unset (the default) auto-probes for the fastest usable one in that order. | | `CLIO_PORT` | Override the RPC port (default: `9413`). Takes priority over YAML config. | -| `CLIO_SERVER_ADDR` | Override the server address clients connect to (default: `127.0.0.1`). | -| `CTP_LOG_LEVEL` | Logging verbosity: `debug`, `info`, `warning`, `error`, `fatal` | +| `CLIO_SERVER_ADDR` | Address clients dial to reach the runtime (default: `127.0.0.1`). | +| `CLIO_BIND_ADDR` | Address the runtime's listeners bind to (default: `0.0.0.0`). | +| `CLIO_EPHEMERAL` | `1` starts the runtime bare — the `compose` section is skipped. | +| `CLIO_CTE_POOL` | Bind the CTE client to an interposing pool, e.g. `563.0` for the chain top. | +| `CTP_LOG_LEVEL` | Logging verbosity: `debug`, `info`, `success`, `warning`, `error`, `fatal` | +| `CTP_LOG_OUT` | Also write log output to this file. | + +See the [Configuration Reference](../deployment/configuration#environment-variables) +for the full list. ## Next Steps diff --git a/docs/sdk/context-runtime/2.module_dev_guide.md b/docs/sdk/context-runtime/2.module_dev_guide.md index 5c1ad12d..865a3969 100644 --- a/docs/sdk/context-runtime/2.module_dev_guide.md +++ b/docs/sdk/context-runtime/2.module_dev_guide.md @@ -8,17 +8,17 @@ - [Task Definition](#task-definition-mod_name_tasksh) - [Client Implementation](#client-implementation-mod_name_clienthcc) - [Runtime Container](#runtime-container-mod_name_runtimehcc) - - [Execution Modes and Dynamic Scheduling](#execution-modes-and-dynamic-scheduling) + - [Dynamic Scheduling (ScheduleTask)](#dynamic-scheduling-scheduletask) 5. [Configuration and Code Generation](#configuration-and-code-generation) 6. [Task Development](#task-development) 7. [Synchronization Primitives](#synchronization-primitives) 8. [Pool Query and Task Routing](#pool-query-and-task-routing) 9. [Client-Server Communication](#client-server-communication) 10. [Memory Management](#memory-management) - - [CLIO_IPC Buffer Allocation](#chi_ipc-buffer-allocation) + - [CLIO_IPC Buffer Allocation](#clio_ipc-buffer-allocation) - [Task Data Structures](#task-data-structures) 11. [Build System Integration](#build-system-integration) -12. [External Module Development](#external-chimod-development) +12. [External Module Development](#external-module-development) 13. [Example Module](#example-module) ## Overview @@ -212,7 +212,7 @@ CreateParamsT GetParamsExample(bool do_compose, ``` The `clio::run::PoolConfig` struct carries the YAML compose entry: -- `mod_name_` - Module library name (e.g., `"chimaera_bdev"`) +- `mod_name_` - Module library name (e.g., `"clio_bdev"`) - `pool_name_` - Pool identifier or file path - `pool_id_` - Pool ID - `pool_query_` - Scheduling query (dynamic or local) @@ -768,7 +768,7 @@ Located at `chimods/clio_repo.yaml`, this file defines repository-wide settings: ```yaml # Repository Configuration -namespace: chimaera # MUST match namespace in all clio_mod.yaml files +namespace: clio::run # MUST match namespace in all clio_mod.yaml files version: 1.0.0 description: "CLIO Runtime Module Repository" @@ -790,7 +790,7 @@ Each Module must have its own configuration file specifying methods and metadata ```yaml # MOD_NAME Module Configuration module_name: MOD_NAME -namespace: chimaera # MUST match clio_repo.yaml namespace +namespace: clio::run # MUST match clio_repo.yaml namespace version: 1.0.0 # Inherited Methods (fixed IDs) @@ -2945,7 +2945,7 @@ struct ReadTask : public clio::run::Task { ## Build System Integration ### CMakeLists.txt Template -Module CMakeLists.txt files should use the standardized ChimaeraCommon.cmake functions for consistency and proper configuration: +Module CMakeLists.txt files should use the standardized ClioCoreCommon.cmake functions for consistency and proper configuration: ```cmake cmake_minimum_required(VERSION 3.10) @@ -3006,13 +3006,20 @@ add_chimod_client( - **`INCLUDE_DIRECTORIES`** (optional): Additional include directories beyond automatic ones **Automatic Behavior:** -- Creates target: `${NAMESPACE}_${CHIMOD_NAME}_client` +- Creates target: `${PACKAGE_NAME}_${CHIMOD_NAME}_client` - Creates alias: `${NAMESPACE}::${CHIMOD_NAME}_client` -- Automatically links core CLIO Runtime library (`chimaera::cxx` or `ctp::cxx`) -- For non-admin ChiMods: automatically links `chimaera_admin_client` +- Automatically links the core CLIO Runtime library (`clio::run::cxx`, falling back to `ctp::cxx` for external builds) +- For non-admin ChiMods: automatically links `clio_admin_client` and `clio_bdev_client` - Automatically includes module headers from `include/` directory - Installs library and headers with proper CMake export configuration +:::note `add_chimod_client()` vs `add_clio_module_client()` +`add_chimod_client()` / `add_chimod_runtime()` are thin macro wrappers that +forward to the canonical `add_clio_module_client()` / +`add_clio_module_runtime()`. Both spellings work and take identical +arguments. +::: + **Example:** ```cmake add_chimod_client( @@ -3046,12 +3053,12 @@ add_chimod_runtime( - **`INCLUDE_DIRECTORIES`** (optional): Additional include directories beyond automatic ones **Automatic Behavior:** -- Creates target: `${NAMESPACE}_${CHIMOD_NAME}_runtime` +- Creates target: `${PACKAGE_NAME}_${CHIMOD_NAME}_runtime` - Creates alias: `${NAMESPACE}::${CHIMOD_NAME}_runtime` -- Automatically defines `CHIMAERA_RUNTIME=1` for runtime code -- Automatically links core CLIO Runtime library (`chimaera::cxx` or `ctp::cxx`) -- Automatically links `rt` library for POSIX real-time support -- For non-admin ChiMods: automatically links both `chimaera_admin_runtime` and `chimaera_admin_client` +- Automatically defines `CLIO_RUNTIME=1` for runtime code +- Automatically links the core CLIO Runtime library (`clio::run::cxx`, falling back to `ctp::cxx` for external builds) +- Automatically links `rt` library for POSIX real-time support (Linux only — Windows ships its AIO support in `kernel32`/winsock) +- For non-admin ChiMods: automatically links both `clio_admin_runtime` and `clio_admin_client` - Automatically includes module headers from `include/` directory - Links to client library if it exists - Installs library and headers with proper CMake export configuration @@ -3109,45 +3116,66 @@ This eliminates the need for manual dependency configuration in individual Modul ### Target Naming and Linking #### Target Format -The system uses underscore-based target names for consistency with CMake conventions: -**Target Names:** -- **Runtime**: `${NAMESPACE}_${CHIMOD_NAME}_runtime` (e.g., `chimaera_admin_runtime`) -- **Client**: `${NAMESPACE}_${CHIMOD_NAME}_client` (e.g., `chimaera_admin_client`) +The C++ namespace comes from `clio_repo.yaml` (e.g. `clio::run`, +`clio::cte`). Because `::` is illegal in filenames, a **package name** is +derived from it by replacing `::` with `_` — `clio::run` → `clio_run`, +`clio::cte` → `clio_cte`. Raw target names and install paths use the package +name; CMake aliases use the namespace. + +**Target Names** (also the library filename, `lib.so`): +- **Runtime**: `${PACKAGE_NAME}_${CHIMOD_NAME}_runtime` (e.g., `clio_run_myMod_runtime`) +- **Client**: `${PACKAGE_NAME}_${CHIMOD_NAME}_client` (e.g., `clio_run_myMod_client`) **CMake Aliases (Recommended):** -- **Runtime**: `${NAMESPACE}::${CHIMOD_NAME}_runtime` (e.g., `chimaera::admin_runtime`) -- **Client**: `${NAMESPACE}::${CHIMOD_NAME}_client` (e.g., `chimaera::admin_client`) +- **Runtime**: `${NAMESPACE}::${CHIMOD_NAME}_runtime` (e.g., `clio::run::admin_runtime`) +- **Client**: `${NAMESPACE}::${CHIMOD_NAME}_client` (e.g., `clio::run::admin_client`) **Package Names:** -- Format: `${NAMESPACE}_${CHIMOD_NAME}` (e.g., `chimaera_admin`) -- Used with: `find_package(chimaera_admin REQUIRED)` -- Core package: `chimaera` (automatically included by `find_package(chimaera)`) +- Per-module package: `${PACKAGE_NAME}_${CHIMOD_NAME}` (e.g., `clio_run_admin`), installed under `lib/cmake/` +- Umbrella package: `clio-core` (legacy alias: `iowarp-core`). A single + `find_package(clio-core CONFIG REQUIRED)` provides all `ctp::*` targets, + `clio::run::cxx`, the admin client/runtime, and the ChiMod build + utilities — external projects normally need nothing else. + +:::note `LIB_NAME` overrides the derived name +Passing `LIB_NAME foo` to `add_chimod_client()` pins the target and library +filename to `foo_client` instead of the derived form. The in-tree `admin` +and `bdev` modules use this, which is why their targets are `clio_admin_client` +and `clio_bdev_client` rather than `clio_run_admin_client` / `clio_run_bdev_client`. +The `::` aliases are unaffected. +::: -**External Application Linking (Based on test/unit/external/CMakeLists.txt):** +**External Application Linking:** ```cmake -# External applications typically only need Module client libraries -# Core library dependencies are automatically included -find_package(chimaera REQUIRED) -find_package(chimaera_admin REQUIRED) +# External applications typically only need Module client libraries. +# The umbrella package provides all ctp::* targets, clio::run::cxx, +# clio::run::admin_client / admin_runtime, and the ChiMod build utilities. +find_package(clio-core CONFIG REQUIRED) +find_package(Threads REQUIRED) target_link_libraries(my_external_app - chimaera::admin_client # Admin client (includes all dependencies) + clio::run::admin_client # Admin client (includes all dependencies) ${CMAKE_THREAD_LIBS_INIT} # Threading support ) -# Note: chimaera::cxx is automatically included by Module client libraries +# Note: clio::run::cxx is automatically included by Module client libraries ``` **Internal Development Linking:** ```cmake # For internal development within the CLIO Runtime project target_link_libraries(internal_app - chimaera::admin_client # Module client - chimaera::bdev_client # BDev client + clio::run::admin_client # Module client + clio::run::bdev_client # BDev client # Core dependencies are automatically linked by Module libraries ) ``` +**Compatibility aliases.** Installed module packages also emit the historical +`clio_::` alias namespace (e.g. `clio_cte::core_client` alongside +`clio::cte::core_client`), plus `wrp_::` for non-`run` namespaces, so +existing downstream `target_link_libraries` lines keep resolving. + ### Automatic Dependencies The Module build functions automatically handle common dependencies: @@ -3161,13 +3189,13 @@ The Module build functions automatically handle common dependencies: - Creates both client and runtime shared libraries - Sets proper include directories (`include/`, `${CMAKE_SOURCE_DIR}/include`) - Automatically links core CLIO Runtime dependencies -- Sets required compile definitions (CHI_CHIMOD_NAME, CHI_NAMESPACE) +- Sets required compile definitions (`CLIO_RUNTIME=1` for runtime targets, plus `DEBUG` / `NDEBUG` per build config) - Configures proper build flags and settings **Simplified Development:** Module developers no longer need to manually specify: - `rt` library dependencies -- Admin Module dependencies (`chimaera_admin_runtime`, `chimaera_admin_client`) +- Admin Module dependencies (`clio_admin_runtime`, `clio_admin_client`) - Admin include directories - Core CLIO Runtime library dependencies - Common linking patterns @@ -3188,69 +3216,68 @@ The Module build functions automatically handle installation: When you call `add_chimod_client()` and `add_chimod_runtime()` with `CHIMOD_NAME YOUR_MODULE_NAME`, they create the following CMake targets using the underscore-based naming format: #### Target Naming System -- **Actual Target Names**: `${NAMESPACE}_${CHIMOD_NAME}_runtime` and `${NAMESPACE}_${CHIMOD_NAME}_client` +- **Actual Target Names**: `${PACKAGE_NAME}_${CHIMOD_NAME}_runtime` and `${PACKAGE_NAME}_${CHIMOD_NAME}_client` - **CMake Aliases**: `${NAMESPACE}::${CHIMOD_NAME}_runtime` and `${NAMESPACE}::${CHIMOD_NAME}_client` (**recommended**) -- **Package Names**: `${NAMESPACE}_${CHIMOD_NAME}` (for `find_package()`) +- **Package Names**: `${PACKAGE_NAME}_${CHIMOD_NAME}` (for `find_package()`) -#### Runtime Target: `${NAMESPACE}_${CHIMOD_NAME}_runtime` -- **Target Name**: `chimaera_YOUR_MODULE_NAME_runtime` (e.g., `chimaera_admin_runtime`, `chimaera_MOD_NAME_runtime`) -- **CMake Alias**: `chimaera::YOUR_MODULE_NAME_runtime` (e.g., `chimaera::admin_runtime`) - **recommended for linking** +#### Runtime Target: `${PACKAGE_NAME}_${CHIMOD_NAME}_runtime` +- **Target Name**: `clio_run_YOUR_MODULE_NAME_runtime` (or `clio_admin_runtime` when `LIB_NAME` pins it) +- **CMake Alias**: `clio::run::YOUR_MODULE_NAME_runtime` (e.g., `clio::run::admin_runtime`) - **recommended for linking** - **Type**: Shared library (`.so` file) - **Purpose**: Contains server-side execution logic, runs in the CLIO Runtime process - **Compile Definitions**: - - `CHI_CHIMOD_NAME="${CHIMOD_NAME}"` - Module name for runtime identification - - `CHI_NAMESPACE="${NAMESPACE}"` - Project namespace + - `CLIO_RUNTIME=1` - Selects the runtime-side code paths + - `DEBUG` / `NDEBUG` - Per build configuration - **Include Directories**: - `include/` - Local module headers - `$\{CMAKE_SOURCE_DIR\}/include` - Clio framework headers -- **Dependencies**: Links against `chimaera` library, rt library (automatic), admin dependencies (automatic) +- **Dependencies**: Links against `clio::run::cxx`, rt library (automatic, Linux), admin dependencies (automatic) -#### Client Target: `${NAMESPACE}_${CHIMOD_NAME}_client` -- **Target Name**: `chimaera_YOUR_MODULE_NAME_client` (e.g., `chimaera_admin_client`, `chimaera_MOD_NAME_client`) -- **CMake Alias**: `chimaera::YOUR_MODULE_NAME_client` (e.g., `chimaera::admin_client`) - **recommended for linking** -- **Type**: Shared library (`.so` file) +#### Client Target: `${PACKAGE_NAME}_${CHIMOD_NAME}_client` +- **Target Name**: `clio_run_YOUR_MODULE_NAME_client` (or `clio_admin_client` when `LIB_NAME` pins it) +- **CMake Alias**: `clio::run::YOUR_MODULE_NAME_client` (e.g., `clio::run::admin_client`) - **recommended for linking** +- **Type**: Shared library (`.so` file) - **Purpose**: Contains client-side API, runs in user processes - **Compile Definitions**: - - `CHI_CHIMOD_NAME="${CHIMOD_NAME}"` - Module name for client identification - - `CHI_NAMESPACE="${NAMESPACE}"` - Project namespace + - `DEBUG` / `NDEBUG` - Per build configuration (no `CLIO_RUNTIME`) - **Include Directories**: - `include/` - Local module headers - `$\{CMAKE_SOURCE_DIR\}/include` - Clio framework headers -- **Dependencies**: Links against `chimaera` library, admin dependencies (automatic) +- **Dependencies**: Links against `clio::run::cxx`, admin dependencies (automatic) #### Namespace Configuration The namespace is automatically read from `clio_repo.yaml` files. The system searches up the directory tree from the CMakeLists.txt location to find the first `clio_repo.yaml` file: **Main project `clio_repo.yaml`:** ```yaml -namespace: chimaera # Main project namespace +namespace: clio::run # Main project namespace ``` **Module repository `chimods/clio_repo.yaml`:** ```yaml -namespace: chimods # Modules get this namespace +namespace: clio::mods # Modules get this namespace ``` -This means modules in the `chimods/` directory will use the "chimods" namespace, creating targets like `chimods_admin_runtime`, while other components use the main project namespace. +This means modules in the `chimods/` directory will use the `clio::mods` namespace, creating targets like `clio_mods_admin_runtime` (the package name is the namespace with `::` replaced by `_`), while other components use the main project namespace. #### Example Output Files -For a module named "admin" with namespace "chimods" (from `chimods/clio_repo.yaml`), the build produces: +For a module named "admin" with namespace `clio::mods` (from `chimods/clio_repo.yaml`), the build produces: ``` -build/bin/libchimods_admin_runtime.so # Runtime library -build/bin/libchimods_admin_client.so # Client library +build/bin/libclio_mods_admin_runtime.so # Runtime library +build/bin/libclio_mods_admin_client.so # Client library ``` #### Using the Targets You can reference these targets in your CMakeLists.txt using the full target name: ```cmake # Add custom properties to the runtime target -set_target_properties(chimaera_${CHIMOD_NAME}_runtime PROPERTIES +set_target_properties(${CLIO_RUN_PACKAGE_NAME}_${CHIMOD_NAME}_runtime PROPERTIES VERSION 1.0.0 SOVERSION 1 ) # Add additional dependencies if needed -target_link_libraries(chimaera_${CHIMOD_NAME}_runtime PRIVATE some_external_lib) +target_link_libraries(${CLIO_RUN_PACKAGE_NAME}_${CHIMOD_NAME}_runtime PRIVATE some_external_lib) # Or use the global property to get the actual target name get_property(RUNTIME_TARGET GLOBAL PROPERTY ${CHIMOD_NAME}_RUNTIME_TARGET) @@ -3321,7 +3348,7 @@ This macro automatically generates all required extern "C" functions and gets th 1. Your runtime class must define a public typedef: `using CreateParams = your_namespace::CreateParams;` 2. Your CreateParams struct must have: `static constexpr const char* chimod_lib_name = "your_module_name";` -**IMPORTANT:** The `chimod_lib_name` should **NOT** include the `_runtime` suffix. The module manager automatically appends `_runtime` when loading the library. For example, use `"chimaera_mymodule"` not `"chimaera_mymodule_runtime"`. +**IMPORTANT:** The `chimod_lib_name` should **NOT** include the `_runtime` suffix. The module manager automatically appends `_runtime` when loading the library. For example, use `"clio_run_mymodule"` not `"clio_run_mymodule_runtime"`. Example: ```cpp @@ -3491,7 +3518,7 @@ cmake --install build --prefix /usr/local This installs: - Core CLIO Runtime library (`libcxx.so`) -- Module libraries (`libchimaera_admin_runtime.so`, `libchimaera_admin_client.so`, etc.) +- Module libraries (`libclio_admin_runtime.so`, `libclio_admin_client.so`, etc.) - CMake package configuration files for external discovery - Header files for development @@ -3530,7 +3557,7 @@ Create a `clio_repo.yaml` file in your repository root to define the namespace: ```yaml # Repository-level configuration -namespace: myproject # Your custom namespace (replaces "chimaera") +namespace: myproject # Your custom namespace (replaces "clio::run") ``` This namespace will be used for: @@ -3551,13 +3578,15 @@ set(CMAKE_CXX_STANDARD_REQUIRED ON) # Find required CLIO Runtime packages # These packages are installed by 'cmake --install build --prefix /usr/local' -find_package(chimaera REQUIRED) # Core CLIO Runtime (automatically includes ChimaeraCommon.cmake) -find_package(chimaera_admin REQUIRED) # Admin Module (often required) +# `clio-core` is the canonical umbrella package; `iowarp-core` still resolves +# to the same Config. It brings in clio::run::cxx, the admin client/runtime, +# every ctp::* target, and the ChiMod build utilities. +find_package(clio-core CONFIG REQUIRED) # Set CMAKE_PREFIX_PATH if CLIO Runtime is installed in a custom location -# set(CMAKE_PREFIX_PATH "/path/to/[namespace]/install" ${CMAKE_PREFIX_PATH}) +# set(CMAKE_PREFIX_PATH "/path/to/install" ${CMAKE_PREFIX_PATH}) -# ChimaeraCommon.cmake utilities are automatically included by find_package(chimaera) +# ClioCoreCommon.cmake is included automatically by find_package(clio-core). # This provides add_chimod_client(), add_chimod_runtime(), and other build functions # Add subdirectories containing your ChiMods @@ -3572,7 +3601,7 @@ Each Module's `CMakeLists.txt` uses the standard CLIO Runtime build utilities: cmake_minimum_required(VERSION 3.20) # Create both client and runtime libraries using standard CLIO Runtime utilities -# These functions are provided by ChimaeraCommon.cmake (automatically included via find_package(chimaera)) +# These functions are provided by ClioCoreCommon.cmake (included via find_package(clio-core)) # Creates targets: my_namespace_my_module_client, my_namespace_my_module_runtime # Creates aliases: my_namespace::my_module_client, my_namespace::my_module_runtime add_chimod_client( @@ -3601,14 +3630,13 @@ Once installed, external applications can find and link to your Module. Based on ```cmake # External application CMakeLists.txt +find_package(clio-core CONFIG REQUIRED) # Core CLIO Runtime + utilities + admin find_package(my_namespace_my_module REQUIRED) # Your Module package -find_package(chimaera REQUIRED) # Core CLIO Runtime (automatically includes utilities) -find_package(chimaera_admin REQUIRED) # Admin Module (often required) # Simple linking pattern - Module libraries include all dependencies target_link_libraries(my_external_app my_namespace::my_module_client # Your Module client - chimaera::admin_client # Admin client (if needed) + clio::run::admin_client # Admin client (if needed) ${CMAKE_THREAD_LIBS_INIT} # Threading support ) # Core CLIO Runtime library is automatically included by Module dependencies @@ -3671,7 +3699,7 @@ make install The build system will automatically: - Link all necessary core CLIO Runtime dependencies -- Link against `chimaera::admin_client` and `chimaera::admin_runtime` (for non-admin modules) +- Link against `clio::run::admin_client` and `clio::run::admin_runtime` (for non-admin modules) - Generate libraries with your custom namespace: `libmyproject_my_module_runtime.so` - Configure proper include paths and dependencies @@ -3755,12 +3783,12 @@ void example() { External Module development requires these components to be installed: -1. **Core Package**: `chimaera` (includes main library and ChimaeraCommon.cmake utilities) -2. **Admin Module**: `chimaera::admin_client` and `chimaera::admin_runtime` (required for most modules) +1. **Core Package**: `clio-core` (includes the main library and the ClioCoreCommon.cmake utilities) +2. **Admin Module**: `clio::run::admin_client` and `clio::run::admin_runtime` (required for most modules) 3. **CMake Configs**: Package discovery files (automatically installed with packages) 4. **Headers**: All Clio framework headers (installed with packages) -All build utilities (`add_chimod_client()`, `add_chimod_runtime()`) are automatically available via `find_package(chimaera)`. +All build utilities (`add_chimod_client()`, `add_chimod_runtime()`) are automatically available via `find_package(clio-core CONFIG)`. If CLIO Runtime is installed in a custom location, set `CMAKE_PREFIX_PATH`: @@ -3770,10 +3798,10 @@ export CMAKE_PREFIX_PATH="/path/to/[namespace]/install:$CMAKE_PREFIX_PATH" ### Common External Development Issues -**ChimaeraCommon.cmake Not Found:** +**ClioCoreCommon.cmake Not Found:** - Ensure CLIO Runtime was installed with `cmake --install build --prefix ` - Verify `CMAKE_PREFIX_PATH` includes the CLIO Runtime installation directory -- Check that `find_package(chimaera REQUIRED)` succeeded (ChimaeraCommon.cmake is included automatically) +- Check that `find_package(clio-core CONFIG REQUIRED)` succeeded (ClioCoreCommon.cmake is included automatically) **Library Name Mismatch:** - Ensure `CreateParams::chimod_lib_name` exactly matches your namespace and module name @@ -3790,7 +3818,7 @@ export CMAKE_PREFIX_PATH="/path/to/[namespace]/install:$CMAKE_PREFIX_PATH" ### External Module Checklist - [ ] **Repository Configuration**: `clio_repo.yaml` with custom namespace -- [ ] **CMake Setup**: Root CMakeLists.txt finds `chimaera` package +- [ ] **CMake Setup**: Root CMakeLists.txt finds the `clio-core` package - [ ] **Module Configuration**: `clio_mod.yaml` with method definitions - [ ] **Library Name**: `CreateParams::chimod_lib_name` matches namespace pattern - [ ] **C++ Namespace**: All code uses custom namespace consistently @@ -3901,7 +3929,7 @@ TaskResume Custom(shared_ptr& task) { ## Custom Namespace Configuration ### Overview -While the default namespace is `chimaera`, you can customize the namespace for your Module modules. This is useful for: +While the default namespace is `clio::run`, you can customize the namespace for your Module modules. This is useful for: - **Project Branding**: Use your own project or company namespace - **Avoiding Conflicts**: Prevent naming conflicts with other Module collections - **Module Organization**: Group related modules under a custom namespace @@ -3956,7 +3984,7 @@ namespace mycompany::your_module { #### 3. **CMake Library Names** The CMake system automatically uses your custom namespace. Libraries will be named: -- Default: `libchimaera_module_runtime.so`, `libchimaera_module_client.so` +- Default: `libclio_run_module_runtime.so`, `libclio_run_module_client.so` - Custom: `libmycompany_module_runtime.so`, `libmycompany_module_client.so` #### 4. **Runtime Integration** @@ -4172,7 +4200,7 @@ When creating a new CLIO Runtime module, ensure you have: - [ ] **Include clio_runtime.h**: In methods file for GLOBAL_CROSS_CONST macro - [ ] **GLOBAL_CROSS_CONST constants**: Use namespace constants, not enum class - [ ] Proper install targets configured -- [ ] Links against chimaera library +- [ ] Links against the `clio::run::cxx` library ### Common Pitfalls to Avoid - [ ] ❌ **CRITICAL: Not updating pool_id_ in Create methods** (leads to incorrect pool ID for subsequent operations) @@ -4411,7 +4439,7 @@ When adding compose support to a Module: - [ ] Parse all module-specific parameters from YAML config - [ ] Handle optional parameters with defaults - [ ] Use `ctp::ConfigParse::ParseSize()` for size strings -- [ ] Include `` and `` in tasks header +- [ ] Include `` and `` in tasks header - [ ] Test with compose configuration before release ### Example Admin Module LoadConfig diff --git a/docs/sdk/context-runtime/20.base-modules/1.admin.md b/docs/sdk/context-runtime/20.base-modules/1.admin.md index b21563b7..49dfb11f 100644 --- a/docs/sdk/context-runtime/20.base-modules/1.admin.md +++ b/docs/sdk/context-runtime/20.base-modules/1.admin.md @@ -17,11 +17,11 @@ The Admin Module is a critical component of the CLIO Runtime system that manages To use the Admin Module in external projects: ```cmake -find_package(chimaera_admin REQUIRED) # Admin Module package -find_package(chimaera REQUIRED) # Core CLIO Runtime (automatically includes ChimaeraCommon.cmake) +find_package(clio-core CONFIG REQUIRED) # Core CLIO Runtime + ClioCoreCommon.cmake +find_package(clio_run_admin REQUIRED) # Admin Module package target_link_libraries(your_application - chimaera::admin_client # Admin client library + clio::run::admin_client # Admin client library ${CMAKE_THREAD_LIBS_INIT} # Threading support ) # Core CLIO Runtime library dependencies are automatically included by Module libraries diff --git a/docs/sdk/context-runtime/20.base-modules/2.bdev.md b/docs/sdk/context-runtime/20.base-modules/2.bdev.md index 45a430d8..118dbeac 100644 --- a/docs/sdk/context-runtime/20.base-modules/2.bdev.md +++ b/docs/sdk/context-runtime/20.base-modules/2.bdev.md @@ -19,13 +19,12 @@ The Bdev (Block Device) Module provides a high-performance interface for block d To use the Bdev Module in external projects: ```cmake -find_package(chimaera_bdev REQUIRED) # BDev Module package -find_package(chimaera_admin REQUIRED) # Admin Module (always required) -find_package(chimaera REQUIRED) # Core CLIO Runtime (automatically includes ChimaeraCommon.cmake) +find_package(clio-core CONFIG REQUIRED) # Core CLIO Runtime + admin + ClioCoreCommon.cmake +find_package(clio_run_bdev REQUIRED) # BDev Module package target_link_libraries(your_application - chimaera::bdev_client # Bdev client library - chimaera::admin_client # Admin client (required) + clio::run::bdev_client # Bdev client library + clio::run::admin_client # Admin client (required) ${CMAKE_THREAD_LIBS_INIT} # Threading support ) # Core CLIO Runtime library dependencies are automatically included by Module libraries diff --git a/docs/sdk/context-runtime/20.base-modules/3.MOD_NAME.md b/docs/sdk/context-runtime/20.base-modules/3.MOD_NAME.md index 9cd3a9ab..08dfbf8d 100644 --- a/docs/sdk/context-runtime/20.base-modules/3.MOD_NAME.md +++ b/docs/sdk/context-runtime/20.base-modules/3.MOD_NAME.md @@ -19,17 +19,17 @@ The MOD_NAME Module serves as a template and example module for developing custo To use the MOD_NAME Module in external projects: ```cmake -find_package(chimaera-MOD_NAME REQUIRED) -find_package(chimaera-admin REQUIRED) # Always required -find_package(chimaera-core REQUIRED) +# The umbrella package brings in clio::run::cxx, the admin client/runtime, +# and every ctp::* target. +find_package(clio-core CONFIG REQUIRED) +find_package(clio_run_MOD_NAME REQUIRED) # MOD_NAME Module package target_link_libraries(your_application - chimaera::MOD_NAME_client # MOD_NAME client library - chimaera::admin_client # Admin client (required) - chimaera::cxx # Main chimaera library - ctp::cxx # HermesShm library + clio::run::MOD_NAME_client # MOD_NAME client library + clio::run::admin_client # Admin client (required) ${CMAKE_THREAD_LIBS_INIT} # Threading support ) +# clio::run::cxx and ctp::cxx are transitive dependencies of the Module clients ``` ### Required Headers @@ -43,7 +43,7 @@ target_link_libraries(your_application ## API Reference -### Client Class: `chimaera::MOD_NAME::Client` +### Client Class: `clio::run::MOD_NAME::Client` The MOD_NAME client provides the primary interface for module operations and testing. @@ -260,7 +260,7 @@ clio::run::Future AsyncWaitTest( ## Task Types ### CreateTask -Container creation task for the MOD_NAME module. This is an alias for `chimaera::admin::GetOrCreatePoolTask`. +Container creation task for the MOD_NAME module. This is an alias for `clio::run::admin::GetOrCreatePoolTask`. **Key Fields:** - Inherits from `BaseCreateTask` with MOD_NAME-specific `CreateParams` @@ -301,7 +301,7 @@ Task for testing recursive task.Wait() functionality. - `result_`: Test result code (OUT) ### DestroyTask -Standard destruction task (alias for `chimaera::admin::DestroyTask`). +Standard destruction task (alias for `clio::run::admin::DestroyTask`). ## Configuration diff --git a/docs/sdk/context-runtime/3.module_test_guide.md b/docs/sdk/context-runtime/3.module_test_guide.md index 0c58977d..fe5dd009 100644 --- a/docs/sdk/context-runtime/3.module_test_guide.md +++ b/docs/sdk/context-runtime/3.module_test_guide.md @@ -26,7 +26,7 @@ export CLIO_SERVER_CONF="/path/to/clio_default.yaml" Tests use the same configuration format as production. The runtime looks for configuration in this order: 1. `CLIO_SERVER_CONF` environment variable -2. `~/.clio/clio.yaml` (legacy: `~/.chimaera/chimaera.yaml`) +2. `~/.clio/clio.yaml` A minimal test configuration: diff --git a/docs/sdk/context-runtime/5.scheduler.md b/docs/sdk/context-runtime/5.scheduler.md index 7df1517d..28ccc1c4 100644 --- a/docs/sdk/context-runtime/5.scheduler.md +++ b/docs/sdk/context-runtime/5.scheduler.md @@ -725,7 +725,7 @@ This ensures tasks in the same group (e.g., operations on the same file handle) ### Code Reference See implementation in: -- Header: `context-runtime/include/chimaera/scheduler/default_sched.h` +- Header: `context-runtime/include/clio_runtime/scheduler/default_sched.h` - Implementation: `context-runtime/src/scheduler/default_sched.cc` ## Best Practices @@ -1138,9 +1138,9 @@ class ExampleScheduler : public Scheduler { ## References -- **Scheduler Interface**: `context-runtime/include/chimaera/scheduler/scheduler.h` +- **Scheduler Interface**: `context-runtime/include/clio_runtime/scheduler/scheduler.h` - **DefaultScheduler**: `context-runtime/src/scheduler/default_sched.cc` -- **Container Base**: `context-runtime/include/chimaera/container.h` +- **Container Base**: `context-runtime/include/clio_runtime/container.h` - **IpcManager (RouteTask/RouteLocal)**: `context-runtime/src/ipc_manager.cc` - **WorkOrchestrator**: `context-runtime/src/work_orchestrator.cc` - **Configuration**: [Configuration Reference](../../deployment/configuration) diff --git a/docs/sdk/context-transfer-engine/chimod-chain.md b/docs/sdk/context-transfer-engine/chimod-chain.md new file mode 100644 index 00000000..0b56c866 --- /dev/null +++ b/docs/sdk/context-transfer-engine/chimod-chain.md @@ -0,0 +1,511 @@ +--- +sidebar_position: 2 +title: Cache / Replication / Indexing ChiMods +description: The CTE interposition chain — node-local caching, persistent replication, and the semantic-search index, stacked over the CTE core. +--- + +# The CTE Interposition Chain + +The CTE core (`clio_cte_core`, pool `512.0`) stores blobs. Everything +*policy* — where a durable copy lives, which node keeps a hot copy, what is +searchable — is implemented by separate ChiMods that **interpose** on the +core's own task interface. + +An interposer is a pool that: + +1. speaks the CTE core's **method ids and task structs** verbatim, +2. **overrides** a handful of data verbs (`PutBlob`, `GetBlob`, + `GetBlobSize`, `MultiPutBlob`, …), and +3. **forwards every other core method** to the pool named by its + `next_pool_id` untouched. + +The consequence is that interposition is completely transparent to callers. +A `clio::cte::core::Client` pointed at the *top* of the chain works +unchanged — no new API, no new client class. You choose your policy by +choosing which pool you address. + +``` + clio::cte::core::Client (or the CFS / FUSE / POSIX adapters) + │ + ▼ + cache 563.0 node-local raw copies (locality) + │ + ▼ + indexer 564.0 BM25 term index (search) + │ + ▼ + [ compressor 562.0 transparent compression ] (encoding, optional) + │ + ▼ + replication 561.0 persistent replica set (reliability) + │ + ▼ + core 512.0 blobs, targets, DPE, organizer +``` + +The `clio_cte_filesystem` ChiMod (pool `560.0`, driven by the FUSE and POSIX +adapters) sits above the whole thing and points its own `next_pool_id` at +the chain top. + +:::info Separation of concerns +Each layer owns exactly one axis: **replication** = reliability, +**cache** = locality, **compressor** = encoding, **indexer** = search. They +compose because they all speak the same task vocabulary, and each is +independently removable — delete its `compose` entry and re-point the entry +above it. +::: + +--- + +## Addressing the chain + +Three equivalent ways to make a client talk to an interposer instead of the +raw core: + +```bash +# 1. Environment variable (no code changes at all) +export CLIO_CTE_POOL=563.0 # top of the standard chain +``` + +```cpp +// 2. Construct a core client on the interposer's pool id +#include +#include + +clio::cte::core::Client client(clio::cte::cache::kCachePoolId); // 563.0 +``` + +```yaml +# 3. From another ChiMod's compose entry — next_pool_id names the chain +- mod_name: clio_cte_filesystem + pool_id: "560.0" + next_pool_id: "563.0" +``` + +`CLIO_CTE_POOL` is honored **only** by `ContentTransferEngine::ClientInit()` +(the process-wide CTE client singleton). It is deliberately *not* applied in +`Client::Init()` or the constructor: runtime-internal module clients build +`Client(next_pool_id)` to reach the pool below them, and redirecting those +would make an interposer forward to itself. + +### Well-known pool ids + +| ChiMod | Library | Pool id | Pool name constant | +|--------|---------|---------|--------------------| +| Filesystem | `clio_cte_filesystem` | `560.0` | `filesystem::kCfsPoolId` | +| Replication | `clio_cte_replication` | `561.0` | `replication::kReplicationPoolId` | +| Compressor | `clio_cte_compressor` | `562.0` | — | +| Cache | `clio_cte_cache` | `563.0` | `cache::kCachePoolId` | +| Indexer | `clio_cte_indexer` | `564.0` | `indexer::kIndexerPoolId` | +| CTE core | `clio_cte_core` | `512.0` | `core::kCtePoolId` | + +Each module's own verbs (`ReplicateBlob`, `FlushTag`, `ReindexScan`, …) are +numbered at **100 and above**, deliberately outside the core's method-id +space, so an interposer can carry both vocabularies without collision. + +:::warning Compose ordering matters +The runtime creates `compose` entries in file order, and a pool cannot be +created before the pool its `next_pool_id` names. Put each entry **after** +its target: core → replication → compressor → indexer → cache → filesystem. +::: + +--- + +## Replication ChiMod (`clio_cte_replication`) + +**Reliability.** The CTE core already knows how to *store* replicas and +address them (`Context::replica_`). The replication module decides **what +gets copied where**. + +### Behavior + +- **Write-through to a fixed replica set.** Every default `PutBlob` through + this pool also writes replicas `1..num_replicas`, each stamped + `REPLICA_FIXED | REPLICA_PERSISTENT`. Those flags tell the CTE organizer + two things: never migrate this copy, and never place it on a volatile + tier. The replicas only move if a device is evacuated. +- **Asynchronous by default.** A put acks after the **primary** write; + a periodic sweep (`ReplicateSweepTask`, `replicate_period_ms`) re-copies + each dirty blob's *current* primary bytes, so rapid overwrites coalesce + into a single replica write. Set `replicate_period_ms: 0` for synchronous + write-through — the put then blocks until every replica is written. +- **Replica-served reads with primary re-cache.** Reads serve the primary + when it is present. When the organizer (or a reboot) dropped the DRAM + primary, the read is transparently served from a disk replica and the + primary is re-created at `cache_score`. +- **`replica_score` doubles as the drop threshold.** When the primary's + score sinks below the best persistent replica's score, the primary is + *dropped* rather than migrated — the durable copy already exists, so + there is nothing to move. + +### Configuration + +| Key | Default | Description | +|-----|---------|-------------| +| `next_pool_id` | *(none)* | CTE core pool whose blobs this module replicates (e.g. `"512.0"`). | +| `num_replicas` | `1` | Persistent copies maintained per put. | +| `cache_score` | `1.0` | Score given to the primary on a replica → primary re-cache. High pins it to the fast tier. | +| `replica_score` | `0.2` | Score stamped on the persistent replicas, and the drop threshold for the primary. Should mirror the slow tier the durable copies live on. | +| `replicate_period_ms` | `50` | Async sweep cadence in ms. `0` = synchronous write-through. | + +```yaml +- mod_name: clio_cte_replication + pool_name: clio_cte_replication + pool_query: local + pool_id: "561.0" + next_pool_id: "512.0" + num_replicas: 1 + cache_score: 1.0 + replica_score: 0.2 + replicate_period_ms: 50 +``` + +:::caution Persistent replicas need a persistent tier +`REPLICA_PERSISTENT` constrains placement to non-volatile storage. A CTE +`storage` entry left at the default `persistence_level: volatile` can never +satisfy that request. At least one tier must declare +`persistence_level: temporary` or `long_term` — see +[Configuration Reference](../../deployment/configuration#storage-tiers-storage). +::: + +### Module verbs + +Beyond the interposed data path, the replication client exposes two explicit +operations: + +```cpp +#include + +clio::cte::replication::Client repl(clio::cte::replication::kReplicationPoolId, + clio::cte::core::kCtePoolId); + +// Bring replica 1 of one blob up to date with the primary. +auto f = repl.AsyncReplicateBlob(tag_id, "checkpoint_0", /*replica=*/1); + +// "Make this dataset's RAM cache durable": ReplicateBlob every blob in the +// tag whose score is >= min_score (0 = all of them). +clio::cte::replication::Context ctx; +ctx.min_persistence_level_ = 1; // pin replica blocks to persistent tiers +auto g = repl.AsyncFlushTag(tag_id, /*replica=*/1, /*min_score=*/0.0f, ctx); +``` + +`ReplicateBlobTask` reports `bytes_copied_`; `FlushTagTask` reports +`blobs_replicated_` and `bytes_copied_`. Both copy primary → replica in +chunks (one `GetBlob` + one replica-targeted `PutBlob` per chunk), so an +arbitrarily large blob never needs a full-size bounce buffer. + +--- + +## Cache ChiMod (`clio_cte_cache`) + +**Locality.** The cache module keeps a node-local, **untransformed** copy of +each blob in the CTE core's `REPLICA_CACHE` slot while pushing the +authoritative bytes down the chain. It is the standard **top** of the chain. + +### Why the raw copy matters + +The local copy is stored uncompressed and unencoded. That is what keeps the +**zero-IPC shared-memory read path alive** — a compressed primary alone +would force every read back through an RPC round trip, because the client +cannot decode it in place. The cache copy also serves raw task reads +directly. + +### Behavior + +- **Asynchronous write-through, not write-back.** A put lands below — + authoritatively — *before* the ack. There is no dirty state and no flush + period at this layer; the layers underneath keep their own async + machinery. What is "asynchronous" here is the work those lower layers + defer, not the durability of the put itself. +- **Writer-local routing.** Reads *and* writes route submitter-local, so the + hot path (a rank touching its own data) is entirely `PoolQuery::Local()`. + Only the authoritative hop crosses nodes. +- **Coherent by construction.** The node-local copy is written *first*, then + the authoritative put carries `origin_node_` to the blob's owner. The + owner atomically invalidates every *other* registered copy and registers + this one under the write token. Because the local write strictly precedes + registration, any later foreign write's invalidation always catches it. +- **Never a partial prefix.** A local copy is created only when the put + demonstrably covers the whole blob. When that cannot be known up front, + the copy is created speculatively and the owner verifies it + (`REPLICA_VERIFY_COMPLETE`); if the blob pre-existed beyond this put, the + owner refuses the registration and the speculative copy is dropped. +- **The owner node keeps no cache copy.** Its primary is already node-local + and zero-IPC readable, and an owner-side copy would sit outside the + register/invalidate protocol — it would never be invalidated and the node + would serve its own stale bytes forever. +- **Misses fall through.** A read miss goes down the chain to the blob's + owner and re-populates the local copy (in 4 MiB chunks), coherently. + +### Configuration + +| Key | Default | Description | +|-----|---------|-------------| +| `next_pool_id` | *(none)* | Pool that receives the authoritative bytes — the indexer in the standard chain. | +| `min_score` | `0.5` | Score **floor** for cache copies (propagated as `Context::replica_min_score_`). The organizer never rescores a cache replica below it; only genuine capacity pressure on the tier evicts one. | + +```yaml +- mod_name: clio_cte_cache + pool_name: clio_cte_cache + pool_query: local + pool_id: "563.0" + next_pool_id: "564.0" # indexer + min_score: 0.5 +``` + +The cache module has **no data-path client wrappers** by design. Point a +`clio::cte::core::Client` at pool `563.0` and ordinary `PutBlob` / `GetBlob` +get the caching behavior. `clio::cte::cache::Client` exists only to create +the pool programmatically: + +```cpp +#include +#include + +clio::cte::cache::CacheConfig cfg; +cfg.next_pool_id_ = clio::cte::indexer::kIndexerPoolId; +cfg.min_score_ = 0.5f; + +clio::cte::cache::Client cache; +auto f = cache.AsyncCreateCache(clio::run::PoolQuery::Local(), + clio::cte::cache::kCachePoolName, + clio::cte::cache::kCachePoolId, cfg); +``` + +--- + +## Indexer ChiMod (`clio_cte_indexer`) + +**Search.** The indexer owns the reverse index that serves +`Method::kSemanticSearch`. **The CTE core no longer implements semantic +search** — without an indexer pool in the chain, there is nothing to answer +the query. + +### Behavior + +- **Indexing is off the ack path.** A `PutBlob` is forwarded down first, + then costs only an **O(1) coalesced dirty-key enqueue**. N overwrites of a + hot blob cost **one** re-tokenize, because the drain re-reads the blob's + *current* bytes. +- **The drain.** A periodic `IndexSweepTask` (`index_sweep_period_ms`) + tokenizes the dirty set. Setting the period to `0` disables the sweep + entirely — the index is then updated only when a search runs (lazy + indexing). +- **Read-your-writes.** `SemanticSearch` drains the pending set *before* + evaluating the query, so every acked mutation is visible to the very next + search regardless of sweep cadence. +- **Search reads no blobs.** BM25 (Okapi, `k1 = 1.5`, `b = 0.75`) is + evaluated over the maintained term-frequency index. Corpus statistics + (`avgdl`, `df`) are computed over the *matched* slice only — the query + means "find the best matches within this regex slice", not "rank against + everything CTE has ever seen". +- **Tokenization.** Lowercase alphanumeric runs of length ≥ 2, ASCII, + C-locale semantics. +- **Which verbs are intercepted.** `PutBlob`, `MultiPutBlob`, `DelBlob`, + `DelTag`, `TruncateBlob`, `RenameTag` maintain the index and are forwarded + down unchanged. `SemanticSearch` is served locally. Everything else passes + through. + +### Placement in the chain + +The indexer sits **above the compressor** so its read-backs see logical +(uncompressed) bytes, and **above the core** because the mutating verbs must +flow through it for the index to stay current. Search clients address it — +or any pool that forwards down into it. + +### Persistence + +The index is derived state, but rebuilding it from storage is expensive, so +the module persists its own snapshot plus an append-only WAL: + +- `index_log_path` set → a restart **restores** the index and never rescans + storage. +- `index_log_path` empty → in-memory only. A restart comes up **empty** and + re-indexes incrementally: a tag's first insertion backfills that tag, and + `ReindexScan` backfills the rest on demand. + +The periodic sweep compacts (snapshot + truncate) once the WAL exceeds +`index_wal_compact_bytes`. + +### Scope + +Only blobs whose **full tag name** matches `tag_re` **and** whose blob name +matches `blob_re` are tokenized (`std::regex_match`, full-string — use +`.*pattern.*` for substring matching). Out-of-scope content costs nothing: +no scan, no index memory, no WAL. + +### Configuration + +| Key | Default | Description | +|-----|---------|-------------| +| `next_pool_id` | *(none)* | Pool the non-search verbs are forwarded to (replication, or the compressor when enabled). | +| `index_sweep_period_ms` | `100` | Async drain cadence in ms. `0` = lazy (index updated only on search). | +| `tag_re` | `".*"` | Index scope: only matching tag names are tokenized. | +| `blob_re` | `".*"` | Index scope: only matching blob names are tokenized. | +| `index_log_path` | *(empty)* | Snapshot path; the WAL is `.wal`. Empty = in-memory only. Supports `${HOME}` expansion. | +| `index_wal_compact_bytes` | `8MB` | WAL size that triggers snapshot + truncate compaction. | + +```yaml +- mod_name: clio_cte_indexer + pool_name: clio_cte_indexer + pool_query: local + pool_id: "564.0" + next_pool_id: "561.0" # replication (562.0 with compressor) + index_log_path: "${HOME}/.clio/cte_indexer_index" + index_sweep_period_ms: 100 + index_wal_compact_bytes: "8MB" + tag_re: ".*" + blob_re: ".*" +``` + +### Searching + +`SemanticSearch` is a core verb, so it is called through the ordinary core +client: + +```cpp +#include +#include + +// Address the chain top (or the indexer directly). +clio::cte::core::Client client(clio::cte::cache::kCachePoolId); + +auto fut = client.AsyncSemanticSearch(/*tag_regex=*/".*\\.txt", + /*blob_regex=*/".*", + /*query_text=*/"turbulence simulation", + /*k=*/10); +CLIO_CO_AWAIT(fut); +for (const auto &r : fut->results_) { + // r.tag_name_, r.blob_name_, r.score_ (BM25; higher = better) +} +``` + +Both regexes use `std::regex_match` (full-string), matching `BlobQuery` +semantics. + +### Backfilling existing data + +Pre-existing cold data enters the index in exactly two ways: a tag's +first-insertion backfill, or an explicit scan. + +```cpp +#include + +clio::cte::indexer::Client idx(clio::cte::indexer::kIndexerPoolId, + clio::cte::core::kCtePoolId); + +// Enumerate matching blobs below and ENQUEUE them; the async drain indexes. +auto f = idx.AsyncReindexScan(/*tag_regex=*/".*", /*blob_regex=*/".*"); +CLIO_CO_AWAIT(f); +// f->blobs_enqueued_ +``` + +The requested regexes are **intersected** with the module's configured +`tag_re` / `blob_re` scope. The default `PoolQuery` is `Broadcast()`, so one +call covers every node. + +### Triage + +`CLIO_INDEXER_PASSIVE=1` turns the module into a pure forwarder: no index +maintenance at all. It is the production kill switch, and the measured +baseline that separates interposition cost from indexing cost. + +--- + +## Compressor ChiMod (`clio_cte_compressor`) + +**Encoding.** Optional; requires a build with `CLIO_CTE_ENABLE_COMPRESS=ON`. +Compresses transparently on the way down and decompresses on the way out. + +| Key | Default | Description | +|-----|---------|-------------| +| `next_pool_id` | *(none)* | Pool below — replication in the standard chain. | +| `tracking_enabled` | `true` | Track per-tag consumer node sets from `Decompress` requests and route `Compress` placement toward the most recent consumer of the same tag. `false` falls back to pure hash routing. | + +To enable it, uncomment its `compose` entry **and** re-point the indexer's +`next_pool_id` at `562.0`. + +--- + +## Putting it together + +The full standard chain as shipped in the default `~/.clio/clio.yaml`. Note +the ordering: every entry comes after the entry its `next_pool_id` names. + +```yaml +compose: + # ... clio_bdev and clio_cte_core (512.0) first ... + + - mod_name: clio_cte_replication + pool_name: clio_cte_replication + pool_query: local + pool_id: "561.0" + next_pool_id: "512.0" + num_replicas: 1 + cache_score: 1.0 + replica_score: 0.2 + + # - mod_name: clio_cte_compressor # optional, needs CLIO_CTE_ENABLE_COMPRESS=ON + # pool_name: clio_cte_compressor + # pool_query: local + # pool_id: "562.0" + # next_pool_id: "561.0" + + - mod_name: clio_cte_indexer + pool_name: clio_cte_indexer + pool_query: local + pool_id: "564.0" + next_pool_id: "561.0" # 562.0 with the compressor enabled + index_log_path: "${HOME}/.clio/cte_indexer_index" + + - mod_name: clio_cte_cache + pool_name: clio_cte_cache + pool_query: local + pool_id: "563.0" + next_pool_id: "564.0" + min_score: 0.5 + + - mod_name: clio_cte_filesystem + pool_name: clio_cte_filesystem + pool_query: local + pool_id: "560.0" + next_pool_id: "563.0" # chain top +``` + +What a put through the chain top now does: + +1. **cache** writes the node-local raw copy, then sends the authoritative + put down; +2. **indexer** forwards it and enqueues the dirty key; +3. **compressor** (if enabled) encodes it; +4. **replication** writes the primary, acks, and sweeps the persistent + replicas up to date; +5. **core** places the blocks via the DPE and records the metadata. + +And a read: **cache** serves the raw local copy — including over the +zero-IPC SHM fast path — or falls through to the owner and re-populates. + +### Trimming the chain + +Every layer is optional. Remove the entry and re-point the one above it: + +| You don't need | Remove | Re-point | +|----------------|--------|----------| +| Durable copies | `clio_cte_replication` | indexer's `next_pool_id` → `512.0` | +| Semantic search | `clio_cte_indexer` | cache's `next_pool_id` → `561.0` | +| Node-local caching | `clio_cte_cache` | filesystem's `next_pool_id` → `564.0` | +| The whole chain | all of the above | filesystem's `next_pool_id` → `512.0`, or address `512.0` directly | + +--- + +## Related environment variables + +| Variable | Description | +|----------|-------------| +| `CLIO_CTE_POOL` | `major.minor` — bind the process-wide CTE client singleton to an interposing pool. | +| `CLIO_INDEXER_PASSIVE` | Set to `1` to disable all index maintenance (forward-only). | +| `CLIO_CTE_SHM_TAG_CAPACITY` | Tag slots in the SHM metadata mirror (default `65536`, ~80 B each, resident). | +| `CLIO_CTE_SHM_BLOB_CAPACITY` | Blob slots in the SHM metadata mirror (default `262144`, ~376 B each, resident). | + +See the [Configuration Reference](../../deployment/configuration#environment-variables) +for the full list. diff --git a/docs/sdk/context-transfer-engine/cte.md b/docs/sdk/context-transfer-engine/cte.md index 7072a784..65344221 100644 --- a/docs/sdk/context-transfer-engine/cte.md +++ b/docs/sdk/context-transfer-engine/cte.md @@ -1,3 +1,9 @@ +--- +sidebar_position: 1 +title: CTE Core API +description: Blob storage API, storage tiers, data placement, and configuration for the Context Transfer Engine core. +--- + # Core API Documentation ## Overview @@ -19,7 +25,7 @@ CTE Core implements a Module (CLIO Runtime Module) that integrates with the CLIO - CMake 3.20 or higher - C++17 compatible compiler -- Clio framework (chimaera and chimaera_admin packages) +- Clio framework (`clio-core` umbrella package, which provides `clio::run::cxx` and the admin Module) - yaml-cpp library - Python 3.7+ (for Python bindings) - nanobind (for Python bindings) @@ -50,8 +56,7 @@ To use CTE Core in your CMake project, follow the patterns established in the MO ```cmake # Find required Clio framework packages -find_package(chimaera REQUIRED) # Core Clio framework -find_package(chimaera_admin REQUIRED) # Admin Module (required) +find_package(clio-core CONFIG REQUIRED) # Core Clio framework + admin Module # Find CTE Core Module package find_package(clio_cte_core REQUIRED) # CTE Core Module @@ -64,7 +69,7 @@ target_link_libraries(my_app PRIVATE clio_cte::core_client # CTE Core client library # clio_cte::core_runtime # Optional - if you need runtime functionality - # chimaera::admin_client # Optional - if you need admin functionality + # clio::run::admin_client # Optional - if you need admin functionality ) # Note: Include directories are handled automatically by the Module targets @@ -86,7 +91,7 @@ CTE Core follows the CLIO Runtime Module naming conventions: The CTE Core Module targets automatically include all required dependencies: - **Core CLIO Runtime Framework**: Automatically linked via `clio_cte::core_client` target -- **Admin Module**: Available via `chimaera::admin_client` if needed +- **Admin Module**: Available via `clio::run::admin_client` if needed - **Include Paths**: Automatically configured by Module targets - **System Dependencies**: Handled by the build system (threading, YAML, etc.) @@ -109,8 +114,7 @@ set(CMAKE_CXX_STANDARD_20) set(CMAKE_CXX_STANDARD_REQUIRED ON) # Find required packages -find_package(chimaera REQUIRED) # Core Clio framework -find_package(chimaera_admin REQUIRED) # Admin Module +find_package(clio-core CONFIG REQUIRED) # Core Clio framework + admin Module find_package(clio_cte_core REQUIRED) # CTE Core Module # Find additional dependencies @@ -124,7 +128,7 @@ add_executable(my_cte_app main.cpp) target_link_libraries(my_cte_app clio_cte::core_client # CTE Core client (required) # clio_cte::core_runtime # Optional - if needed - # chimaera::admin_client # Optional - if needed + # clio::run::admin_client # Optional - if needed ${CMAKE_THREAD_LIBS_INIT} # Threading support ) ``` @@ -1795,7 +1799,7 @@ void example() { Enable debug logging by setting environment variables: ```bash -export CHIMAERA_LOG_LEVEL=DEBUG +export CTP_LOG_LEVEL=debug export CTE_LOG_LEVEL=DEBUG ``` diff --git a/docs/sdk/context-transfer-engine/gpu-inf-mem.md b/docs/sdk/context-transfer-engine/gpu-inf-mem.md index 190aa2fe..fefb66ed 100644 --- a/docs/sdk/context-transfer-engine/gpu-inf-mem.md +++ b/docs/sdk/context-transfer-engine/gpu-inf-mem.md @@ -1,3 +1,9 @@ +--- +sidebar_position: 4 +title: GPU Infinite Memory (UVM) +description: Software-managed GPU demand paging over the CUDA VMM primitives. +--- + # GPU Infinite Memory (UVM) The `clio_cte_uvm` module provides a **software-managed GPU demand-paging