Skip to content

docs: sync with repo — CTE chimod chain, env vars, default compose - #18

Merged
lukemartinlogan merged 3 commits into
mainfrom
docs/cte-chimod-chain-and-env-vars
Aug 12, 2026
Merged

lukemartinlogan merged 3 commits into
mainfrom
docs/cte-chimod-chain-and-env-vars

Conversation

@lukemartinlogan

@lukemartinlogan lukemartinlogan commented Aug 8, 2026 •

Copy link
Copy Markdown
Contributor

Compared the docs against the current state of iowarp/clio-core and fixed what had drifted.

New page: Cache / Replication / Indexing ChiMods

docs/sdk/context-transfer-engine/chimod-chain.md documents the CTE interposition chain:

cache(563.0) → indexer(564.0) → [compressor(562.0) →] replication(561.0) → core(512.0)

It explains what an interposer is — a pool that speaks the CTE core's own method ids and task structs, overrides a handful of data verbs, and forwards everything else to next_pool_id, so a plain clio::cte::core::Client works unchanged — and then each layer:

  • Replication — write-through to a fixed REPLICA_FIXED | REPLICA_PERSISTENT set, the async sweep (replicate_period_ms, 0 = synchronous), replica-served reads with primary re-cache, replica_score doubling as the drop threshold, plus ReplicateBlob / FlushTag.
  • Cache — why the copy is stored raw (it's what keeps the zero-IPC SHM read path alive for compressed blobs), writer-local routing, the write-local-then-register coherence protocol, speculative-copy verification, why the owner node deliberately keeps no copy, and min_score as a score floor.
  • Indexer — indexing is off the ack path (O(1) coalesced enqueue), sweep vs. lazy, the read-your-writes drain before every search, BM25 over the matched slice only, snapshot+WAL persistence vs. in-memory, scope regexes, ReindexScan, and CLIO_INDEXER_PASSIVE.

Plus addressing (CLIO_CTE_POOL), compose ordering rules, and a trim-the-chain table.

Environment variables

Added a full reference section to deployment/configuration.md: startup, client/transport, SHM ingest tuning (CLIO_SHM_IN_SHARDS / _ASYNC_SEND / _CLIENT_SPIN_US), CTE, CFS adapters, bdev stats, task scheduling, logging, and the ADIOS2 large-scale init-stagger vars.

Corrected CLIO_IPC_MODE in configuration.md and quick-start.mdx — it was documented as "TCP default", but unset actually auto-probes SHM → IPC → TCP.

Default compose file

The docs showed a three-module compose. Replaced with the actually-shipped clio_default.yaml: CAE at its own pool 400.0 forwarding to CTE, a persistent disk tier with persistence_level, performance.metadata_log_path, and the full chain.

Also documented config keys that had no coverage: learning_rate, main_segment_size, metadata_segment_size, conf_dir, the gpu and swim sections, existing_pool_id / existing_pool_module, gpu_metadata_cache, and CAE transparent LLM labeling.

Refreshed both Docker Compose examples, including a volume for ~/.clio/ — the default config now writes the persistent tier, metadata log, and search index there, so without one they vanish on docker compose down.

FUSE adapter: Linux, macOS, and Windows

The FUSE adapter page documented Linux only, but the adapter runs on all three platforms — the callbacks and the CTE data path below them are identical everywhere; only the kernel backend and the mount mechanics differ.

Added a platform support matrix and split Installation, Mount, and Unmount into per-platform tabs:

Linux macOS Windows
Backend libfuse3 macFUSE 5 (ships libfuse3) WinFsp 2.0
Ships in the wheel Yes No — source build Yes
Mountpoint Any directory Directory (kext) or under /Volumes (FSKit) Drive letter or directory
Unmount fusermount3 -u umount / diskutil unmount force Stop the daemon
Discovery pkg-config fuse3 pkg-config fuse3 WINFSP_ROOT (no .pc file)

The details that actually bite:

  • macOS is a source build — the wheel does not ship clio_cte_fuse. The release-mac-fuse preset builds it against macFUSE 5. Mounting needs either the kext (one-time Recovery-Mode approval + reboot) or the kext-free FSKit backend (-o backend=fskit, macFUSE 5.1+ on macOS 15.4+, mountpoint must be under /Volumes). CI enforces the macOS build and unit/ops suites but the mount smoke is a non-blocking probe, so this is the least-proven path and the page says so.
  • Windows needs ADDLOCAL=ALL on the WinFsp MSI — the Developer feature is what provides inc/fuse3 + the import library. WinFsp ships no pkg-config file, so it's located by path (WINFSP_ROOT, default %ProgramFiles(x86)%\WinFsp).
  • A missing backend skips the adapter rather than failing the build, so the symptom is a silently absent binary. The troubleshooting section leads with that.

Also documented the Apptainer --fusemount path (CLIO_CTE_FUSE_MOUNTPOINT is mandatory since Apptainer communicates the mountpoint through neither argv nor the environment; needs CAP_SYS_ADMIN in the userns) — including that it is compiled out of the published Linux wheel, because manylinux ships libfuse 3.10.2 and the custom-io path needs 3.14+ headers at build time. Plus the CI mount-smoke scripts as an end-to-end deployment check, and a platform-keyed troubleshooting section.

The supported-operations table was stale. rename, chmod, chown, symlink, readlink, link, statfs, and all four xattr ops are implemented, and truncate is a real AsyncTruncate rather than the documented "updates cached size". Replaced the "not yet supported" list with what genuinely isn't (RENAME_EXCHANGE/RENAME_WHITEOUT, fallocate punch/collapse/insert) and documented the caching semantics — kernel attr/entry caches are off because there is no invalidation upcall, page cache is on because mmap depends on it.

Finally, noted in the adapter comparison that POSIX/STDIO/VFD are Linux-only (glibc ELF/dlsym interceptor, no macOS or Windows port), so FUSE is the only transparent-interception option on the other two.

Two things worth a closer look

1. deprecation-notes.md was wrong in a dangerous direction. It claimed the CHI_* env vars, <chimaera/…> / <hermes_shm/…> headers, the hshm:: / hipc:: / HSHM_* / CHI_* macros, the ~/.chimaera/ config paths, and the chimaera CLI symlink all still worked as aliases. None of them do — GetCompat() now reads only CLIO_<suffix>, the shim trees are gone, and all four config-lookup candidates are ~/.clio/clio.yaml.

The failure mode is quiet: an unset env var is not an error, so a launch script still exporting CHI_SERVER_CONF or CHI_PORT starts the runtime successfully on the wrong configuration. Rewrote the page around what was removed, made the migration sweep mandatory rather than optional, and kept the two aliases that genuinely survive (clio_cae_omni, and the nested clio_run runtime … / clio_run repo refresh forms).

2. CMake naming was stale across six pages. The real targets are clio::run::admin_client / clio_cte::core_client, the umbrella package is find_package(clio-core CONFIG REQUIRED), the helper is ClioCoreCommon.cmake, and the runtime define is CLIO_RUNTIME=1 (not CHIMAERA_RUNTIME=1). Fixed in the module dev guide, the three base-module pages, cte.md, the scheduler guide, monitoring, hpc-cluster, and the module test guide.

Also

  • Fixed three pre-existing broken TOC anchors in the module dev guide.
  • Added sidebar_position / frontmatter to the CTE pages so the section orders sensibly.

Verification

npx docusaurus build completes with no broken links and no broken anchors. Every claim above was checked against the source in clio-core (task/config headers, clio_default.yaml, config_manager.cc, ipc_manager.cc, ClioCoreCommon.cmake, the Dockerfiles) rather than inferred.

🤖 Generated with Claude Code

lukemartinlogan and others added 3 commits August 8, 2026 04:33
Compare docs against the current source and fix what drifted.

New page (sdk/context-transfer-engine/chimod-chain.md): the CTE
interposition chain — cache(563) -> indexer(564) -> [compressor(562)] ->
replication(561) -> core(512). Covers what an interposer is (speaks the
core's method ids, overrides a few data verbs, forwards the rest via
next_pool_id, so a plain core::Client works unchanged), then each layer:

  - replication: write-through to a REPLICA_FIXED|REPLICA_PERSISTENT set,
    async sweep (replicate_period_ms, 0 = synchronous), replica-served
    reads with primary re-cache, replica_score as the drop threshold,
    ReplicateBlob/FlushTag.
  - cache: why the copy is raw (keeps the zero-IPC SHM read path alive),
    writer-local routing, the write-local-then-register coherence
    protocol, speculative-copy verification, why the owner keeps no copy,
    min_score as a floor.
  - indexer: O(1) coalesced enqueue off the ack path, sweep vs lazy, the
    read-your-writes drain before search, BM25 over the matched slice,
    snapshot+WAL persistence vs in-memory, scope regexes, ReindexScan,
    CLIO_INDEXER_PASSIVE.

Environment variables: add a full reference section to
deployment/configuration.md — startup, client/transport, SHM ingest
tuning (CLIO_SHM_IN_SHARDS/_ASYNC_SEND/_CLIENT_SPIN_US), CTE, CFS
adapters, bdev stats, task scheduling, logging, and the ADIOS2
large-scale init-stagger vars. CLIO_IPC_MODE was documented as "TCP
default"; unset actually auto-probes SHM -> IPC -> TCP.

Default compose: the docs showed a 3-module compose. Replace with the
shipped clio_default.yaml (CAE at its own pool 400.0 forwarding to CTE,
a persistent disk tier with persistence_level, metadata_log_path, and
the full chain), document the new config keys (learning_rate,
main/metadata_segment_size, conf_dir, gpu and swim sections,
existing_pool_id, gpu_metadata_cache, CAE LLM labeling), and refresh
both docker-compose examples, including a volume for ~/.clio/ now that
the default config writes persistent state there.

Deprecation notes were wrong in a dangerous direction: they claimed the
CHI_* env vars, <chimaera/...>/<hermes_shm/...> headers,
hshm::/hipc::/HSHM_*/CHI_* macros, ~/.chimaera/ config paths and the
chimaera CLI symlink still worked. None do — GetCompat() reads only
CLIO_<suffix>, the shim trees are gone, and all config-lookup candidates
are ~/.clio/clio.yaml. A script still exporting CHI_SERVER_CONF starts
successfully on the wrong config. Rewrite around what was removed, make
the migration sweep mandatory, and keep the two aliases that survive
(clio_cae_omni, the nested `clio_run runtime ...` forms).

CMake naming was stale across six pages: real targets are
clio::run::admin_client / clio_cte::core_client, the umbrella package is
find_package(clio-core CONFIG REQUIRED), the helper is
ClioCoreCommon.cmake, and the runtime define is CLIO_RUNTIME=1.

Also fix three pre-existing broken TOC anchors in the module dev guide
and add sidebar_position/frontmatter to the CTE pages. Site builds with
no broken links or anchors.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The FUSE adapter page was Linux-only, but the adapter runs on all three
platforms — the callbacks and the CTE data path below them are identical
everywhere; only the kernel backend and mount mechanics differ.

Add a platform support matrix (backend, wheel availability, mountpoint
rules, unmount, discovery mechanism, CI coverage) and split Installation,
Mount, and Unmount into per-platform tabs:

  - Linux: libfuse3 via pkg-config; shipped in the wheel (libfuse3 itself
    is deliberately not bundled, so the distro packages are still needed).
  - macOS: macFUSE 5, which does ship libfuse3 — the release-mac-fuse
    preset builds it. The wheel does NOT ship the binary, so macOS is a
    source build. Mounting needs either the kext (one-time Recovery-Mode
    approval + reboot) or the kext-free FSKit backend (-o backend=fskit,
    macFUSE 5.1+ / macOS 15.4+, mountpoint under /Volumes).
  - Windows: WinFsp 2.0. No pkg-config, so discovery is by path via
    WINFSP_ROOT; the MSI needs ADDLOCAL=ALL for the Developer feature to
    build against. Mountpoint is a drive letter; unmount = stop the
    daemon. The wheel's console script prepends WinFsp's bin to PATH.

Also document the Apptainer --fusemount path (CLIO_CTE_FUSE_MOUNTPOINT is
mandatory, needs CAP_SYS_ADMIN in the userns, and is compiled OUT of the
published Linux wheel because manylinux ships libfuse 3.10.2 while the
custom-io path needs 3.14+ headers), the CI mount-smoke scripts as an
end-to-end deployment check, and a platform-keyed troubleshooting section.

The supported-operations table was stale: rename, chmod, chown, symlink,
readlink, link, statfs, and the four xattr ops are all implemented, and
truncate is a real AsyncTruncate rather than a cached-size update. Replace
the "not yet supported" list with what actually is not supported
(RENAME_EXCHANGE/WHITEOUT, fallocate punch/collapse/insert) and document
the caching semantics — kernel attr/entry caches off because there is no
invalidation upcall, page cache on because mmap depends on it.

Note in the adapter comparison that POSIX/STDIO/VFD are Linux-only (glibc
ELF/dlsym interceptor, no macOS or Windows port), so FUSE is the only
transparent-interception option on those platforms.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Verified against the published iowarp-core 2.2.0 artifacts on PyPI: the
Linux and Windows wheels contain iowarp_core/bin/clio_cte_fuse[.exe]; the
macOS wheel does not.

Two things that make the macOS absence confusing in practice:

  - The clio_cte_fuse console script is declared in EVERY wheel, macOS
    included, so the command exists on PATH there with no binary behind
    it. It prints "Error: clio_cte_fuse binary not found at <path>" and
    exits 1 — worth naming as expected, not as a broken install.
  - macOS wheels are Apple Silicon only (macosx_14_0_arm64) and there is
    no sdist, so pip has nothing to resolve on an Intel Mac.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@lukemartinlogan
lukemartinlogan merged commit 45ce779 into main Aug 12, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant