diff --git a/.env.example b/.env.example index 410bc7b..3a410a3 100644 --- a/.env.example +++ b/.env.example @@ -81,6 +81,25 @@ KALI_GPU_RENDER_NODE=/dev/dri/renderD128 KALI_GPU_RENDER_GID=990 KALI_GPU_VIDEO_GID=44 +# ============================================================================== +# HINDSIGHT (optional β€” agent memory server, claude-code LLM provider) +# ============================================================================== +# Host ports for the API and the Control Plane UI (loopback-bound by default). +HINDSIGHT_API_PORT=8888 +HINDSIGHT_UI_PORT=9999 + +# UID/GID that must match your host user (not the fleet's usual PUID/PGID): +# the container mounts your live ~/.claude credentials and needs matching +# ownership to read them, and to write the embedded pg0 data directory if it +# isn't a named volume. Check with: id -u / id -g +HOST_UID=1000 +HOST_GID=1000 + +# Claude CLI version installed at ~/.local/share/claude/versions/ on the host, +# mounted in to override the older CLI bundled in the image. Check installed +# versions with: ls ~/.local/share/claude/versions/ +CLAUDE_CLI_VERSION=2.1.277 + # ============================================================================== # LOCAL STORAGE VOLUMES # ============================================================================== @@ -93,3 +112,9 @@ MEDIA_MOVIES=./media/Movies # Host target path for qBittorrent downloads DOWNLOADS_DIR=./downloads + +# Directory holding a LINUX claude CLI at versions/, mounted +# into the container. On Windows the host CLI is a .exe and cannot be reused: +# copy the ELF from a Linux/WSL install, e.g. +# wsl cp ~/.local/share/claude/versions/2.1.277 /versions/ +CLAUDE_CLI_DIR=./claude-cli diff --git a/.gitignore b/.gitignore index f64bf69..80bcfb0 100644 --- a/.gitignore +++ b/.gitignore @@ -15,3 +15,7 @@ kali-desktop/opt/ .DS_Store Thumbs.db *.log + +# Linux claude CLI copied for the hindsight container (large binary) +hindsight/claude-cli/ +hindsight/claude.json diff --git a/.kandev/workflows/.gitkeep b/.kandev/workflows/.gitkeep deleted file mode 100644 index e69de29..0000000 diff --git a/.vscode/extensions.json b/.vscode/extensions.json deleted file mode 100644 index cb99a89..0000000 --- a/.vscode/extensions.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "recommendations": [ - "ms-azuretools.vscode-docker" - ] -} \ No newline at end of file diff --git a/AGENTS.md b/AGENTS.md index 5322146..2cab787 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,6 +1,6 @@ -# AI Coding Agent Playbook: Resilient Home-Lab Compose Stack πŸ€– +# Home-Lab Compose Stack Playbook: Modular Architecture & Hardening Standards πŸ€– -Welcome, AI Agent! This document provides the architectural blueprint, structural patterns, and strict resiliency rules governing this repository. Read this playbook thoroughly before modifying or adding services to maintain our production-grade home-lab environment. +Welcome! This document provides the architectural blueprint, structural patterns, and strict resiliency rules governing this repository. Read this playbook thoroughly before modifying or adding services to maintain our production-grade home-lab environment. --- @@ -9,7 +9,7 @@ Welcome, AI Agent! This document provides the architectural blueprint, structura This stack is organized as a **highly modular, decentralized multi-container deployment** utilizing the modern Docker Compose `include` directive. Instead of a single monolithic compose file, each service is fully self-contained within its own directory. ``` -/home/jond/.docker/ +/ # C:/AIonUI/Work/docker on the current machine β”œβ”€β”€ docker-compose.yml # Root orchestrator (includes active services) β”œβ”€β”€ .env.example # Global environment variables blueprint β”œβ”€β”€ .gitignore # Excludes sensitive data (.env, .env.local, config folders) @@ -34,7 +34,7 @@ This stack is organized as a **highly modular, decentralized multi-container dep └── docker-compose.yml # Headless Chromium automation engine ``` -The root `/home/jond/.docker/docker-compose.yml` serves as the entry point, selectively importing active service configurations: +The root `docker-compose.yml` serves as the entry point, selectively importing active service configurations: ```yaml include: # - path: ./changedetection/docker-compose.yml @@ -169,37 +169,65 @@ Create a dedicated subdirectory for the service (e.g., `./my-service`). ### Step 2: Write the `docker-compose.yml` Create `./my-service/docker-compose.yml`. Make sure to: -- Use standard images (prefer verified, official, or reputable publishers like LinuxServer). +- Use standard images (prefer verified, official, or reputable publishers like LinuxServer, Docker Official, or trusted open-source maintainers). - Define ports using loopback `127.0.0.1` and customizable env variables (e.g., `${MY_SERVICE_PORT:-9000}`). - Set up the **4-layered env_file block** pointing to `../.env`, `../.env.local`, `.env`, and `.env.local`. -- Apply **Resource limits**, **Log rotation**, **Healthchecks**, and **init: true** (if running subprocesses). +- Apply **Resource limits**, **Log rotation**, **Healthchecks**, **Capability hardening**, and **init: true** (if running subprocesses). - Use local folder mappings for config storage (e.g., `./config:/config`). +- Never hardcode secrets; all sensitive values must come from environment variables. ### Step 3: Update Environments -- Add the default configuration key value pairs to `/home/jond/.docker/.env.example`. -- If the user has a local `.env`, append the new service variables to it with helpful comments. +- Add default configuration key-value pairs to the root `.env.example` with clear comments. +- Document all new environment variables with their purpose, defaults, and any required values. +- Keep all host-specific paths (storage, credentials) parameterized in `.env` β€” never hardcode them. ### Step 4: Register in Orchestrator -Add the service directory relative path to the root `/home/jond/.docker/docker-compose.yml` under `include:`. +Add the service directory relative path to the root `docker-compose.yml` under `include:`. Comment it out if it should be optional. --- ## 🚦 Verification Command Playbook -Before marking any task as complete, run these commands to verify syntax, config values, and deployment validity: +Before marking any service deployment as complete, run these commands to verify syntax, configuration, and runtime health: ```bash # 1. Verify compose syntax and structural formatting docker compose config -# 2. Start the services (dry-run/daemon mode) +# 2. Start the services in daemon mode docker compose up -d -# 3. Check health and lifecycle status +# 3. Check health and lifecycle status of all containers docker compose ps -# 4. View logs of specific containers to ensure error-free startups +# 4. View logs to ensure clean startups (follow mode for real-time) docker compose logs -f + +# 5. Inspect resource usage and container stats +docker stats + +# 6. Verify healthcheck status +docker ps --format "table {{.Names}}\t{{.Status}}" ``` -Happy hacking, AI Agent! Keep the stack secure, resilient, and blazing fast. πŸš€ +--- + +## ⚠️ Common Pitfalls + +**Hardening mistakes:** +- `cap_drop: ALL` without `cap_add` for rootβ†’user privilege-drop services breaks entrypoints. +- Forgetting `init: true` on containers with subprocess spawning (Node, browser engines, scrapers) leads to zombie processes. +- Omitting `security_opt: [no-new-privileges:true]` allows containers to re-escalate if a process is exploited. + +**Resource & stability issues:** +- Missing memory limits cause OOM kills and crash-loops; always set `deploy.resources.limits.memory`. +- Unbounded log files fill the host disk; always use `logging.options.max-size` and `max-file`. +- No healthchecks hide crashing services; `docker compose ps` alone won't catch them without `-a`. +- Unbuffered volumes (bind mounts to non-existent host paths) are created as root, causing permission errors in rootless containers. + +**Environment & secrets:** +- Hardcoding secrets in compose files exposes them in version control; use env variables + `.gitignore`. +- Not documenting env vars in `.env.example` makes onboarding impossible. +- Mixing host-specific paths in compose creates non-portable configs; parameterize them. + +Keep the stack secure, resilient, and maintainable. πŸš€ diff --git a/docker-compose.yml b/docker-compose.yml index bedae85..06b9e38 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -4,6 +4,8 @@ include: # - path: ./browser/docker-compose.yml # - path: ./metamcp/docker-compose.yml # - path: ./kali-desktop/docker-compose.yml - - path: ./searxng/docker-compose.yml - - path: ./jellyfin/docker-compose.yml - - path: ./qbittorrent/docker-compose.yml + # - path: ./hindsight/docker-compose.yml +# - path: ./searxng/docker-compose.yml +# - path: ./jellyfin/docker-compose.yml +# - path: ./qbittorrent/docker-compose.yml + - path: ./hindsight/docker-compose.yml diff --git a/hindsight/MODELS.md b/hindsight/MODELS.md new file mode 100644 index 0000000..50455ea --- /dev/null +++ b/hindsight/MODELS.md @@ -0,0 +1,41 @@ +## Divid to Conquere +### reflect and consolidation require the most reasoning. +They must resolve contradictions, deduplicate overlapping facts, and synthesize complex answers while following your bank's mission and directives. +- Use DeepSeek V4 Pro for these. + +### retain requires minimal reasoning. +It simply parses raw text into structured facts. +- Use DeepSeek V4 Flash for this to keep ingestion fast and cheap. + +Set DeepSeek V4 Flash as your global default. Then override the model specifically for the heavier operations using these environment variables. + +### DeepSeek Version +```yaml +HINDSIGHT_API_LLM_PROVIDER: deepseek +HINDSIGHT_API_LLM_MODEL: deepseek-v4-flash +HINDSIGHT_API_REFLECT_LLM_MODEL: deepseek-v4-pro +HINDSIGHT_API_CONSOLIDATION_LLM_MODEL: deepseek-v4-pro +``` + +### Claude Code Version +```yaml +HINDSIGHT_API_LLM_PROVIDER: claude-code +HINDSIGHT_API_LLM_MODEL: claude-haiku-4-5 +HINDSIGHT_API_RETAIN_LLM_MODEL: claude-haiku-4-5 +HINDSIGHT_API_REFLECT_LLM_MODEL: claude-sonnet-5 +HINDSIGHT_API_CONSOLIDATION_LLM_MODEL: claude-sonnet-5 +``` + +### Mix Version +```yaml +# Global & Retain (Uses your Claude subscription) +HINDSIGHT_API_LLM_PROVIDER: claude-code +HINDSIGHT_API_LLM_MODEL: claude-haiku-4-5 +HINDSIGHT_API_RETAIN_LLM_PROVIDER: claude-code +HINDSIGHT_API_RETAIN_LLM_MODEL: claude-haiku-4-5 + +# Reflect & Mental Model Refresh (Routes to a cheaper API) +HINDSIGHT_API_REFLECT_LLM_PROVIDER: deepseek +HINDSIGHT_API_REFLECT_LLM_API_KEY: sk-your-deepseek-api-key +HINDSIGHT_API_REFLECT_LLM_MODEL: deepseek-v4-flash +``` diff --git a/hindsight/README.md b/hindsight/README.md new file mode 100644 index 0000000..669af0e --- /dev/null +++ b/hindsight/README.md @@ -0,0 +1,113 @@ +## Hindsight with Claude Code (Claude Pro/Max subscription) + +Run Hindsight inside Docker using the `claude-code` LLM provider, backed by +your host machine's Claude Pro or Max subscription credentials. + +The standalone Hindsight Docker image ships `claude-agent-sdk` but does **not** +bundle the host `claude` CLI binary or any Claude credentials. This Compose +file bind-mounts the host's CLI install and credentials into the container so +the `claude-code` provider works without an API key. + +## When to use this + +- You have an active Claude Pro or Max subscription and want to use it for +Hindsight without paying separate Anthropic API costs. +- You want a one-command `docker compose up` instead of a long `docker run` +invocation with many flags. +- You are running on **Linux/amd64** β€” macOS Docker Desktop and Windows host +paths differ and are not yet covered (please open an issue if you'd like to +contribute a verified recipe for either). + +> **Personal-use only.** Anthropic's +Agent SDK documentation +states that third-party developers should not offer claude.ai login or rate +limits for their products. Hindsight does **not** perform any login on your +behalf β€” it uses credentials you've already authenticated via +`claude auth login`. In January 2026, Anthropic +enforced restrictions +against tools that spoofed the Claude Code client identity; Hindsight uses +the official Claude Agent SDK instead. +> +> +> Do not deploy this configuration to shared environments or production. For +> that, use the `anthropic` provider with an API key from the +> Anthropic Console. Usage counts against +> your Claude Pro/Max subscription limits. +> + +## Prerequisites + +- Host has `claude` CLI installed (e.g., `npm install -g @anthropics/claude-code`) +and `claude auth login` has been run successfully. +- `~/.claude.json` and `~/.claude/.credentials.json` exist on the host. +- Host `claude` CLI version is **2.1.128 or newer** β€” the version bundled with +`claude-agent-sdk` 0.5.x has a protocol incompatibility in containers, so +the recipe overrides it with the host binary. + +## Quick start + +``` +# Set your host UID/GID (defaults to 1000:1000 if unset) +export HOST_UID=$(id -u) +export HOST_GID=$(id -g) +docker compose -f docker/docker-compose/claude-code/docker-compose.yaml up -d +``` + +- API: http://localhost:8888 +- Control Plane: http://localhost:9999 + +## Post-setup (one-time) + +After the container starts for the first time, run these commands to fix +permissions and symlink the host `claude` binary into `$PATH`: + +``` +# Make ~/.claude writable by your UID (the CLI writes session/project state) +docker exec --user 0:0 hindsight-claude-code chown $(id -u):$(id -g) /home/hindsight/.claude +docker exec --user 0:0 hindsight-claude-code chmod 755 /home/hindsight/.claude +# Symlink the host claude binary into PATH +docker exec --user 0:0 hindsight-claude-code \ + ln -sf /home/hindsight/.local/share/claude/versions/2.1.128 /usr/local/bin/claude +``` + +If you set `CLAUDE_CLI_VERSION` to a version other than `2.1.128`, update the +symlink path accordingly. + +## Notes on the bind-mount surface (every flag is load-bearing) + +- **Host `claude` binary required** β€” the image ships only `claude-agent-sdk`, +not the CLI itself. +- **SDK bundled-binary override** β€” the override of +`claude_agent_sdk/_bundled/claude` works around a protocol issue in the +bundled v2.1.121 binary inside containers. Once `claude-agent-sdk` ships +with v2.1.128+ this override can be dropped. Set `CLAUDE_CLI_VERSION` to +match your installed version. +- **Single-file credential mounts** β€” credentials are mounted as individual +`:ro` files rather than a whole-directory `:ro` mount of `~/.claude`, +because the CLI writes session/project state at runtime and a read-only +directory mount silently breaks it. +- **`-user` / `user:`** β€” the `user: ${HOST_UID}:${HOST_GID}` pattern +requires `chmod 755 /home/hindsight`, which is built into the image since +v0.6.0 (see #1481). +- **`~/.hindsight-docker` data directory** β€” the pg0 data bind mount must be +writable by your host UID (see +#1483). +- **Verified** on `linux/amd64` against `ghcr.io/vectorize-io/hindsight:latest` +v0.5.6+. + +## Using a different Claude CLI version + +If your host has a `claude` version other than 2.1.128, set +`CLAUDE_CLI_VERSION` before starting: + +``` +export CLAUDE_CLI_VERSION=2.2.0 +docker compose -f docker/docker-compose/claude-code/docker-compose.yaml up -d +``` + +Then update the post-setup symlink to match: + +``` +docker exec --user 0:0 hindsight-claude-code \ + ln -sf /home/hindsight/.local/share/claude/versions/2.2.0 /usr/local/bin/claude +``` \ No newline at end of file diff --git a/hindsight/TEMPLATES.md b/hindsight/TEMPLATES.md new file mode 100644 index 0000000..a436ba5 --- /dev/null +++ b/hindsight/TEMPLATES.md @@ -0,0 +1,22 @@ +## Research Assistant β€” deploy that one, and only that one. +Why it, over the alternatives. Its own reflect_mission reads: +- "monitors sources, writes briefings and digests, and maintains a second brain." + +### That's your notes/ideas/blog bank verbatim. +It ships three mental models +1. Interests & Focus +2. Knowledge Base +3. Sources & Reliability +- plus a "learn from what's ignored" directive that uses what you skip to filter future curation. +- Dispositions are skepticism 4 / empathy 2: a thinking partner that pushes back, not a warm helper. + +## Personal Assistant +is the near-miss β€” but it's routines, schedule, people, commitments. That's task management. + +## For vibecoding +don't deploy a template at all. The Coding Agent template is for a bank you drive yourself. The coding-agents plugin does this automatically: one bank per repo (coding-agent::{gitProject}), and with manageBankConfig: true it seeds the missions, the five retain strategies and the knowledge label group itself. Hand-importing on top just duplicates that. + +## Self-Hosted +The coding side is a separate install: `npx @vectorize-io/hindsight-coding-agents install claude-code --server self-hosted --api-url http://localhost:8888` + +The `--server` flag matters. Default is Hindsight Cloud, a bare install prompts for it, and re-running never re-asks β€” so a wrong answer there quietly ships your prompts off-machine and sticks. diff --git a/hindsight/docker-compose.yml b/hindsight/docker-compose.yml new file mode 100644 index 0000000..5ab0d0d --- /dev/null +++ b/hindsight/docker-compose.yml @@ -0,0 +1,136 @@ +# name: hindsight +# url: https://github.com/vectorize-io/hindsight/blob/main/docker/docker-compose/claude-code/docker-compose.yaml +# Run Hindsight with the claude-code LLM provider, using your host machine's +# Claude Pro/Max subscription credentials. Linux/amd64 only for now. +# +# docker compose -f docker/docker-compose/claude-code/docker-compose.yaml up -d + +services: + hindsight: + image: ghcr.io/vectorize-io/hindsight:latest + container_name: hindsight + restart: unless-stopped + user: "${HOST_UID:-1000}:${HOST_GID:-1000}" + # start-all.sh's SIGTERM trap gives pg0 up to 30s to checkpoint and flush + # WAL before force-killing. Docker's default 10s grace period would SIGKILL + # the container mid-flush, risking embedded-database corruption. + stop_grace_period: 30s + # start-all.sh backgrounds hindsight-api (which spawns the embedded pg0 + # postgres subprocess) and the control-plane node process, tracking only + # their top-level PIDs. tini as real PID 1 reaps any orphaned grandchildren + # those two trees leave behind (docs/HARD-WON-GOTCHAS.md, "subprocess" rule). + init: true + # Runs as a fixed non-root user from the start (no root->UID privilege-drop + # dance in the entrypoint), so the full drop-and-restore set isn't needed β€” + # except CHOWN: Docker auto-creates /home/hindsight/.claude as root:root + # (only the .credentials.json file inside it is bind-mounted, not the + # directory), and the README's documented one-time post-setup step needs + # `docker exec --user 0:0 ... chown` to hand it to HOST_UID so the CLI can + # write session/project state into it. Confirmed live: chown fails with + # "Operation not permitted" under bare cap_drop: ALL, even as UID 0. + cap_drop: + - ALL + cap_add: + - CHOWN + security_opt: + - no-new-privileges:true + ports: + - "${HINDSIGHT_API_PORT:-8888}:8888" + - "${HINDSIGHT_UI_PORT:-9999}:9999" +# https://hindsight.vectorize.io/developer/configuration#control-plane + env_file: + - path: ../.env + required: false + - path: ../.env.local + required: false + - path: .env + required: false + - path: .env.local + required: false + environment: + HOME: /home/hindsight + USER: hindsight + LOGNAME: hindsight + # The Dockerfile's own ENV puts the venv first (PATH="/app/api/.venv/bin:...") + # so plain `python3` resolves to the venv where hindsight_api is installed. + # Upstream's claude-code example compose file reverses that order, which + # puts the base image's system python3 (no hindsight_api) ahead of it β€” + # start-all.sh's own require_http_probe_runtime() check then fails with + # "python3 with hindsight_api.http_probe importable" before the API ever + # starts. Restoring venv-first fixes it. + PATH: /app/api/.venv/bin:/usr/local/bin:/usr/bin:/bin + HINDSIGHT_API_LLM_PROVIDER: claude-code + HINDSIGHT_API_LLM_MODEL: claude-haiku-4-5 + HINDSIGHT_API_RETAIN_LLM_MODEL: claude-haiku-4-5 + HINDSIGHT_API_REFLECT_LLM_MODEL: claude-sonnet-5 + HINDSIGHT_API_CONSOLIDATION_LLM_MODEL: claude-sonnet-5 + HINDSIGHT_API_AUDIT_LOG_ENABLED: true + # The image ships no curl/wget (docker/standalone/Dockerfile installs + # neither in the runtime stage). Reuse the app's own stdlib-only prober + # instead of guessing β€” it's what start-all.sh uses for its own readiness + # wait. /health/live is the DB-independent liveness route (hindsight's + # own tests pin this split so a slow Postgres doesn't flip the container + # unhealthy); /api/health confirms the control-plane's Next server answers. + healthcheck: + test: + - CMD-SHELL + - >- + /app/api/.venv/bin/python3 -m hindsight_api.http_probe http://127.0.0.1:8888/health/live 5 && + /app/api/.venv/bin/python3 -m hindsight_api.http_probe http://127.0.0.1:9999/api/health 5 + interval: 10s + timeout: 10s + retries: 3 + start_period: 60s + deploy: + resources: + limits: + cpus: '2.0' + memory: 3072M + logging: + driver: "json-file" + options: + max-size: "10m" + max-file: "3" + volumes: + # ── Persistent data ──────────────────────────────────────────── + # Named volume, not a host bind mount: Docker seeds a fresh named + # volume from the image's /home/hindsight/.pg0 (created at build time + # as the hindsight user, UID 1000), so ownership is correct on first + # use. A bind mount to a not-yet-existing host path is created by the + # Linux dockerd as root instead, which the rootless container then + # can't write to β€” hit exactly that (issue #1483) with a bind mount + # here before switching to this named volume. + - hindsight-pg0-data:/home/hindsight/.pg0 + + # ── Claude credentials ────────────────────────────────────────── + # A single-file :ro bind of .credentials.json freezes at whatever + # inode existed at container start: OAuth refresh writes via a + # temp-file-then-rename, which swaps the host's directory entry to + # a NEW inode the bind mount never follows. The container then + # keeps presenting an old access token that Anthropic has since + # revoked (confirmed live: container's mtime was ~7.5h stale vs + # host's, causing "Mental Model Refresh" jobs to fail with + # "OAuth access token has been revoked"). Bind-mounting the + # containing directory instead means lookups resolve by name + # against the live host directory, so renames propagate. + # Read-write also lets the container persist its own token + # refreshes and the session/project state the CLI writes at + # runtime β€” the reason a prior whole-directory attempt was :ro + # (and broke on that) doesn't apply once it's writable. + - ${HOME:-${USERPROFILE}}/.claude:/home/hindsight/.claude + # Copy of the host ~/.claude.json: Docker Desktop shares folders only, + # not single files. Refresh it after re-login on the host. + - ./claude.json:/home/hindsight/.claude.json:ro + + # ── Claude CLI install (read-only) ───────────────────────────── + - ${CLAUDE_CLI_DIR:-./claude-cli}:/home/hindsight/.local/share/claude:ro + + # ── SDK bundled-binary override ──────────────────────────────── + # The claude-agent-sdk 0.5.x image bundles v2.1.121 which has a + # protocol incompatibility in containers. Override it with the + # host's v2.1.128+ binary. Drop this mount once claude-agent-sdk + # ships with v2.1.128+. +# - ${HOME}/.local/share/claude/versions/${CLAUDE_CLI_VERSION:-2.1.128}:/app/api/.venv/lib/python3.11/site-packages/claude_agent_sdk/_bundled/claude:ro + +volumes: + hindsight-pg0-data: \ No newline at end of file