diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index fd46735..ffe16c5 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -127,8 +127,12 @@ jobs: env: DAKERA_ROOT_API_KEY: test-integration-key DAKERA_AUTH_ENABLED: 'false' + # The v0.12.3+ images have no curl; `dakera healthcheck` probes the + # REST port instead. This check works on both, so moving the image to + # 0.12.3 once it is released is a one-line change. The v0.12.3 tests + # in tests/integration.rs skip themselves on an older server. options: >- - --health-cmd "curl -sf http://localhost:3000/health || exit 1" + --health-cmd "if command -v curl >/dev/null 2>&1; then curl -sf http://localhost:3000/health; else dakera healthcheck; fi" --health-interval 5s --health-timeout 3s --health-retries 10 diff --git a/CHANGELOG.md b/CHANGELOG.md index 0f0536b..5e3e4cb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,85 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [Unreleased] + +## [0.8.2] - 2026-10-22 + +Support for Dakera server **v0.12.3**. Follows the engine's recall API correctness and security +hardening work (Dakera-AI/dakera#917, #920, engine PR #922; merge this after it, as its merge tags +v0.12.3). Compatible with v0.11.108 and v0.12.0 – v0.12.2 servers as before; `--neighborhood`, +`--include-summaries`, the Context section's `context_of`, the unknown-field hint and `dk document` need v0.12.3. + +### Breaking + +- **`dk --format json|compact memory recall` prints one object, + `{"memories": [...], "context_memories": [...]}`, instead of an array of result rows** (approved + by the owner). Read the ranked results from `.memories` (for example + `dk -f json memory recall a q | jq '.memories[].id'`); the neighbourhood context is in + `.context_memories`. The old output could not be parsed as it was anyway: an `ℹ Found N memories` + line came before the JSON on stdout. + +### Fixed + +- **`dk memory recall` shows at most `--top-k` ranked results.** The server appended + session-neighbourhood context after the ranked results, so `--top-k 5` could print 7 rows. The + command now asks for `neighborhood_mode: "separate"` and prints the context in its own + **Context** section after the ranked table, each row saying which result it belongs to + (`#2 `). A server that predates `neighborhood_mode` (v0.12.2 and older) appends its context + unmarked: the rows past `--top-k` are left out with a warning (under `--neighborhood inline` they + are shown as context). +- **`dk memory recall` lists each result with the server's `rank_score`** (v0.12.3), the score + that follows the order of the results. `score` and `smart_score` can rise down the list, because + later ranking stages reorder the results after they are scored. Older servers send no + `rank_score`; their `smart_score` is listed as before. +- **A `429` answer exits 6 (server busy), like a `503`, with the `Retry-After` hint.** Since v0.12.3 + an overloaded server (a full inference or recall queue) answers `429` + `Retry-After` where it + answered `503`; `dk` exited 1 on it. +- `dk --format json|compact memory recall` printed `ℹ Found N memories` before the JSON on + stdout; stdout is now one JSON document and notes go to stderr. +- `dk keys get|delete|deactivate|rotate|usage` and `dk admin optimize|backup-delete` percent-encode + the id in the path; `dk keys *` errors carry the usual exit code and hint instead of the raw body. +- `dk keys create --format json` prints the server's answer as JSON. +- README: the config file example showed an `api_key` and `--key` that `dk` never read. + +### Added + +- **`dk document`** (Dakera v0.12.3, Dakera-AI/dakera#923): `import ` turns a text, + Markdown, HTML, PDF or DOCX file into chunk memories of `--agent` (text formats through + `POST /v1/memory/documents`; PDF / DOCX, and text that is not UTF-8, uploaded as an attachment to + the agent's namespace and imported through `POST /v1/namespaces/{ns}/attachments/{ref}/document`), + with `--title`, `--document-id`, `--media-type` (else from the extension), `--tag`, `--session-id`, + `--type`, `--importance`, `--lang`, `--chunk-tokens`, `--overlap`, `--strategy structure|fixed`, + `--decompose`, `--wait` / `--timeout`; `job `, `list` (`--limit`, `--cursor`), `show ` + (the chunks with pages and heading paths) and `delete `, in table and JSON. On a server + before v0.12.3 the commands say it is needed. Shell completions include them. +- `dk memory recall` cites document chunks: a `SOURCE` column (`Field Manual, p. 39-40 › Install`) + and `document` (`document_id`, `title`, `page_start` / `page_end`, `heading_path`, + `chunk_index`) in JSON, on ranked results and context. +- `dk memory recall --neighborhood ` (default `separate`; `off` also sends + `neighborhood: false`, which older servers honour) and `--include-summaries[=false]` (sent only + when given). +- A `MATCHED_TEXT` column when the server collapsed derived sentence sub-memories into a result + (`DAKERA_RECALL_COLLAPSE_DERIVED`). +- `--format json` for `memory recall` carries `matched_text` / `matched_ids` on ranked results and + `context_of` / `context_of_rank` on context (see **Breaking** for the new shape). +- A hint for the `422` (or `400`) "unknown field" refusal that the admin key, backup and + rotate-key routes answer from Dakera v0.12.3 on, naming the field. + +### Security + +- `~/.dakera/config.toml` is written `0600` (its directory `0700` when created), an existing file is + narrowed to `0600` on write, and a file readable by other users gets a warning on stderr. +- `dk config` shows `DAKERA_API_KEY` masked (first 3 and last 4 characters) and the transport (TLS + or plain http); a password in the server URL is masked. +- A warning on stderr when `DAKERA_API_KEY` is set and the server URL is plain `http://` to a host + other than the loopback interface. +- Raw requests time out: 10 s to connect, 330 s between reads (longer than the 300 s + `encryption-rotate --wait-secs` may ask the server to wait). +- Dependencies: rustls 0.23.46 (RUSTSEC-2026-0346, -0347, -0348, -0349), quinn-proto 0.11.19 + (RUSTSEC-2026-0338 to -0342), chacha20 0.10.2 (0.10.1 was yanked), rustls-webpki 0.103.16 + (RUSTSEC-2026-0382) and webpki-root-certs 1.0.10 (RUSTSEC-2026-0383). + ## [0.8.1] - 2026-10-08 Support for Dakera server **v0.12.2** (Dakera-AI/dakera#916). Compatible with v0.11.108, diff --git a/Cargo.lock b/Cargo.lock index fdec38a..e316307 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -194,9 +194,9 @@ checksum = "f079e83a288787bcd14a6aea84cee5c87a67c5a3e660c30f557a3d24761b3527" [[package]] name = "chacha20" -version = "0.10.1" +version = "0.10.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d524456ba66e72eb8b115ff89e01e497f8e6d11d78b70b1aa13c0fbd97540a81" +checksum = "65c35e4b699c7e15ccbe7ee35c005e4fc0a278d22238a2857e6ce2dadeda1b06" dependencies = [ "cfg-if", "cpufeatures 0.3.0", @@ -345,7 +345,7 @@ dependencies = [ [[package]] name = "dakera-cli" -version = "0.8.1" +version = "0.8.2" dependencies = [ "anyhow", "assert_cmd", @@ -1283,9 +1283,9 @@ dependencies = [ [[package]] name = "quinn-proto" -version = "0.11.16" +version = "0.11.19" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2f4bfc015262b9df63c8845072ce59068853ff5872180c2ce2f13038b970e560" +checksum = "0e750cca55fe4f0439a15d0bb529da9651e79993e8e72c61a899a36d462befbe" dependencies = [ "aws-lc-rs", "bytes", @@ -1485,9 +1485,9 @@ dependencies = [ [[package]] name = "rustls" -version = "0.23.45" +version = "0.23.46" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0d41d731c7d2f962d1ccc364cec258de3c0e93b38c2fb3ba97ac74513048d634" +checksum = "48e13bd8c0e9365c43cfa5c9e8f9ad49d3c8444926c9aac819e0e4dc503c8fdf" dependencies = [ "aws-lc-rs", "once_cell", @@ -1548,9 +1548,9 @@ checksum = "f87165f0995f63a9fbeea62b64d10b4d9d8e78ec6d7d51fb2125fda7bb36788f" [[package]] name = "rustls-webpki" -version = "0.103.15" +version = "0.103.16" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f3c3cf1d8b1e7d4927e2d154c3fcb02979afb9939629c62cd9048d4f07b60ac2" +checksum = "4443b1b6d39d7c663139942cc4558262562d2c84123bb6282ef177009ca8b281" dependencies = [ "aws-lc-rs", "ring", @@ -2303,9 +2303,9 @@ dependencies = [ [[package]] name = "webpki-root-certs" -version = "1.0.8" +version = "1.0.10" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0d46a5a140e6f7afeccd8eae97eff335163939eac8b929834875168b29b3d267" +checksum = "58ec80e73fd8baf57bf0d9615554c75348bda990d6d5bbeab964dd068319fecb" dependencies = [ "rustls-pki-types", ] diff --git a/Cargo.toml b/Cargo.toml index d262e33..96f8319 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "dakera-cli" -version = "0.8.1" +version = "0.8.2" edition = "2021" license = "MIT" description = "Command-line interface for Dakera AI Agent Memory Platform" diff --git a/README.md b/README.md index 921e737..45a8deb 100644 --- a/README.md +++ b/README.md @@ -122,21 +122,26 @@ dk text search "user preferences" --namespace default ### Config file -`dk init` creates `~/.dakera/config.toml`: +`dk init` creates `~/.dakera/config.toml` (one profile per server): ```toml -[server] -url = "http://localhost:3000" -api_key = "dk-mykey" +active_profile = "default" -[defaults] -namespace = "default" +[profiles.default] +url = "http://localhost:3000" +default_namespace = "default" ``` +The API key is never stored in it: `dk` reads it from `DAKERA_API_KEY` only. `dk` writes the file +readable by its owner only (`0600`, in a `0700` directory) and warns on stderr when it finds the +file readable by other users. `dk config` shows the key masked (`dk_…9876`), never in full, and +whether the server URL uses TLS; a key sent over plain `http://` to a host other than the loopback +interface gets a warning on stderr. TLS certificates are always verified (system trust store). + ### Named profiles ```bash -dk config profile add staging --url http://staging:3000 --key dk-staging-key +dk config profile add staging --url https://staging.example.com:3000 dk config profile use staging dk --profile staging namespace list ``` @@ -235,6 +240,31 @@ to stderr) and exits 5 if the job fails with a `400`, 6 on a `5xx`. --- +### `dk document` + +Import a document as chunk memories of an agent (Dakera v0.12.3): each chunk keeps its heading path +and page, so `dk memory recall` can cite it (a `SOURCE` column, `document` in JSON). Text, Markdown +and HTML are sent as text; PDF and DOCX (and text that is not UTF-8) are uploaded as an attachment to +the agent's namespace (`--namespace` to pick another) and imported from there, which needs +`DAKERA_ATTACHMENTS` on the server. The format comes from the extension (`.txt`, `.md`, `.html`, +`.pdf`, `.docx`; `-` reads stdin as text) unless `--media-type` says otherwise. + +```bash +dk document import guide.md --agent docs-bot --wait +dk document import manual.pdf --agent docs-bot --document-id manual-v3 --tag manual --wait +dk document import notes.txt --agent docs-bot --chunk-tokens 300 --overlap 30 --strategy fixed +cat page.html | dk document import - --agent docs-bot --media-type text/html --title "Release notes" +dk document job job_17_3 --agent docs-bot # Pending / Running (progress) / Completed / Failed +dk document list --agent docs-bot --limit 20 # then --cursor +dk document show manual-v3 --agent docs-bot # the document and its chunks (pages, headings) +dk document delete manual-v3 --agent docs-bot # every chunk memory of it +``` + +Re-importing the same `--document-id` replaces the document (unchanged chunks are kept as they are). +Other flags: `--title`, `--session-id`, `--type`, `--importance`, `--lang`, `--decompose` (also +derive sentence sub-memories per chunk), `--timeout` (with `--wait`, default 600 s). A server before +v0.12.3 has no document routes: `dk` says so. + ### `dk namespace` Manage namespaces. @@ -255,8 +285,14 @@ Store, recall, search, and manage agent memories. This is the primary interface # Store a memory dk memory store my-agent "The user likes dark mode" --importance 0.8 --type semantic -# Recall by semantic query +# Recall by semantic query: at most --top-k ranked results, then a separate "Context" +# section with the session neighbours of those results (which result each belongs to). +# A MATCHED_TEXT column appears when the server collapsed sentence sub-memories into a result. dk memory recall my-agent "UI preferences" --top-k 10 +# --neighborhood separate (default) | inline | off; --include-summaries[=false] +dk memory recall my-agent "UI preferences" --neighborhood off --include-summaries=false +# JSON: {"memories": [...], "context_memories": [...]} (matched_text, context_of, context_of_rank) +dk --format json memory recall my-agent "UI preferences" | jq '.memories[].id' # Full-text search within an agent's memories dk memory search my-agent "dark mode" --top-k 5 @@ -509,7 +545,23 @@ dk completion powershell ## Dakera v0.12 and compatibility -`dk` 0.8 works against Dakera **v0.11.108, v0.12.0, v0.12.1 and v0.12.2** servers +> **Breaking in dk 0.8.2:** `dk --format json|compact memory recall` prints one object, +> `{"memories": [...], "context_memories": [...]}`, instead of an array of rows: read the ranked +> results from `.memories`. + +The 0.8.2 additions need a **v0.12.3** server; older servers ignore them, as noted: + +| Command (0.8.2) | v0.12.2 and older | v0.12.3 | +|---|---|---| +| `dk memory recall` (`--neighborhood separate`, the default) | the server appends its context to the results unmarked: `dk` shows the first `--top-k` and warns how many it left out | at most `--top-k` ranked results, then the Context section with `context_of` | +| `dk memory recall --include-summaries[=false]` | ignored | yes | +| `MATCHED_TEXT` column | never shown | when the server collapses derived sentences (`DAKERA_RECALL_COLLAPSE_DERIVED`) | +| the unknown-field hint (`keys create`, `admin backup-create`, `admin encryption-rotate`) | not needed: unknown fields are ignored | a misspelled field is a `422`, with a hint naming it | +| `dk document ...` | no route: `dk` says it needs v0.12.3 | yes (`DAKERA_DOCUMENTS`, on by default; PDF / DOCX also need `DAKERA_ATTACHMENTS`) | +| `SOURCE` column / `document` in `memory recall` | never shown | for results that are document chunks | +| `dk admin` / `dk keys` commands | as before | as before; `POST /ops/shutdown` (no `dk` command) needs a `super_admin` key | + +`dk` 0.8.2 works against Dakera **v0.11.108 and v0.12.0 – v0.12.3** servers (it uses `dakera-client` 0.12). The new commands need the server release that has the route or field: @@ -591,7 +643,7 @@ See the server's `docs/v0.12/UPGRADE.md` ("Going back to v0.11.108") for what | 3 | Not found | | 4 | Permission denied / authentication failure (`401`, `403`) | | 5 | Invalid input (`400`, `409`, `413`, `415`, `422`, or a bad argument) | -| 6 | Server-side error (5xx, including `501` feature disabled and `503` busy/starting) | +| 6 | Server-side error (5xx, including `501` feature disabled and `503` busy/starting), or `429` server overloaded (retry after its `Retry-After`) | Scripts can check `$?` after each command. diff --git a/src/api.rs b/src/api.rs index a8e0b82..86f4246 100644 --- a/src/api.rs +++ b/src/api.rs @@ -85,6 +85,17 @@ impl ApiHttpError { None => HINT_BUSY.to_string(), }); } + // Since v0.12.3 the admin key, backup and rotate-key requests refuse a + // field they do not know instead of ignoring it. + if self.status == 422 || self.status == 400 { + if let Some(field) = unknown_field(&self.message) { + return Some(format!( + "the server does not accept the field `{field}` on this route (admin key, \ + backup and encryption requests refuse unknown fields since v0.12.3); this \ + dk may be newer than the server, or the field is misspelled" + )); + } + } let pinned = self.details_contain("namespace: *"); let super_admin = self.details_contain("required: super_admin"); let hint = match (self.status, code) { @@ -104,6 +115,13 @@ impl ApiHttpError { } } +/// The field serde names in an "unknown field `x`, expected ..." refusal. +pub fn unknown_field(message: &str) -> Option<&str> { + let start = message.find("unknown field `")? + "unknown field `".len(); + let len = message[start..].find('`')?; + Some(&message[start..start + len]) +} + const HINT_AUTH: &str = "authentication failed: set DAKERA_API_KEY to a valid, unexpired API key"; const HINT_BUSY: &str = "the server is busy or starting; retry shortly"; const HINT_PINNED: &str = "this API key is pinned to namespaces; since v0.12 such a key is \ @@ -467,6 +485,18 @@ mod tests { assert_eq!(e.message, "404 Not Found"); } + #[test] + fn unknown_field_refusal_names_the_field() { + let error = "Failed to deserialize the JSON body into the target type: unknown field \ + `encrypted`, expected one of `name`, `backup_type`, `encrypt`"; + let e = answer(422, "INVALID_REQUEST", error, None); + let hint = e.hint().unwrap(); + assert!(hint.contains("`encrypted`"), "{hint}"); + assert!(e.to_string().contains("hint:")); + let e = answer(400, "INVALID_REQUEST", "missing field `name`", None); + assert!(e.hint().is_none()); + } + #[test] fn segment_encodes_reserved_characters() { assert_eq!(segment("sha256:ab"), "sha256:ab"); diff --git a/src/cli.rs b/src/cli.rs index 1d8d39b..b05f6bd 100644 --- a/src/cli.rs +++ b/src/cli.rs @@ -51,6 +51,7 @@ pub fn build_cli() -> Command { .subcommand(build_capabilities_command()) .subcommand(build_whoami_command()) .subcommand(build_attachment_command()) + .subcommand(build_document_command()) .subcommand(build_namespace_command()) .subcommand(build_index_command()) .subcommand(build_memory_command()) @@ -225,6 +226,148 @@ fn build_attachment_job_command() -> Command { .arg(timeout) } +/// `dk document`: document import (Dakera v0.12.3). +pub fn build_document_command() -> Command { + let long_about = "Documents imported as chunk memories of an agent (Dakera v0.12.3): text, \ + Markdown and HTML are sent as text; PDF and DOCX are uploaded as an attachment to the \ + agent's namespace (needs DAKERA_ATTACHMENTS on the server) and imported from there. \ + Each chunk keeps its heading path and page, so `dk memory recall` can cite it."; + let agent = Arg::new("agent") + .short('a') + .long("agent") + .required(true) + .help("Agent ID"); + let import = Command::new("import") + .about("Import a file (or - for stdin) as chunk memories; runs as a background job") + .arg( + Arg::new("file") + .required(true) + .help("File to import (.txt, .md, .html, .pdf, .docx), or - for stdin"), + ) + .arg(agent.clone()) + .arg( + Arg::new("title") + .long("title") + .help("Document title (default: from the document)"), + ) + .arg(Arg::new("document-id").long("document-id").help( + "Document id (default: derived from the content; the same id replaces the document)", + )) + .arg( + Arg::new("media-type") + .long("media-type") + .help("Media type (default: from the extension; stdin: text/plain)"), + ) + .arg( + Arg::new("tag") + .long("tag") + .action(ArgAction::Append) + .help("Tag every chunk (repeatable)"), + ) + .arg( + Arg::new("session-id") + .long("session-id") + .help("Session of the chunks"), + ) + .arg( + Arg::new("type") + .short('t') + .long("type") + .value_parser(["episodic", "semantic", "procedural", "working"]) + .help("Memory type of the chunks"), + ) + .arg( + Arg::new("importance") + .long("importance") + .value_parser(value_parser!(f32)) + .help("Importance of the chunks (0.0-1.0)"), + ) + .arg( + Arg::new("lang") + .long("lang") + .help("Language of the document"), + ) + .arg( + Arg::new("chunk-tokens") + .long("chunk-tokens") + .value_parser(value_parser!(u64)) + .help("Chunk budget in tokens (server default 400)"), + ) + .arg( + Arg::new("overlap") + .long("overlap") + .value_parser(value_parser!(u64)) + .help("Tokens repeated from the previous chunk (server default 50)"), + ) + .arg( + Arg::new("strategy") + .long("strategy") + .value_parser(["structure", "fixed"]) + .help("structure (sections at headings, the default) or fixed (token windows)"), + ) + .arg( + Arg::new("decompose") + .long("decompose") + .action(ArgAction::SetTrue) + .help("Also derive sentence sub-memories from every chunk"), + ) + .arg( + Arg::new("namespace") + .short('n') + .long("namespace") + .help("Namespace to upload a PDF/DOCX to (default: _dakera_agent_)"), + ) + .arg( + Arg::new("wait") + .long("wait") + .action(ArgAction::SetTrue) + .help("Wait for the import job to finish"), + ) + .arg( + Arg::new("timeout") + .long("timeout") + .value_parser(value_parser!(u64)) + .default_value("600") + .help("With --wait: give up after this many seconds"), + ); + let job = Command::new("job") + .about("Status of an import job") + .arg(Arg::new("job_id").required(true).help("Job ID")) + .arg(agent.clone()); + let list = Command::new("list") + .about("List an agent's documents") + .arg(agent.clone()) + .arg( + Arg::new("limit") + .long("limit") + .value_parser(value_parser!(u64).range(1..=500)) + .help("Documents per page (1-500, server default 50)"), + ) + .arg( + Arg::new("cursor") + .long("cursor") + .help("The next_cursor of the previous page"), + ); + let show = Command::new("show") + .about("Show a document and its chunks") + .arg(Arg::new("document_id").required(true).help("Document ID")) + .arg(agent.clone()); + let delete = Command::new("delete") + .about("Delete a document (every chunk memory of it)") + .arg(Arg::new("document_id").required(true).help("Document ID")) + .arg(agent); + Command::new("document") + .visible_alias("documents") + .about("Documents: import PDF, DOCX, HTML, Markdown or text as chunk memories (v0.12.3)") + .long_about(long_about) + .subcommand_required(true) + .subcommand(import) + .subcommand(job) + .subcommand(list) + .subcommand(show) + .subcommand(delete) +} + pub fn build_attachment_command() -> Command { let long_about = "Files that agent memories can point at (Dakera v0.12). Off by default on \ the server: the routes answer 501 FEATURE_DISABLED until DAKERA_ATTACHMENTS is set \ @@ -551,7 +694,7 @@ pub fn build_memory_command() -> Command { .long("top-k") .default_value("5") .value_parser(value_parser!(usize)) - .help("Number of results to return"), + .help("Most ranked results to return"), ) .arg( Arg::new("type") @@ -560,7 +703,31 @@ pub fn build_memory_command() -> Command { .value_parser(["episodic", "semantic", "procedural", "working"]) .help("Filter by memory type"), ) - .arg(memory_lang_arg("Language of the query")), + .arg(memory_lang_arg("Language of the query")) + .arg( + Arg::new("neighborhood") + .long("neighborhood") + .value_name("MODE") + .default_value("separate") + .value_parser(["separate", "inline", "off"]) + .help( + "Session-neighbour context of the results: separate (listed in its \ + own Context section, not counted in --top-k), inline or off", + ), + ) + .arg( + Arg::new("include-summaries") + .long("include-summaries") + .value_name("BOOL") + .num_args(0..=1) + .require_equals(true) + .default_missing_value("true") + .value_parser(value_parser!(bool)) + .help( + "Let session summaries be recalled (=false keeps them out); \ + default: the server's choice", + ), + ), ) .subcommand( Command::new("get") @@ -611,7 +778,7 @@ pub fn build_memory_command() -> Command { .long("top-k") .default_value("10") .value_parser(value_parser!(usize)) - .help("Number of results to return"), + .help("Most ranked results to return"), ) .arg( Arg::new("type") @@ -620,7 +787,31 @@ pub fn build_memory_command() -> Command { .value_parser(["episodic", "semantic", "procedural", "working"]) .help("Filter by memory type"), ) - .arg(memory_lang_arg("Language of the query")), + .arg(memory_lang_arg("Language of the query")) + .arg( + Arg::new("neighborhood") + .long("neighborhood") + .value_name("MODE") + .default_value("separate") + .value_parser(["separate", "inline", "off"]) + .help( + "Session-neighbour context of the results: separate (listed in its \ + own Context section, not counted in --top-k), inline or off", + ), + ) + .arg( + Arg::new("include-summaries") + .long("include-summaries") + .value_name("BOOL") + .num_args(0..=1) + .require_equals(true) + .default_missing_value("true") + .value_parser(value_parser!(bool)) + .help( + "Let session summaries be recalled (=false keeps them out); \ + default: the server's choice", + ), + ), ) .subcommand(build_memory_extract_command()) .subcommand( diff --git a/src/commands/admin.rs b/src/commands/admin.rs index 120a71a..2e175ab 100644 --- a/src/commands/admin.rs +++ b/src/commands/admin.rs @@ -67,7 +67,10 @@ pub async fn execute(ctx: &Ctx, matches: &ArgMatches) -> Result<()> { Some(("optimize", sub)) => { let namespace = sub.get_one::("namespace").unwrap(); - let path = format!("/admin/namespaces/{}/optimize", namespace); + let path = format!( + "/admin/namespaces/{}/optimize", + crate::api::segment(namespace) + ); let t = ctx.log_request("POST", &path); let result = admin_post(&ctx.url, &path, None).await; ctx.log_response(t, if result.is_ok() { "200 OK" } else { "ERR" }); @@ -220,7 +223,7 @@ pub async fn execute(ctx: &Ctx, matches: &ArgMatches) -> Result<()> { Some(("backup-delete", sub)) => { let backup_id = sub.get_one::("backup_id").unwrap(); - let path = format!("/admin/backups/{}", backup_id); + let path = format!("/admin/backups/{}", crate::api::segment(backup_id)); let t = ctx.log_request("DELETE", &path); let result = admin_delete(&ctx.url, &path).await; ctx.log_response(t, if result.is_ok() { "200 OK" } else { "ERR" }); diff --git a/src/commands/completion.rs b/src/commands/completion.rs index 4e1ec83..51d1b9a 100644 --- a/src/commands/completion.rs +++ b/src/commands/completion.rs @@ -9,7 +9,7 @@ use crate::output; /// All top-level `dk` subcommands (used in every shell script). const TOP_LEVEL_CMDS: &str = - "init health capabilities whoami attachment namespace index memory session agent knowledge admin keys config completion"; + "init health capabilities whoami attachment document namespace index memory session agent knowledge admin keys config completion"; // ─── Bash ──────────────────────────────────────────────────────────────────── @@ -125,6 +125,10 @@ _dk() {{ [[ -z "$sub" ]] && COMPREPLY=($(compgen -W \ "upload list download delete transcribe index job" -- "$cur")) ;; + document) + [[ -z "$sub" ]] && COMPREPLY=($(compgen -W \ + "import job list show delete" -- "$cur")) + ;; keys) [[ -z "$sub" ]] && COMPREPLY=($(compgen -W \ "create list get edit patch delete deactivate rotate usage" -- "$cur")) @@ -198,6 +202,7 @@ _dk() { 'capabilities:Show what the server supports' 'whoami:Show the API key in use' 'attachment:Attachments: upload, transcribe, index' + 'document:Documents: import PDF, DOCX, HTML, Markdown, text' 'namespace:Manage namespaces' 'index:Index management' 'memory:Memory operations' @@ -368,6 +373,17 @@ _dk() { _arguments '1: :->subcmd' [[ $state == subcmd ]] && _describe 'attachment subcommand' at_cmds ;; + document) + local doc_cmds=( + 'import:Import a document as chunk memories' + 'job:Show an import job' + 'list:List documents' + 'show:Show a document and its chunks' + 'delete:Delete a document' + ) + _arguments '1: :->subcmd' + [[ $state == subcmd ]] && _describe 'document subcommand' doc_cmds + ;; esac ;; esac @@ -385,7 +401,7 @@ fn fish_script() -> &'static str { function __dk_no_subcommand for i in (commandline -opc) - if contains -- $i init health capabilities attachment namespace index \ + if contains -- $i init health capabilities attachment document namespace index \ memory session agent knowledge admin keys config completion return 1 end @@ -424,6 +440,7 @@ complete -c dk -f -n '__dk_no_subcommand' -a 'health' -d 'Check server healt complete -c dk -f -n '__dk_no_subcommand' -a 'capabilities' -d 'Show what the server supports' complete -c dk -f -n '__dk_no_subcommand' -a 'whoami' -d 'Show the API key in use' complete -c dk -f -n '__dk_no_subcommand' -a 'attachment' -d 'Attachments: upload, transcribe, index' +complete -c dk -f -n '__dk_no_subcommand' -a 'document' -d 'Documents: import PDF, DOCX, HTML, Markdown, text' complete -c dk -f -n '__dk_no_subcommand' -a 'namespace' -d 'Manage namespaces' complete -c dk -f -n '__dk_no_subcommand' -a 'index' -d 'Index management' complete -c dk -f -n '__dk_no_subcommand' -a 'memory' -d 'Memory operations' @@ -521,6 +538,11 @@ complete -c dk -f -n '__dk_using_subcommand attachment' -a 'delete' -d 'Dele complete -c dk -f -n '__dk_using_subcommand attachment' -a 'transcribe' -d 'Transcribe audio into a memory' complete -c dk -f -n '__dk_using_subcommand attachment' -a 'index' -d 'Index an image as a visual memory' complete -c dk -f -n '__dk_using_subcommand attachment' -a 'job' -d 'Show a transcription or index job' +complete -c dk -f -n '__dk_using_subcommand document' -a 'import' -d 'Import a document as chunk memories' +complete -c dk -f -n '__dk_using_subcommand document' -a 'job' -d 'Show an import job' +complete -c dk -f -n '__dk_using_subcommand document' -a 'list' -d 'List documents' +complete -c dk -f -n '__dk_using_subcommand document' -a 'show' -d 'Show a document and its chunks' +complete -c dk -f -n '__dk_using_subcommand document' -a 'delete' -d 'Delete a document' complete -c dk -f -n '__dk_using_subcommand admin' -a 'backup-delete' -d 'Delete backup' complete -c dk -n '__dk_using_subcommand admin' -l namespace -s n -d 'Namespace' -r -a '(__dk_namespaces)' diff --git a/src/commands/config.rs b/src/commands/config.rs index 54bc5da..8e2ecb3 100644 --- a/src/commands/config.rs +++ b/src/commands/config.rs @@ -2,7 +2,7 @@ use nu_ansi_term::Color; -use crate::config::{Config, Profile}; +use crate::config::{self as cfg, Config, Profile}; pub async fn execute(sub_matches: &clap::ArgMatches) -> anyhow::Result<()> { match sub_matches.subcommand() { @@ -24,9 +24,28 @@ async fn cmd_show(_matches: &clap::ArgMatches) -> anyhow::Result<()> { let file_cfg = Config::read_config_file()?; println!("Configuration:"); - println!(" Server URL: {}", config.server_url); + println!( + " Server URL: {}", + cfg::redact_url(&config.server_url) + ); println!(" Default namespace: {}", config.default_namespace); println!(" Active profile: {}", file_cfg.active_profile); + // The key itself is never printed: only enough of it to tell keys apart. + let key = std::env::var("DAKERA_API_KEY") + .ok() + .filter(|k| !k.is_empty()); + match &key { + Some(k) => println!(" API key: {} (DAKERA_API_KEY)", cfg::mask_key(k)), + None => println!(" API key: not set (DAKERA_API_KEY)"), + } + let transport = if config.server_url.starts_with("https://") { + "TLS (https), server certificate verified" + } else if cfg::is_cleartext_remote(&config.server_url) { + "plain http to a remote host: requests and the API key are not encrypted" + } else { + "plain http (local)" + }; + println!(" Transport: {transport}"); if let Some(path) = Config::config_path() { println!( " Config file: {}{}", @@ -38,6 +57,7 @@ async fn cmd_show(_matches: &clap::ArgMatches) -> anyhow::Result<()> { println!("Environment overrides:"); println!(" DAKERA_URL - Server URL"); println!(" DAKERA_NAMESPACE - Default namespace"); + println!(" DAKERA_API_KEY - API key (read from the environment only, never stored)"); println!(); println!("Tip: run `dk config profile list` to see all profiles."); Ok(()) diff --git a/src/commands/document.rs b/src/commands/document.rs new file mode 100644 index 0000000..5fe01ee --- /dev/null +++ b/src/commands/document.rs @@ -0,0 +1,657 @@ +//! `dk document` — document import (Dakera v0.12.3, Dakera-AI/dakera#923). +//! +//! A document becomes chunk memories of an agent (heading breadcrumb, page, +//! provenance in `_dakera_document`), written by a background job. +//! +//! * `import` text / Markdown / HTML: `POST /v1/memory/documents`; +//! PDF / DOCX (or text that is not UTF-8): `POST /v1/namespaces/{ns}/attachments`, +//! then `POST /v1/namespaces/{ns}/attachments/{ref}/document` +//! * `job` `GET /v1/memory/documents/jobs/{job_id}?agent_id=` +//! * `list` `GET /v1/memory/documents?agent_id=&cursor=&limit=` +//! * `show` `GET /v1/memory/documents/{document_id}?agent_id=` +//! * `delete` `DELETE /v1/memory/documents/{document_id}?agent_id=` + +use std::io::Read; +use std::time::{Duration, Instant}; + +use anyhow::Result; +use clap::ArgMatches; +use dakera_client::reqwest::Method; +use serde::Serialize; +use serde_json::{json, Value}; + +use crate::api::{self, str_of, u64_of, ApiHttpError}; +use crate::context::Context; +use crate::error::input_error; +use crate::output; +use crate::OutputFormat; + +const POLL_INTERVAL: Duration = Duration::from_secs(2); + +/// The media types the text route accepts. +pub const TEXT_MEDIA_TYPES: &[&str] = &["text/plain", "text/markdown", "text/html"]; + +/// The DOCX media type. +pub const DOCX: &str = "application/vnd.openxmlformats-officedocument.wordprocessingml.document"; + +/// What a missing document route means. +const NEEDS_SERVER: &str = "document import needs a Dakera v0.12.3 server: this server has no \ + /v1/memory/documents route (`dk health` shows its version)"; + +pub async fn execute(ctx: &Context, matches: &ArgMatches) -> Result<()> { + let result = match matches.subcommand() { + Some(("import", sub)) => import(ctx, sub).await, + Some(("job", sub)) => job(ctx, sub).await, + Some(("list", sub)) => list(ctx, sub).await, + Some(("show", sub)) => show(ctx, sub).await, + Some(("delete", sub)) => delete(ctx, sub).await, + _ => { + output::error("Unknown document subcommand. Use --help for usage."); + std::process::exit(1); + } + }; + result.map_err(explain_missing_route) +} + +/// A `404 ROUTE_NOT_FOUND` / `405` / bodiless `404`: the server predates the +/// document routes. Other errors pass unchanged. +fn explain_missing_route(err: anyhow::Error) -> anyhow::Error { + let missing = match err.downcast_ref::() { + Some(e) => match e.status { + 405 => true, + 404 => matches!(e.code.as_deref(), None | Some("ROUTE_NOT_FOUND")), + _ => false, + }, + None => false, + }; + if missing { + err.context(NEEDS_SERVER) + } else { + err + } +} + +/// The media type of a document file, from its extension; `None` when the +/// extension says nothing usable. +pub fn media_type_for(path: &str) -> Option<&'static str> { + if path == "-" { + return Some("text/plain"); + } + let ext = std::path::Path::new(path) + .extension() + .and_then(|e| e.to_str()) + .unwrap_or("") + .to_ascii_lowercase(); + match ext.as_str() { + "txt" | "text" | "log" => Some("text/plain"), + "md" | "markdown" | "mdown" | "mkd" => Some("text/markdown"), + "html" | "htm" => Some("text/html"), + "xhtml" => Some("application/xhtml+xml"), + "pdf" => Some("application/pdf"), + "docx" => Some(DOCX), + _ => None, + } +} + +/// Which route imports a document: the text route takes UTF-8 text, Markdown +/// and HTML; anything else (PDF, DOCX, XHTML, text that is not UTF-8) is +/// uploaded as an attachment and imported from there. +#[derive(Debug, PartialEq, Eq)] +pub enum Route { + /// `POST /v1/memory/documents` with this `media_type` and content. + Text { media_type: String, content: String }, + /// Upload with this `Content-Type`, then import the attachment. + Attachment { media_type: String, bytes: Vec }, +} + +/// Choose the route for `bytes` of `media_type`. +pub fn route_for(media_type: &str, bytes: Vec) -> Route { + if TEXT_MEDIA_TYPES.contains(&media_type) { + match String::from_utf8(bytes) { + Ok(content) => { + return Route::Text { + media_type: media_type.to_string(), + content, + } + } + // UTF-16 or Windows-1252: the server decodes it from the bytes. + Err(e) => { + return Route::Attachment { + media_type: media_type.to_string(), + bytes: e.into_bytes(), + } + } + } + } + Route::Attachment { + media_type: media_type.to_string(), + bytes, + } +} + +/// The fields both import routes share, from the flags: only what was given +/// (the server refuses unknown `chunking` fields). +pub fn common_body(sub: &ArgMatches, agent_id: &str) -> Value { + let mut body = json!({ "agent_id": agent_id }); + for (flag, field) in [ + ("title", "title"), + ("document-id", "document_id"), + ("session-id", "session_id"), + ("type", "memory_type"), + ("lang", "lang"), + ] { + if let Some(v) = sub.get_one::(flag) { + body[field] = json!(v); + } + } + if let Some(v) = sub.get_one::("importance") { + body["importance"] = json!(v); + } + if let Some(tags) = sub.get_many::("tag") { + let tags: Vec<&String> = tags.collect(); + body["tags"] = json!(tags); + } + if sub.get_flag("decompose") { + body["decompose"] = json!(true); + } + let mut chunking = serde_json::Map::new(); + if let Some(n) = sub.get_one::("chunk-tokens") { + chunking.insert("max_tokens".into(), json!(n)); + } + if let Some(n) = sub.get_one::("overlap") { + chunking.insert("overlap_tokens".into(), json!(n)); + } + if let Some(s) = sub.get_one::("strategy") { + chunking.insert("strategy".into(), json!(s)); + } + if !chunking.is_empty() { + body["chunking"] = Value::Object(chunking); + } + body +} + +fn read_source(file: &str) -> Result> { + if file == "-" { + let mut data = Vec::new(); + std::io::stdin() + .read_to_end(&mut data) + .map_err(|e| input_error(format!("failed to read stdin: {e}")))?; + return Ok(data); + } + std::fs::read(file).map_err(|e| input_error(format!("failed to read file {file}: {e}"))) +} + +/// `?agent_id=` with the id encoded. +fn agent_query(agent_id: &str) -> String { + let mut query = String::from("agent_id="); + query.push_str(&encode_query(agent_id)); + query +} + +/// Percent-encode a query value. +fn encode_query(raw: &str) -> String { + api::segment(raw).replace(':', "%3A") +} + +/// The job-status route of an agent's import job. +pub fn job_path(agent_id: &str, job_id: &str) -> String { + let job = api::segment(job_id); + let query = agent_query(agent_id); + format!("/v1/memory/documents/jobs/{job}?{query}") +} + +/// The route of one document. +pub fn document_path(agent_id: &str, document_id: &str) -> String { + let id = api::segment(document_id); + let query = agent_query(agent_id); + format!("/v1/memory/documents/{id}?{query}") +} + +async fn import(ctx: &Context, sub: &ArgMatches) -> Result<()> { + let file = sub.get_one::("file").unwrap(); + let agent_id = sub.get_one::("agent").unwrap(); + let media_type = match sub.get_one::("media-type") { + Some(m) => m.clone(), + None => match media_type_for(file) { + Some(m) => m.to_string(), + None => { + return Err(input_error(format!( + "cannot tell the format of {file} from its extension; pass --media-type \ + (text/plain, text/markdown, text/html, application/pdf or {DOCX})" + ))) + } + }, + }; + let bytes = read_source(file)?; + let mut body = common_body(sub, agent_id); + let accepted = match route_for(&media_type, bytes) { + Route::Text { + media_type, + content, + } => { + body["content"] = json!(content); + body["media_type"] = json!(media_type); + api::request_json_logged(ctx, Method::POST, "/v1/memory/documents", Some(&body)).await? + } + Route::Attachment { media_type, bytes } => { + let namespace = match sub.get_one::("namespace") { + Some(ns) => ns.clone(), + None => format!("_dakera_agent_{agent_id}"), + }; + let ns = api::segment(&namespace); + let upload = format!("/v1/namespaces/{ns}/attachments"); + let uploaded = api::post_bytes(ctx, &upload, &media_type, bytes).await?; + let Some(reference) = str_of(&uploaded, "attachment_ref") else { + anyhow::bail!("the attachment upload answered no attachment_ref: {uploaded}"); + }; + let reference = api::segment(reference); + let path = format!("/v1/namespaces/{ns}/attachments/{reference}/document"); + api::request_json_logged(ctx, Method::POST, &path, Some(&body)).await? + } + }; + let table = matches!(ctx.format, OutputFormat::Table); + let job_id = str_of(&accepted, "job_id").unwrap_or("").to_string(); + let document_id = str_of(&accepted, "document_id").unwrap_or("").to_string(); + if !sub.get_flag("wait") || job_id.is_empty() { + if table { + output::success(&format!( + "Import of document '{document_id}' started (job {job_id})" + )); + output::info(&format!( + "Follow it with `dk document job {job_id} --agent {agent_id}`" + )); + } else { + output::print_item(&accepted, ctx.format); + } + return Ok(()); + } + let timeout = sub.get_one::("timeout").copied().unwrap_or(600); + let finished = wait_for_job(ctx, agent_id, &job_id, timeout).await?; + let status = str_of(&finished, "status").unwrap_or(""); + if status != "Completed" { + return Err(job_failed(&finished)); + } + if table { + let message = str_of(&finished, "message").unwrap_or("done"); + output::success(&format!("Document '{document_id}' imported: {message}")); + } else { + let out = json!({ "accepted": accepted, "job": finished }); + output::print_item(&out, ctx.format); + } + Ok(()) +} + +/// The error of a job that did not complete, with its status and code. +fn job_failed(job: &Value) -> anyhow::Error { + let status = str_of(job, "status").unwrap_or("?"); + let message = str_of(job, "message").unwrap_or(""); + let code = api::ptr_str(job, "/error/code").unwrap_or(""); + let http = api::ptr_u64(job, "/error/status").unwrap_or(0); + if http > 0 { + anyhow::anyhow!("document import {status} ({http} {code}): {message}") + } else { + anyhow::anyhow!("document import {status}: {message}") + } +} + +/// Poll the job until it ends or `timeout_secs` pass. +async fn wait_for_job( + ctx: &Context, + agent_id: &str, + job_id: &str, + timeout_secs: u64, +) -> Result { + let deadline = Instant::now() + Duration::from_secs(timeout_secs); + let path = job_path(agent_id, job_id); + loop { + let job = api::request_json_logged(ctx, Method::GET, &path, None).await?; + match str_of(&job, "status") { + Some("Completed") | Some("Failed") | Some("Cancelled") => return Ok(job), + _ => {} + } + if Instant::now() + POLL_INTERVAL >= deadline { + let progress = u64_of(&job, "progress").unwrap_or(0); + anyhow::bail!( + "the import job {job_id} is still running after {timeout_secs}s ({progress}%); \ + follow it with `dk document job {job_id} --agent {agent_id}`" + ); + } + tokio::time::sleep(POLL_INTERVAL).await; + } +} + +fn show_value(v: Option<&Value>) -> String { + match v { + None | Some(Value::Null) => "-".to_string(), + Some(Value::String(s)) => s.clone(), + Some(other) => other.to_string(), + } +} + +async fn job(ctx: &Context, sub: &ArgMatches) -> Result<()> { + let agent_id = sub.get_one::("agent").unwrap(); + let job_id = sub.get_one::("job_id").unwrap(); + let job = api::request_json_logged(ctx, Method::GET, &job_path(agent_id, job_id), None).await?; + if !matches!(ctx.format, OutputFormat::Table) { + output::print_item(&job, ctx.format); + return Ok(()); + } + let error = match ( + api::ptr_u64(&job, "/error/status"), + api::ptr_str(&job, "/error/code"), + ) { + (Some(status), Some(code)) => format!("{status} {code}"), + _ => "-".to_string(), + }; + let progress = format!("{}%", u64_of(&job, "progress").unwrap_or(0)); + let pairs = vec![ + ("Job", show_value(job.get("id"))), + ("Status", show_value(job.get("status"))), + ("Progress", progress), + ("Message", show_value(job.get("message"))), + ("Document", show_value(job.pointer("/metadata/document_id"))), + ("Error", error), + ]; + output::print_kv(&pairs, ctx.format); + Ok(()) +} + +/// One row of `dk document list`. +#[derive(Serialize)] +struct DocumentRow { + document_id: String, + title: String, + media_type: String, + chunks: u64, + bytes: u64, + pages: String, + updated_at: u64, +} + +fn document_row(d: &Value) -> DocumentRow { + DocumentRow { + document_id: str_of(d, "document_id").unwrap_or("").to_string(), + title: str_of(d, "title").unwrap_or("-").to_string(), + media_type: str_of(d, "media_type").unwrap_or("").to_string(), + chunks: u64_of(d, "chunk_count").unwrap_or(0), + bytes: u64_of(d, "bytes").unwrap_or(0), + pages: show_value(d.get("pages")), + updated_at: u64_of(d, "updated_at").unwrap_or(0), + } +} + +/// The listing route with its query. +pub fn list_path(agent_id: &str, limit: Option, cursor: Option<&str>) -> String { + let mut path = format!("/v1/memory/documents?{}", agent_query(agent_id)); + if let Some(limit) = limit { + path.push_str(&format!("&limit={limit}")); + } + if let Some(cursor) = cursor { + path.push_str("&cursor="); + path.push_str(&encode_query(cursor)); + } + path +} + +async fn list(ctx: &Context, sub: &ArgMatches) -> Result<()> { + let agent_id = sub.get_one::("agent").unwrap(); + let limit = sub.get_one::("limit").copied(); + let cursor = sub.get_one::("cursor").map(String::as_str); + let path = list_path(agent_id, limit, cursor); + let result = api::request_json_logged(ctx, Method::GET, &path, None).await?; + if !matches!(ctx.format, OutputFormat::Table) { + output::print_item(&result, ctx.format); + return Ok(()); + } + let empty = Vec::new(); + let documents = result + .get("documents") + .and_then(|d| d.as_array()) + .unwrap_or(&empty); + if documents.is_empty() { + output::info(&format!("No documents for agent '{agent_id}'")); + } else { + let rows: Vec = documents.iter().map(document_row).collect(); + output::print_data(&rows, ctx.format); + } + if let Some(next) = str_of(&result, "next_cursor") { + output::info(&format!("More: --cursor {next}")); + } + Ok(()) +} + +/// One chunk row of `dk document show`. +#[derive(Serialize)] +struct ChunkRow { + index: u64, + id: String, + pages: String, + heading_path: String, + chars: String, +} + +/// A page range as `3` or `3-4`, `-` when the chunk has none. +pub fn page_range(start: Option, end: Option) -> String { + match (start, end) { + (Some(s), Some(e)) if e != s => format!("{s}-{e}"), + (Some(s), _) => s.to_string(), + (None, Some(e)) => e.to_string(), + (None, None) => "-".to_string(), + } +} + +/// A heading path as one breadcrumb (`Manual › Install › Linux`). +pub fn breadcrumb(path: Option<&Value>) -> String { + let parts: Vec<&str> = path + .and_then(|p| p.as_array()) + .map(|items| items.iter().filter_map(|h| h.as_str()).collect()) + .unwrap_or_default(); + parts.join(" › ") +} + +async fn show(ctx: &Context, sub: &ArgMatches) -> Result<()> { + let agent_id = sub.get_one::("agent").unwrap(); + let document_id = sub.get_one::("document_id").unwrap(); + let path = document_path(agent_id, document_id); + let detail = api::request_json_logged(ctx, Method::GET, &path, None).await?; + if !matches!(ctx.format, OutputFormat::Table) { + output::print_item(&detail, ctx.format); + return Ok(()); + } + let doc = detail.get("document").cloned().unwrap_or(Value::Null); + let pairs = vec![ + ("Document", show_value(doc.get("document_id"))), + ("Title", show_value(doc.get("title"))), + ("Media type", show_value(doc.get("media_type"))), + ("Chunks", show_value(doc.get("chunk_count"))), + ("Bytes", show_value(doc.get("bytes"))), + ("Pages", show_value(doc.get("pages"))), + ("SHA-256", show_value(doc.get("sha256"))), + ("Attachment", show_value(doc.get("attachment_ref"))), + ("Created at", show_value(doc.get("created_at"))), + ("Updated at", show_value(doc.get("updated_at"))), + ]; + output::print_kv(&pairs, ctx.format); + let empty = Vec::new(); + let chunks = detail + .get("chunks") + .and_then(|c| c.as_array()) + .unwrap_or(&empty); + if chunks.is_empty() { + return Ok(()); + } + println!(); + let rows: Vec = chunks + .iter() + .map(|c| ChunkRow { + index: u64_of(c, "chunk_index").unwrap_or(0), + id: str_of(c, "id").unwrap_or("").to_string(), + pages: page_range(u64_of(c, "page_start"), u64_of(c, "page_end")), + heading_path: breadcrumb(c.get("heading_path")), + chars: format!( + "{}-{}", + u64_of(c, "char_start").unwrap_or(0), + u64_of(c, "char_end").unwrap_or(0) + ), + }) + .collect(); + output::print_data(&rows, ctx.format); + Ok(()) +} + +async fn delete(ctx: &Context, sub: &ArgMatches) -> Result<()> { + let agent_id = sub.get_one::("agent").unwrap(); + let document_id = sub.get_one::("document_id").unwrap(); + let path = document_path(agent_id, document_id); + let result = api::request_json_logged(ctx, Method::DELETE, &path, None).await?; + if !matches!(ctx.format, OutputFormat::Table) { + output::print_item(&result, ctx.format); + return Ok(()); + } + let chunks = u64_of(&result, "deleted_chunks").unwrap_or(0); + output::success(&format!( + "Document '{document_id}' deleted ({chunks} chunk memories)" + )); + Ok(()) +} + +/// The citation of a recall result that is a document chunk: from +/// `memory.metadata._dakera_document` (flat results carry `metadata` at the +/// top level). +pub fn citation(item: &Value) -> Option { + let memory = item.get("memory").unwrap_or(item); + let doc = memory.get("metadata")?.get("_dakera_document")?; + let id = doc.get("id")?.as_str()?.to_string(); + let heading_path = doc + .get("heading_path") + .and_then(|p| p.as_array()) + .map(|items| { + items + .iter() + .filter_map(|h| h.as_str().map(str::to_string)) + .collect() + }) + .unwrap_or_default(); + Some(Citation { + document_id: id, + title: str_of(doc, "title").map(str::to_string), + page_start: u64_of(doc, "page_start"), + page_end: u64_of(doc, "page_end"), + heading_path, + chunk_index: u64_of(doc, "chunk_index"), + }) +} + +/// Where a recalled chunk comes from. +#[derive(Debug, Clone, Serialize, PartialEq, Eq)] +pub struct Citation { + /// The document id. + pub document_id: String, + /// The document title, when it has one. + #[serde(skip_serializing_if = "Option::is_none")] + pub title: Option, + /// First page of the chunk's text (PDF, form-feed pages of text). + #[serde(skip_serializing_if = "Option::is_none")] + pub page_start: Option, + /// Last page of the chunk's text. + #[serde(skip_serializing_if = "Option::is_none")] + pub page_end: Option, + /// Headings above the chunk, outermost first. + pub heading_path: Vec, + /// Position of the chunk in the document, from 0. + #[serde(skip_serializing_if = "Option::is_none")] + pub chunk_index: Option, +} + +impl Citation { + /// One line: `Field Manual, p. 39-40 › Install › Linux`. + pub fn line(&self) -> String { + let mut out = self + .title + .clone() + .unwrap_or_else(|| self.document_id.clone()); + if self.page_start.is_some() || self.page_end.is_some() { + out.push_str(&format!( + ", p. {}", + page_range(self.page_start, self.page_end) + )); + } + // The title is the breadcrumb's root: do not repeat it. + let skip = usize::from(self.heading_path.first() == self.title.as_ref()); + let rest: Vec<&str> = self.heading_path[skip.min(self.heading_path.len())..] + .iter() + .map(String::as_str) + .collect(); + if !rest.is_empty() { + out.push_str(" › "); + out.push_str(&rest.join(" › ")); + } + out + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn media_types_follow_the_extension() { + assert_eq!(media_type_for("a.MD"), Some("text/markdown")); + assert_eq!(media_type_for("a.htm"), Some("text/html")); + assert_eq!(media_type_for("manual.pdf"), Some("application/pdf")); + assert_eq!(media_type_for("x.docx"), Some(DOCX)); + assert_eq!(media_type_for("-"), Some("text/plain")); + assert_eq!(media_type_for("x.bin"), None); + } + + #[test] + fn text_goes_to_the_text_route_and_the_rest_to_attachments() { + let text = route_for("text/markdown", b"# T".to_vec()); + assert_eq!( + text, + Route::Text { + media_type: "text/markdown".into(), + content: "# T".into() + } + ); + assert!(matches!( + route_for("application/pdf", b"%PDF".to_vec()), + Route::Attachment { .. } + )); + // Not UTF-8: the server decodes the bytes itself. + let latin = route_for("text/plain", vec![0x63, 0x61, 0x66, 0xE9]); + assert!(matches!(latin, Route::Attachment { ref bytes, .. } if bytes.len() == 4)); + } + + #[test] + fn paths_encode_ids() { + assert_eq!( + job_path("a b", "job_1"), + "/v1/memory/documents/jobs/job_1?agent_id=a%20b" + ); + assert_eq!( + document_path("bot", "v3/manual"), + "/v1/memory/documents/v3%2Fmanual?agent_id=bot" + ); + assert_eq!( + list_path("bot", Some(10), Some("m:1")), + "/v1/memory/documents?agent_id=bot&limit=10&cursor=m%3A1" + ); + } + + #[test] + fn a_citation_names_title_pages_and_sections() { + let item = json!({"memory": {"id": "doc_1", "metadata": {"_dakera_document": { + "id": "manual", "title": "Field Manual", "page_start": 39, "page_end": 40, + "heading_path": ["Field Manual", "Install", "Linux"], "chunk_index": 41 + }}}}); + let c = citation(&item).unwrap(); + assert_eq!(c.line(), "Field Manual, p. 39-40 › Install › Linux"); + let plain = json!({"memory": {"id": "m", "metadata": {"source": "x"}}}); + assert!(citation(&plain).is_none()); + let untitled = json!({"memory": {"metadata": {"_dakera_document": { + "id": "notes", "heading_path": ["Setup"] + }}}}); + assert_eq!(citation(&untitled).unwrap().line(), "notes › Setup"); + } +} diff --git a/src/commands/keys.rs b/src/commands/keys.rs index e22be15..013b30f 100644 --- a/src/commands/keys.rs +++ b/src/commands/keys.rs @@ -1,7 +1,8 @@ //! API key management commands -use anyhow::{Context, Result}; +use anyhow::Result; use clap::ArgMatches; +use dakera_client::reqwest::Method; use dakera_client::{DakeraClient, KeyInfo, UpdateKeyRequest, WhoamiResponse}; use serde_json::Value; @@ -118,63 +119,25 @@ pub async fn whoami(ctx: &Ctx) -> Result<()> { Ok(()) } +// Through `api::request_json`: a refusal becomes an `ApiHttpError`, with its +// exit code and hint (403 scope, 422 unknown field, ...), not raw JSON text. async fn keys_get(url: &str, path: &str) -> Result { - let client = super::authed_client(); - let resp = client - .get(format!("{}{}", url, path)) - .send() - .await - .with_context(|| format!("Failed to GET {}", path))?; - - let status = resp.status(); - let body = resp.text().await?; - if !status.is_success() { - anyhow::bail!("Request failed ({}): {}", status, body); - } - serde_json::from_str(&body).with_context(|| "Failed to parse response JSON") + crate::api::request_json(url, Method::GET, path, None).await } async fn keys_post(url: &str, path: &str, body: Option<&Value>) -> Result { - let client = super::authed_client(); - let mut req = client.post(format!("{}{}", url, path)); - if let Some(b) = body { - req = req.json(b); - } - let resp = req - .send() - .await - .with_context(|| format!("Failed to POST {}", path))?; - - let status = resp.status(); - let text = resp.text().await?; - if !status.is_success() { - anyhow::bail!("Request failed ({}): {}", status, text); - } - if text.is_empty() { - Ok(Value::Object(serde_json::Map::new())) - } else { - serde_json::from_str(&text).with_context(|| "Failed to parse response JSON") - } + crate::api::request_json(url, Method::POST, path, body).await } async fn keys_delete(url: &str, path: &str) -> Result { - let client = super::authed_client(); - let resp = client - .delete(format!("{}{}", url, path)) - .send() - .await - .with_context(|| format!("Failed to DELETE {}", path))?; - - let status = resp.status(); - let text = resp.text().await?; - if !status.is_success() { - anyhow::bail!("Request failed ({}): {}", status, text); - } - if text.is_empty() { - Ok(Value::Object(serde_json::Map::new())) - } else { - serde_json::from_str(&text).with_context(|| "Failed to parse response JSON") - } + crate::api::request_json(url, Method::DELETE, path, None).await +} + +/// `/admin/keys/{key_id}{suffix}` with the id percent-encoded. +fn key_path(key_id: &str, suffix: &str) -> String { + let mut path = format!("/admin/keys/{}", crate::api::segment(key_id)); + path.push_str(suffix); + path } pub async fn execute(ctx: &Ctx, matches: &ArgMatches) -> Result<()> { @@ -201,6 +164,10 @@ pub async fn execute(ctx: &Ctx, matches: &ArgMatches) -> Result<()> { ctx.log_response(t, if result.is_ok() { "200 OK" } else { "ERR" }); let result = result?; + if !matches!(ctx.format, OutputFormat::Table) { + output::print_item(&result, ctx.format); + return Ok(()); + } if let Some(key) = result.get("key").and_then(|k| k.as_str()) { output::success(&format!("API key created: {}", name)); output::warning("Save this key now - it will not be shown again!"); @@ -227,7 +194,7 @@ pub async fn execute(ctx: &Ctx, matches: &ArgMatches) -> Result<()> { Some(("get", sub)) => { let key_id = sub.get_one::("key_id").unwrap(); - let path = format!("/admin/keys/{}", key_id); + let path = key_path(key_id, ""); let t = ctx.log_request("GET", &path); let result = keys_get(&ctx.url, &path).await; ctx.log_response(t, if result.is_ok() { "200 OK" } else { "ERR" }); @@ -236,7 +203,7 @@ pub async fn execute(ctx: &Ctx, matches: &ArgMatches) -> Result<()> { Some(("delete", sub)) => { let key_id = sub.get_one::("key_id").unwrap(); - let path = format!("/admin/keys/{}", key_id); + let path = key_path(key_id, ""); let t = ctx.log_request("DELETE", &path); let result = keys_delete(&ctx.url, &path).await; ctx.log_response(t, if result.is_ok() { "200 OK" } else { "ERR" }); @@ -246,7 +213,7 @@ pub async fn execute(ctx: &Ctx, matches: &ArgMatches) -> Result<()> { Some(("deactivate", sub)) => { let key_id = sub.get_one::("key_id").unwrap(); - let path = format!("/admin/keys/{}/deactivate", key_id); + let path = key_path(key_id, "/deactivate"); let t = ctx.log_request("POST", &path); let result = keys_post(&ctx.url, &path, None).await; ctx.log_response(t, if result.is_ok() { "200 OK" } else { "ERR" }); @@ -266,7 +233,7 @@ pub async fn execute(ctx: &Ctx, matches: &ArgMatches) -> Result<()> { } } let client = DakeraClient::new(&ctx.url)?; - let path = format!("/admin/keys/{}/rotate", key_id); + let path = key_path(key_id, "/rotate"); let t = ctx.log_request("POST", &path); let result = match grace { Some(g) => client.rotate_key_with_grace(key_id, g).await, @@ -321,7 +288,7 @@ pub async fn execute(ctx: &Ctx, matches: &ArgMatches) -> Result<()> { Some(("usage", sub)) => { let key_id = sub.get_one::("key_id").unwrap(); - let path = format!("/admin/keys/{}/usage", key_id); + let path = key_path(key_id, "/usage"); let t = ctx.log_request("GET", &path); let result = keys_get(&ctx.url, &path).await; ctx.log_response(t, if result.is_ok() { "200 OK" } else { "ERR" }); diff --git a/src/commands/memory.rs b/src/commands/memory.rs index 60d6338..42495d7 100644 --- a/src/commands/memory.rs +++ b/src/commands/memory.rs @@ -64,6 +64,273 @@ fn found_message(returned: usize, total: usize) -> String { } } +/// One ranked result of `dk memory recall`. +#[derive(Debug, Serialize)] +pub struct RecallRow { + /// Memory id. + pub id: String, + /// Memory content (stored data, shown as is). + pub content: String, + /// `episodic`, `semantic`, `procedural` or `working`. + pub memory_type: String, + /// Importance, 0.0 to 1.0. + pub importance: f32, + /// The ranking score: the server's `rank_score` (Dakera v0.12.3, the field + /// that follows the order of the results), else `smart_score`, else + /// `weighted_score`, else `score`. + pub score: f32, + /// The derived sentence that matched best, when the server collapsed + /// sentence sub-memories into this source memory and the sentence + /// outranked the source's own text. + #[serde(skip_serializing_if = "Option::is_none")] + pub matched_text: Option, + /// The ids of the collapsed sentence sub-memories, best first. + #[serde(skip_serializing_if = "Option::is_none")] + pub matched_ids: Option>, + /// Where the result comes from, when it is a chunk of an imported + /// document (`_dakera_document`, Dakera v0.12.3). + #[serde(skip_serializing_if = "Option::is_none")] + pub document: Option, +} + +/// One neighbourhood-context entry of `dk memory recall`. +#[derive(Debug, Serialize)] +pub struct ContextRow { + /// Memory id. + pub id: String, + /// Memory content (stored data, shown as is). + pub content: String, + /// The id of the ranked result this memory is context for (absent when + /// the server does not say: inline context, or a server before v0.12.3). + #[serde(skip_serializing_if = "Option::is_none")] + pub context_of: Option, + /// The 1-based place of that result in `memories`. + #[serde(skip_serializing_if = "Option::is_none")] + pub context_of_rank: Option, + /// The raw similarity of the neighbourhood search: not comparable with + /// the ranked results' scores. + pub score: f32, + /// Where the memory comes from, when it is a document chunk. + #[serde(skip_serializing_if = "Option::is_none")] + pub document: Option, +} + +/// What `dk memory recall` shows: at most `--top-k` ranked results, and the +/// neighbourhood context apart from them. +#[derive(Debug, Serialize)] +pub struct RecallView { + /// The ranked results, at most `--top-k`. + pub memories: Vec, + /// The neighbourhood context of those results. + pub context_memories: Vec, + /// Context items an older server appended to `memories` that were left + /// out (under `--neighborhood separate` / `off`). Not printed in JSON. + #[serde(skip)] + pub omitted: usize, +} + +/// The body of `POST /v1/memory/recall`: the SDK request's fields plus +/// `neighborhood_mode` (and `neighborhood: false` for `off`, which servers +/// before `neighborhood_mode` honour) and `include_summaries` when given. +pub fn recall_body( + request: &RecallRequest, + mode: &str, + include_summaries: Option, +) -> Result { + let mut body = serde_json::to_value(request)?; + body["neighborhood_mode"] = serde_json::json!(mode); + if mode == "off" { + body["neighborhood"] = serde_json::json!(false); + } + if let Some(include) = include_summaries { + body["include_summaries"] = serde_json::json!(include); + } + Ok(body) +} + +/// One recall result as a row, read with the SDK's result parser (which +/// takes the server's `{memory, score, ...}` items and flat ones). +fn recalled(item: &serde_json::Value) -> Result { + serde_json::from_value(item.clone()) + .map_err(|e| anyhow::anyhow!("Failed to parse a recall result: {e}")) +} + +/// The score a ranked result is listed with. Since v0.12.3 the server sends +/// `rank_score`, which never rises down `memories`; `score` and `smart_score` +/// can, because later stages (the precision rerank, the per-session cap, the +/// temporal supplement) reorder the results after they are scored. Older +/// servers send no `rank_score`: `fallback` is the SDK's pick then. +fn ranking_score(item: &serde_json::Value, fallback: f32) -> f32 { + item.get("rank_score") + .and_then(serde_json::Value::as_f64) + .map(|v| v as f32) + .unwrap_or(fallback) +} + +fn recall_row(item: &serde_json::Value) -> Result { + let m = recalled(item)?; + let matched_ids = item.get("matched_ids").and_then(|v| { + v.as_array().map(|ids| { + ids.iter() + .filter_map(|id| id.as_str().map(str::to_string)) + .collect() + }) + }); + Ok(RecallRow { + id: m.id, + content: m.content, + memory_type: memory_type_to_string(&m.memory_type), + importance: m.importance, + score: ranking_score(item, m.score), + matched_text: api::str_of(item, "matched_text").map(str::to_string), + matched_ids, + document: super::document::citation(item), + }) +} + +/// Split a recall answer into ranked results (at most `top_k`) and context. +/// Inline neighbourhood items (`retrieval: "neighborhood"`) and the +/// `context_memories` of `separate` mode become context. A server before +/// `neighborhood_mode` appends its context to `memories` unmarked: past +/// `top_k` it is context under `inline` and left out otherwise. +pub fn shape_recall(answer: &serde_json::Value, top_k: usize, mode: &str) -> Result { + let empty = Vec::new(); + let items = answer + .get("memories") + .and_then(|v| v.as_array()) + .unwrap_or(&empty); + let (context_items, mut ranked_items): (Vec<&serde_json::Value>, Vec<&serde_json::Value>) = + items + .iter() + .partition(|item| api::str_of(item, "retrieval") == Some("neighborhood")); + let mut context_items = context_items; + let overflow = if ranked_items.len() > top_k { + ranked_items.split_off(top_k) + } else { + Vec::new() + }; + let mut omitted = 0; + if mode == "inline" { + context_items.extend(overflow); + } else { + omitted = overflow.len(); + } + if let Some(separate) = answer.get("context_memories").and_then(|v| v.as_array()) { + context_items.extend(separate.iter()); + } + let memories = ranked_items + .into_iter() + .map(recall_row) + .collect::>>()?; + let mut context_memories = Vec::with_capacity(context_items.len()); + for item in context_items { + let m = recalled(item)?; + let context_of = api::str_of(item, "context_of").map(str::to_string); + let context_of_rank = context_of + .as_deref() + .and_then(|id| memories.iter().position(|r| r.id == id)) + .map(|i| i + 1); + context_memories.push(ContextRow { + id: m.id, + content: m.content, + context_of, + context_of_rank, + score: m.score, + document: super::document::citation(item), + }); + } + Ok(RecallView { + memories, + context_memories, + omitted, + }) +} + +/// Print a recall: the ranked table (with a `matched_text` column when a +/// result has one, and a `source` column citing document chunks), then a +/// separate Context section. JSON / compact print +/// `{"memories": [...], "context_memories": [...]}` and nothing else on +/// stdout; the notes go to stderr there. +pub fn print_recall(view: &RecallView, format: crate::OutputFormat) { + let table = matches!(format, crate::OutputFormat::Table); + if view.omitted > 0 { + let note = format!( + "{} neighbourhood context item(s) left out: this server predates neighborhood_mode \ + and returns them inline; only the --top-k ranked results are shown", + view.omitted + ); + if table { + output::warning(¬e); + } else { + eprintln!("warning: {note}"); + } + } + if !table { + output::print_item(view, format); + return; + } + if view.memories.is_empty() { + output::info("No memories found"); + } else { + output::info(&found_message(view.memories.len(), 0)); + let matched = view.memories.iter().any(|r| r.matched_text.is_some()); + let cited = view.memories.iter().any(|r| r.document.is_some()); + let rows: Vec = view + .memories + .iter() + .map(|r| { + let mut row = serde_json::json!({ + "id": r.id, + "content": r.content, + "memory_type": r.memory_type, + "importance": r.importance, + "score": r.score, + }); + if matched { + row["matched_text"] = serde_json::json!(r.matched_text); + } + if cited { + row["source"] = serde_json::json!(r.document.as_ref().map(|d| d.line())); + } + row + }) + .collect(); + output::print_data(&rows, format); + } + if view.context_memories.is_empty() { + return; + } + println!(); + output::info(&format!( + "Context: {} session neighbour(s) of the results above (not ranked, not counted in \ + --top-k; scores come from another search)", + view.context_memories.len() + )); + let cited = view.context_memories.iter().any(|c| c.document.is_some()); + let rows: Vec = view + .context_memories + .iter() + .map(|c| { + let context_of = match (c.context_of_rank, c.context_of.as_deref()) { + (Some(rank), Some(id)) => format!("#{rank} {id}"), + (None, Some(id)) => id.to_string(), + _ => "-".to_string(), + }; + let mut row = serde_json::json!({ + "id": c.id, + "content": c.content, + "context_of": context_of, + "score": c.score, + }); + if cited { + row["source"] = serde_json::json!(c.document.as_ref().map(|d| d.line())); + } + row + }) + .collect(); + output::print_data(&rows, format); +} + fn memory_type_to_string(mt: &MemoryType) -> String { match mt { MemoryType::Episodic => "episodic".to_string(), @@ -232,35 +499,21 @@ pub async fn execute(ctx: &Context, matches: &ArgMatches) -> Result<()> { request = request.with_type(parse_memory_type(t)); } request.lang = sub_matches.get_one::("lang").cloned(); - - let t = ctx.log_request("POST", "/v1/memory/recall"); - let response = client.recall(request).await; - match &response { - Ok(_) => ctx.log_response(t, "200 OK"), - Err(_) => ctx.log_response(t, "ERR"), - } - let response = response?; - - if response.memories.is_empty() { - output::info("No memories found"); - } else { - output::info(&found_message( - response.memories.len(), - response.total_found, - )); - let rows: Vec = response - .memories - .into_iter() - .map(|m| MemoryRow { - id: m.id, - content: m.content, - memory_type: memory_type_to_string(&m.memory_type), - importance: m.importance, - score: m.score, - }) - .collect(); - output::print_data(&rows, ctx.format); - } + let mode = sub_matches + .get_one::("neighborhood") + .map(String::as_str) + .unwrap_or("separate"); + let include_summaries = sub_matches.get_one::("include-summaries").copied(); + let body = recall_body(&request, mode, include_summaries)?; + + // Raw request: the `neighborhood_mode` / `include_summaries` fields + // and the `context_memories` / `matched_text` answer fields are newer + // than the SDK's recall types. + let answer = + api::request_json_logged(ctx, Method::POST, "/v1/memory/recall", Some(&body)) + .await?; + let shaped = shape_recall(&answer, top_k, mode)?; + print_recall(&shaped, ctx.format); } Some(("get", sub_matches)) => { @@ -743,4 +996,42 @@ mod tests { assert_eq!(json["id"], "mem-1"); assert_eq!(json["memory_type"], "episodic"); } + + fn ranked(id: &str, score: f64, smart: f64, rank: Option) -> serde_json::Value { + let mut item = serde_json::json!({ + "memory": { + "id": id, "content": "c", "memory_type": "episodic", "importance": 0.5, + "agent_id": "a", "tags": [], "created_at": 1, "last_accessed_at": 1, + "access_count": 0 + }, + "score": score, + "smart_score": smart, + }); + if let Some(r) = rank { + item["rank_score"] = serde_json::json!(r); + } + item + } + + /// v0.12.3 (#940): the listed score is `rank_score`, which follows the + /// order of the results, while `score` / `smart_score` may rise down it. + #[test] + fn recall_lists_rank_score_when_the_server_sends_it() { + let answer = serde_json::json!({"memories": [ + ranked("m1", 0.2, 0.5, Some(0.9)), + ranked("m2", 0.8, 0.7, Some(0.6)), + ]}); + let view = shape_recall(&answer, 10, "separate").unwrap(); + let scores: Vec = view.memories.iter().map(|r| r.score).collect(); + assert_eq!(scores, vec![0.9, 0.6]); + } + + /// A server before v0.12.3 sends no `rank_score`: the SDK's pick + /// (`smart_score` first) is listed, as before. + #[test] + fn recall_without_rank_score_lists_smart_score() { + let answer = serde_json::json!({"memories": [ranked("m1", 0.2, 0.5, None)]}); + let view = shape_recall(&answer, 10, "separate").unwrap(); + assert_eq!(view.memories[0].score, 0.5); + } } diff --git a/src/commands/mod.rs b/src/commands/mod.rs index b659c0d..691af06 100644 --- a/src/commands/mod.rs +++ b/src/commands/mod.rs @@ -7,6 +7,7 @@ pub mod attachment; pub mod capabilities; pub mod completion; pub mod config; +pub mod document; pub mod health; pub mod index; pub mod init; @@ -17,6 +18,9 @@ pub mod namespace; pub mod session; pub mod text; +/// The longest wait for the next bytes of an answer. +pub(crate) const READ_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(330); + /// Build a reqwest client that forwards DAKERA_API_KEY as a Bearer token when set. pub(crate) fn authed_client() -> dakera_client::reqwest::Client { use dakera_client::reqwest::header::{HeaderMap, HeaderValue, AUTHORIZATION}; @@ -30,6 +34,12 @@ pub(crate) fn authed_client() -> dakera_client::reqwest::Client { dakera_client::reqwest::Client::builder() .user_agent(format!("dakera-cli/{}", env!("CARGO_PKG_VERSION"))) .default_headers(headers) + // A server that does not answer must not hang the command: 10 s to + // connect, and at most READ_TIMEOUT between two reads (longer than + // the 300 s an `encryption-rotate --wait-secs` may ask the server to + // wait before it answers). + .connect_timeout(std::time::Duration::from_secs(10)) + .read_timeout(READ_TIMEOUT) .build() .unwrap_or_default() } diff --git a/src/config.rs b/src/config.rs index 4f91292..ed74c25 100644 --- a/src/config.rs +++ b/src/config.rs @@ -3,7 +3,7 @@ use std::collections::HashMap; use std::env; use std::fs; -use std::path::PathBuf; +use std::path::{Path, PathBuf}; use serde::{Deserialize, Serialize}; @@ -143,7 +143,7 @@ impl Config { file_cfg.active_profile = name.to_string(); } - fs::write(&path, toml::to_string_pretty(&file_cfg)?)?; + write_private(&path, &toml::to_string_pretty(&file_cfg)?)?; Ok(()) } @@ -171,15 +171,175 @@ impl Config { } file_cfg.active_profile = name.to_string(); - fs::write(&path, toml::to_string_pretty(&file_cfg)?)?; + write_private(&path, &toml::to_string_pretty(&file_cfg)?)?; Ok(()) } } +/// Write the config file readable by its owner only (`0600`, in a `0700` +/// directory when this creates it): server URLs may carry credentials, and +/// the file is where a token would go. An existing file is narrowed to `0600` +/// as well. On non-Unix systems the file is written with default permissions. +pub fn write_private(path: &Path, contents: &str) -> anyhow::Result<()> { + #[cfg(unix)] + { + use std::io::Write; + use std::os::unix::fs::{DirBuilderExt, OpenOptionsExt, PermissionsExt}; + if let Some(parent) = path.parent() { + if !parent.exists() { + fs::DirBuilder::new() + .recursive(true) + .mode(0o700) + .create(parent)?; + } + } + let mut file = fs::OpenOptions::new() + .write(true) + .create(true) + .truncate(true) + .mode(0o600) + .open(path)?; + // `mode` applies only when the file is created. + file.set_permissions(fs::Permissions::from_mode(0o600))?; + file.write_all(contents.as_bytes())?; + Ok(()) + } + #[cfg(not(unix))] + { + if let Some(parent) = path.parent() { + fs::create_dir_all(parent)?; + } + fs::write(path, contents)?; + Ok(()) + } +} + +/// A warning when the config file at `path` can be read by other users +/// (the "other" read bit, or the group's), with the command that fixes it. +/// `None` when it cannot (or does not exist, or on non-Unix systems). +pub fn permission_warning(path: &Path) -> Option { + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + let mode = fs::metadata(path).ok()?.permissions().mode() & 0o777; + if mode & 0o044 == 0 { + return None; + } + Some(format!( + "{} is readable by other users (mode {mode:o}); run `chmod 600 {}`", + path.display(), + path.display() + )) + } + #[cfg(not(unix))] + { + let _ = path; + None + } +} + +/// Whether `url` sends requests in clear text to another machine: `http://` +/// to a host that is not the loopback interface. An API key sent there can be +/// read on the network. +pub fn is_cleartext_remote(url: &str) -> bool { + let Some(rest) = url.trim().strip_prefix("http://") else { + return false; + }; + let authority = rest.split(['/', '?', '#']).next().unwrap_or(""); + let host_port = authority.rsplit('@').next().unwrap_or(""); + let host = match host_port.strip_prefix('[') { + Some(v6) => v6.split(']').next().unwrap_or(""), + None => host_port.split(':').next().unwrap_or(""), + }; + let host = host.to_ascii_lowercase(); + let loopback = host == "localhost" + || host.ends_with(".localhost") + || host == "::1" + || host.starts_with("127."); + !loopback +} + +/// `url` with the password of any `user:password@` part replaced by `***`. +pub fn redact_url(url: &str) -> String { + let Some((scheme, rest)) = url.split_once("://") else { + return url.to_string(); + }; + let end = rest.find(['/', '?', '#']).unwrap_or(rest.len()); + let (authority, tail) = rest.split_at(end); + match authority.rsplit_once('@') { + Some((userinfo, host)) => { + let user = userinfo.split(':').next().unwrap_or(""); + let masked = if userinfo.contains(':') { + format!("{user}:***") + } else { + "***".to_string() + }; + format!("{scheme}://{masked}@{host}{tail}") + } + None => url.to_string(), + } +} + +/// An API key as `config show` prints it: its first 3 and last 4 characters, +/// or only `****` when it is too short to show any of it safely. +pub fn mask_key(key: &str) -> String { + let chars: Vec = key.chars().collect(); + if chars.len() < 24 { + return format!("**** ({} characters)", chars.len()); + } + let head: String = chars[..3].iter().collect(); + let tail: String = chars[chars.len() - 4..].iter().collect(); + format!("{head}…{tail} ({} characters)", chars.len()) +} + #[cfg(test)] mod tests { use super::*; + #[cfg(unix)] + #[test] + fn config_file_is_written_owner_only_and_narrowed() { + use std::os::unix::fs::PermissionsExt; + let dir = std::env::temp_dir().join(format!("dk-config-perm-{}", std::process::id())); + let path = dir.join("sub").join("config.toml"); + write_private(&path, "active_profile = \"default\"\n").unwrap(); + let mode = |p: &Path| fs::metadata(p).unwrap().permissions().mode() & 0o777; + assert_eq!(mode(&path), 0o600); + assert_eq!(mode(path.parent().unwrap()), 0o700); + assert!(permission_warning(&path).is_none()); + + fs::set_permissions(&path, fs::Permissions::from_mode(0o644)).unwrap(); + let warning = permission_warning(&path).unwrap(); + assert!(warning.contains("chmod 600"), "{warning}"); + write_private(&path, "x = 1\n").unwrap(); + assert_eq!(mode(&path), 0o600); + fs::remove_dir_all(&dir).unwrap(); + } + + #[test] + fn keys_and_url_passwords_are_masked() { + let key = "dk_live_abcdefghijklmnop1234"; + let masked = mask_key(key); + assert!(masked.starts_with("dk_…1234"), "{masked}"); + assert!(!masked.contains("abcdefgh")); + assert_eq!(mask_key("short"), "**** (5 characters)"); + assert_eq!( + redact_url("https://me:s3cret@db.example.com:3000/x"), + "https://me:***@db.example.com:3000/x" + ); + assert_eq!(redact_url("http://localhost:3000"), "http://localhost:3000"); + } + + #[test] + fn cleartext_remote_urls_are_told_apart() { + assert!(is_cleartext_remote("http://dakera.example.com:3000")); + assert!(is_cleartext_remote("http://10.1.2.3")); + assert!(!is_cleartext_remote("https://dakera.example.com")); + assert!(!is_cleartext_remote("http://localhost:3000")); + assert!(!is_cleartext_remote("http://127.0.0.1:3000/")); + assert!(!is_cleartext_remote("http://[::1]:3000")); + } + #[test] fn test_default_config() { let config = Config::default(); diff --git a/src/error.rs b/src/error.rs index 4fdf2ff..73a34bc 100644 --- a/src/error.rs +++ b/src/error.rs @@ -13,7 +13,8 @@ use crate::api::ApiHttpError; /// - 3 not found (namespace, vector, key, etc.) /// - 4 permission denied / authentication failure /// - 5 invalid input / validation error -/// - 6 server-side error (5xx, including 501 feature disabled / 503 busy) +/// - 6 server-side error (5xx, including 501 feature disabled / 503 busy), or +/// the server is overloaded (429 since Dakera v0.12.3: retry after `Retry-After`) /// /// A `413` (payload too large, hard quota exceeded) is an input error (5); a /// `403` is a permission error (4), whether the key's scope is too low or the @@ -131,9 +132,10 @@ pub fn classify(err: &anyhow::Error) -> CliError { return by_status(*status, msg) } E::QuotaExceeded { .. } | E::PayloadTooLarge { .. } => return CliError::Input(msg), - E::FeatureDisabled { .. } | E::NotImplemented { .. } | E::ServiceUnavailable { .. } => { - return CliError::Server(msg) - } + E::FeatureDisabled { .. } + | E::NotImplemented { .. } + | E::ServiceUnavailable { .. } + | E::RateLimitExceeded { .. } => return CliError::Server(msg), E::NamespaceNotFound(_) | E::VectorNotFound(_) => return CliError::NotFound(msg), E::InvalidRequest(_) => return CliError::Input(msg), E::Connection(_) | E::Timeout => return CliError::Connection(msg), @@ -179,7 +181,10 @@ fn by_status(status: u16, msg: String) -> CliError { 401 | 403 => CliError::Permission(msg), 404 => CliError::NotFound(msg), 400 | 409 | 413 | 415 | 422 => CliError::Input(msg), - 500..=599 => CliError::Server(msg), + // Since v0.12.3 an overloaded server (a full inference or recall + // queue) answers 429 + Retry-After where it answered 503: the same + // "busy, retry later" class. A namespace rate limit is 429 too. + 429 | 500..=599 => CliError::Server(msg), _ => CliError::Other(msg), } } @@ -306,6 +311,26 @@ mod tests { assert!(s.contains("\"server_code\":\"SERVICE_UNAVAILABLE\"")); } + /// v0.12.3: an overloaded server answers 429 + Retry-After (it was 503). + #[test] + fn test_classify_http_429_overload_is_server_busy_and_keeps_retry_after() { + let err = http_err(429, Some(5), "RATE_LIMIT_EXCEEDED"); + let cli_err = classify(&err); + assert!(matches!(cli_err, CliError::Server(_))); + assert_eq!(cli_err.exit_code(), 6); + let s = serde_json::to_string(&JsonError::from_error(&cli_err, &err)).unwrap(); + assert!(s.contains("\"retry_after_secs\":5")); + assert!(s.contains("\"http_status\":429")); + } + + #[test] + fn test_classify_sdk_rate_limit_is_server_busy() { + let err = anyhow::Error::new(dakera_client::ClientError::RateLimitExceeded { + retry_after: Some(5), + }); + assert_eq!(classify(&err).exit_code(), 6); + } + #[test] fn test_classify_input_error_keeps_exit_code_5() { let err = input_error("no such file"); diff --git a/src/main.rs b/src/main.rs index 4157946..e89a991 100644 --- a/src/main.rs +++ b/src/main.rs @@ -14,8 +14,8 @@ use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt}; use crate::cli::build_cli; use crate::commands::{ - admin, agent, attachment, capabilities, completion, config as config_cmd, health, index, init, - keys, knowledge, memory, namespace, session, text, + admin, agent, attachment, capabilities, completion, config as config_cmd, document, health, + index, init, keys, knowledge, memory, namespace, session, text, }; use crate::config::Config; use crate::context::Context; @@ -77,6 +77,25 @@ async fn main() { } } +/// Warnings (on stderr, so `--format json` output stays clean) about where +/// the API key could be read: a config file other users can read, and a key +/// that would cross the network in clear text. +fn warn_about_exposure(url: &str) { + if let Some(path) = Config::config_path() { + if let Some(warning) = config::permission_warning(&path) { + eprintln!("warning: {warning}"); + } + } + let has_key = std::env::var("DAKERA_API_KEY").is_ok_and(|k| !k.is_empty()); + if has_key && config::is_cleartext_remote(url) { + eprintln!( + "warning: {} is plain http:// to a remote host; DAKERA_API_KEY is sent unencrypted \ + (use https://)", + config::redact_url(url) + ); + } +} + async fn run(matches: clap::ArgMatches, format: OutputFormat, verbose: bool) -> anyhow::Result<()> { let config = match matches.get_one::("profile") { Some(p) => Config::load_with_profile(p), @@ -90,6 +109,7 @@ async fn run(matches: clap::ArgMatches, format: OutputFormat, verbose: bool) -> config.server_url.clone() }; + warn_about_exposure(&url); let ctx = Context::new(url, format, verbose); match matches.subcommand() { @@ -98,6 +118,7 @@ async fn run(matches: clap::ArgMatches, format: OutputFormat, verbose: bool) -> Some(("capabilities", _)) => capabilities::execute(&ctx).await?, Some(("whoami", _)) => keys::whoami(&ctx).await?, Some(("attachment", sub_matches)) => attachment::execute(&ctx, sub_matches).await?, + Some(("document", sub_matches)) => document::execute(&ctx, sub_matches).await?, Some(("namespace", sub_matches)) => namespace::execute(&ctx, sub_matches).await?, Some(("index", sub_matches)) => index::execute(&ctx, sub_matches).await?, Some(("memory", sub_matches)) => memory::execute(&ctx, sub_matches).await?, diff --git a/tests/integration.rs b/tests/integration.rs index 34f2121..9e11a45 100644 --- a/tests/integration.rs +++ b/tests/integration.rs @@ -1913,3 +1913,221 @@ fn memory_batch_forget_dry_run_shows_preview() { .success() .stdout(predicate::str::contains("dry-run")); } + +// --------------------------------------------------------------------------- +// Container — Dakera v0.12.3 (documents, recall ranking, empty agents) +// Skipped against an older server: CI's container may be an earlier release. +// --------------------------------------------------------------------------- + +/// The `--format json` output of a `dk` run, which must succeed. +fn container_json(url: &str, key: &str, args: &[&str]) -> serde_json::Value { + let out = container_dk(url, key) + .args(["--format", "json"]) + .args(args) + .output() + .expect("dk runs"); + assert!( + out.status.success(), + "dk {args:?} failed: {}", + String::from_utf8_lossy(&out.stderr) + ); + serde_json::from_slice(&out.stdout) + .unwrap_or_else(|e| panic!("dk {args:?} printed no JSON ({e})")) +} + +/// Whether the server is v0.12.3 or newer (from `dk health`). +fn server_is_v0123(url: &str, key: &str) -> bool { + let health = container_json(url, key, &["health"]); + let version = health["version"].as_str().unwrap_or("0.0.0"); + let parts: Vec = version + .split(['.', '-']) + .take(3) + .map(|p| p.parse().unwrap_or(0)) + .collect(); + let ok = parts >= vec![0, 12, 3]; + if !ok { + eprintln!("skipped: the server is v{version}, this test needs v0.12.3"); + } + ok +} + +/// A file under the temp dir, unique to this test run. +fn temp_file(name: &str, contents: &[u8]) -> std::path::PathBuf { + let path = std::env::temp_dir().join(format!("dk-it-{}-{name}", std::process::id())); + std::fs::write(&path, contents).unwrap(); + path +} + +#[test] +#[ignore = "requires running dakera container (set DAKERA_TEST_URL)"] +fn container_v0123_document_import_cite_and_delete() { + let url = container_url(); + let key = container_key(); + if !server_is_v0123(&url, &key) { + return; + } + let agent = format!("it-doc-{}", std::process::id()); + let file = temp_file( + "manual.md", + b"# Field Manual\n\n## Install\n\nRun the installer and accept the licence. \ + The service listens on port 8080.\n\n## Upgrade\n\nStop the service first, \ + then replace the binary.\n", + ); + + let imported = container_json( + &url, + &key, + &[ + "document", + "import", + file.to_str().unwrap(), + "--agent", + &agent, + "--title", + "Field Manual", + "--wait", + ], + ); + std::fs::remove_file(&file).ok(); + let status = imported["job"]["status"].as_str().unwrap_or(""); + assert!(status.eq_ignore_ascii_case("completed"), "{imported}"); + let document_id = imported["accepted"]["document_id"] + .as_str() + .expect("document_id") + .to_string(); + + let listed = container_json(&url, &key, &["document", "list", "--agent", &agent]); + let docs = listed["documents"].as_array().unwrap(); + assert!(docs + .iter() + .any(|d| d["document_id"] == document_id.as_str())); + + let shown = container_json( + &url, + &key, + &["document", "show", &document_id, "--agent", &agent], + ); + assert!(shown.to_string().contains("Install"), "{shown}"); + + // The ranked result cites its document; its neighbour chunk is context, + // outside --top-k. + let recalled = container_json( + &url, + &key, + &[ + "memory", + "recall", + &agent, + "which port does the service listen on", + "--top-k", + "1", + ], + ); + let memories = recalled["memories"].as_array().unwrap(); + assert_eq!(memories.len(), 1, "{recalled}"); + assert_eq!(memories[0]["document"]["title"], "Field Manual"); + assert_eq!(memories[0]["document"]["document_id"], document_id.as_str()); + for c in recalled["context_memories"].as_array().unwrap() { + assert_eq!(c["context_of"], memories[0]["id"]); + } + + container_dk(&url, &key) + .args(["document", "delete", &document_id, "--agent", &agent]) + .assert() + .success(); + let listed = container_json(&url, &key, &["document", "list", "--agent", &agent]); + assert!(!listed.to_string().contains(&document_id)); +} + +#[test] +#[ignore = "requires running dakera container (set DAKERA_TEST_URL)"] +fn container_v0123_pdf_import_needs_attachments() { + let url = container_url(); + let key = container_key(); + if !server_is_v0123(&url, &key) { + return; + } + let caps = container_json(&url, &key, &["capabilities"]); + if caps["attachments"]["enabled"] != false { + eprintln!("skipped: attachments are enabled on this server"); + return; + } + let file = temp_file("x.pdf", b"%PDF-1.4 not really a pdf"); + let out = container_dk(&url, &key) + .args(["document", "import", file.to_str().unwrap()]) + .args(["--agent", "it-doc-pdf"]) + .output() + .unwrap(); + std::fs::remove_file(&file).ok(); + assert_eq!(out.status.code(), Some(6)); + let stderr = String::from_utf8_lossy(&out.stderr); + assert!(stderr.contains("DAKERA_ATTACHMENTS"), "{stderr}"); +} + +#[test] +#[ignore = "requires running dakera container (set DAKERA_TEST_URL)"] +fn container_v0123_recall_scores_follow_the_order() { + let url = container_url(); + let key = container_key(); + if !server_is_v0123(&url, &key) { + return; + } + let agent = format!("it-rank-{}", std::process::id()); + for text in [ + "Maya moved to Lisbon in 2021 for a job at a bakery.", + "Maya's sister lives in Porto and visits every summer.", + "The bakery in Lisbon opens at six in the morning.", + "Maya started learning the cello last winter.", + "Lisbon trams are crowded on weekday mornings.", + ] { + container_dk(&url, &key) + .args(["memory", "store", &agent, text]) + .assert() + .success(); + } + let recalled = container_json( + &url, + &key, + &[ + "memory", + "recall", + &agent, + "Where does Maya live?", + "--top-k", + "5", + ], + ); + let scores: Vec = recalled["memories"] + .as_array() + .unwrap() + .iter() + .map(|m| m["score"].as_f64().unwrap()) + .collect(); + assert!(!scores.is_empty()); + assert!( + scores.windows(2).all(|w| w[0] >= w[1]), + "listed scores must follow the order: {scores:?}" + ); +} + +#[test] +#[ignore = "requires running dakera container (set DAKERA_TEST_URL)"] +fn container_v0123_search_and_batch_recall_of_an_empty_agent() { + let url = container_url(); + let key = container_key(); + if !server_is_v0123(&url, &key) { + return; + } + let agent = format!("it-empty-{}", std::process::id()); + let recalled = container_json(&url, &key, &["memory", "recall", &agent, "anything"]); + assert_eq!(recalled["memories"], json!([])); + container_dk(&url, &key) + .args(["memory", "search", &agent, "anything"]) + .assert() + .success() + .stdout(predicate::str::contains("No memories found")); + container_dk(&url, &key) + .args(["memory", "batch-recall", &agent]) + .assert() + .success(); +} diff --git a/tests/route_audit.rs b/tests/route_audit.rs index 527aec6..4a0c43b 100644 --- a/tests/route_audit.rs +++ b/tests/route_audit.rs @@ -1,5 +1,5 @@ //! Every server path `dk` calls must be a route of the Dakera v0.12 server -//! (v0.12.0 plus the routes v0.12.2 adds). +//! (v0.12.0 plus the routes v0.12.2 and v0.12.3 add). //! //! The route list (`tests/server_routes_v0.12.txt`) comes from the server's //! router (`crates/api/src/lib.rs`); the paths `dk` calls are the string diff --git a/tests/server_routes_v0.12.txt b/tests/server_routes_v0.12.txt index 9993bac..c2f118f 100644 --- a/tests/server_routes_v0.12.txt +++ b/tests/server_routes_v0.12.txt @@ -1,4 +1,4 @@ -# Routes of the Dakera v0.12.0 server (plus the v0.12.2 routes) (crates/api/src/lib.rs + keys_router + attachment routes), params as *. +# Routes of the Dakera v0.12.3 server (v0.12.0 plus the routes v0.12.2 and v0.12.3 add) (crates/api/src/lib.rs + keys_router + attachment routes), params as *. # Admin routes are served under /admin and /v1/admin. Regenerate when the server's router changes. DELETE /admin/backups/* DELETE /admin/keys/* @@ -295,3 +295,13 @@ POST /admin/derivations/drain POST /v1/admin/derivations/drain POST /v1/agents POST /v1/sessions/*/touch +# Added in v0.12.3 (document import, Dakera-AI/dakera#923) +DELETE /v1/memory/documents/* +GET /v1/memory/documents +GET /v1/memory/documents/* +GET /v1/memory/documents/jobs/* +POST /v1/memory/documents +POST /v1/namespaces/*/attachments/*/document +# Added in v0.12.3: memories removed by deduplication or consolidation, restorable +GET /v1/memory/deduplicated +POST /v1/memory/deduplicated/*/restore diff --git a/tests/v012.rs b/tests/v012.rs index 0034af6..346a4a0 100644 --- a/tests/v012.rs +++ b/tests/v012.rs @@ -1620,7 +1620,8 @@ fn memory_recall_sends_lang() { "top_k": 3, "min_importance": 0.0, "tags": [], - "lang": "fr" + "lang": "fr", + "neighborhood_mode": "separate" })); then.status(200) .json_body(json!({"memories": [], "total_found": 0})); diff --git a/tests/v0123.rs b/tests/v0123.rs new file mode 100644 index 0000000..e064944 --- /dev/null +++ b/tests/v0123.rs @@ -0,0 +1,710 @@ +//! Integration tests for the recall and hardening changes that follow the +//! Dakera engine's recall API correctness work (Dakera-AI/dakera#917) and +//! security hardening (Dakera-AI/dakera#920). +//! +//! Like `v012.rs`, each test starts a local [`httpmock`] server and runs the +//! compiled `dk` binary against it; no Dakera server is needed. +//! +//! Covered: `memory recall` asking for `neighborhood_mode` (default +//! `separate`) and `include_summaries`, the ranked table honouring `--top-k` +//! with the neighbourhood in its own Context section, `matched_text`, the JSON +//! output, an older server that appends its context inline; the `422` for an +//! unknown admin field; `config show` never printing the key; the config file +//! permissions warning. + +use assert_cmd::Command; +use httpmock::prelude::*; +use predicates::prelude::*; +use serde_json::{json, Value}; + +fn dk() -> Command { + let mut cmd = Command::cargo_bin("dk").expect("dk binary not found — run `cargo build` first"); + cmd.env_remove("DAKERA_API_KEY"); + cmd +} + +fn item(id: &str, content: &str, score: f64) -> Value { + json!({ + "memory": {"id": id, "content": content, "memory_type": "episodic", "importance": 0.5}, + "score": score + }) +} + +// --------------------------------------------------------------------------- +// memory recall +// --------------------------------------------------------------------------- + +#[test] +fn recall_asks_for_separate_context_by_default() { + let server = MockServer::start(); + let m = server.mock(|when, then| { + when.method(POST) + .path("/v1/memory/recall") + .json_body(json!({ + "agent_id": "bot", + "query": "port", + "top_k": 2, + "min_importance": 0.0, + "tags": [], + "neighborhood_mode": "separate" + })); + then.status(200).json_body(json!({"memories": []})); + }); + dk().args([ + "--url", + &server.base_url(), + "memory", + "recall", + "bot", + "port", + "-k", + "2", + ]) + .assert() + .success() + .stdout(predicate::str::contains("No memories found")); + m.assert(); +} + +#[test] +fn recall_off_and_include_summaries_are_sent() { + let server = MockServer::start(); + let m = server.mock(|when, then| { + when.method(POST) + .path("/v1/memory/recall") + .json_body_includes(r#"{"neighborhood_mode": "off", "neighborhood": false}"#) + .json_body_includes(r#"{"include_summaries": false}"#); + then.status(200).json_body(json!({"memories": []})); + }); + dk().args([ + "--url", + &server.base_url(), + "memory", + "recall", + "bot", + "q", + "--neighborhood", + "off", + "--include-summaries=false", + ]) + .assert() + .success(); + m.assert(); + + let m = server.mock(|when, then| { + when.method(POST) + .path("/v1/memory/recall") + .json_body_includes(r#"{"include_summaries": true}"#) + .json_body_includes(r#"{"neighborhood_mode": "inline"}"#); + then.status(200).json_body(json!({"memories": []})); + }); + dk().args([ + "--url", + &server.base_url(), + "memory", + "recall", + "bot", + "q", + "--neighborhood", + "inline", + "--include-summaries", + ]) + .assert() + .success(); + m.assert(); +} + +#[test] +fn recall_rejects_an_unknown_neighborhood_mode() { + dk().args([ + "--url", + "http://127.0.0.1:1", + "memory", + "recall", + "bot", + "q", + "--neighborhood", + "sideways", + ]) + .assert() + .failure() + .stderr(predicate::str::contains("separate")); +} + +fn separate_answer() -> Value { + let mut collapsed = item("m1", "Deploy notes for the API", 0.9); + collapsed["matched_text"] = json!("The API now listens on port 9000."); + collapsed["matched_ids"] = json!(["m1-s3"]); + let mut neighbour = item("n1", "We moved it last Tuesday.", 0.41); + neighbour["retrieval"] = json!("neighborhood"); + neighbour["context_of"] = json!("m2"); + json!({ + "memories": [collapsed, item("m2", "Port change approved", 0.8)], + "context_memories": [neighbour], + "query_embedding_time_ms": 1, + "search_time_ms": 2 + }) +} + +#[test] +fn recall_table_shows_ranked_results_then_a_context_section() { + let server = MockServer::start(); + server.mock(|when, then| { + when.method(POST).path("/v1/memory/recall"); + then.status(200).json_body(separate_answer()); + }); + let out = dk() + .args([ + "--url", + &server.base_url(), + "memory", + "recall", + "bot", + "port", + "-k", + "2", + ]) + .assert() + .success() + .get_output() + .stdout + .clone(); + let out = String::from_utf8(out).unwrap(); + assert!(out.contains("Found 2 memories"), "{out}"); + assert!(out.contains("MATCHED_TEXT"), "{out}"); + assert!(out.contains("port 9000"), "{out}"); + let context_at = out.find("Context: 1 session neighbour").expect(&out); + let neighbour_at = out.find("We moved it last Tuesday.").expect(&out); + assert!(neighbour_at > context_at, "context printed apart: {out}"); + assert!(out.contains("#2 m2"), "context_of with its rank: {out}"); +} + +#[test] +fn recall_json_is_one_document_with_both_lists() { + let server = MockServer::start(); + server.mock(|when, then| { + when.method(POST).path("/v1/memory/recall"); + then.status(200).json_body(separate_answer()); + }); + let out = dk() + .args([ + "--url", + &server.base_url(), + "--format", + "json", + "memory", + "recall", + "bot", + "port", + "-k", + "2", + ]) + .assert() + .success() + .get_output() + .stdout + .clone(); + let doc: Value = serde_json::from_slice(&out).expect("stdout is JSON only"); + assert_eq!(doc["memories"].as_array().unwrap().len(), 2); + assert_eq!( + doc["memories"][0]["matched_text"], + "The API now listens on port 9000." + ); + assert_eq!(doc["memories"][0]["matched_ids"], json!(["m1-s3"])); + assert!(doc["memories"][1].get("matched_text").is_none()); + assert_eq!(doc["context_memories"][0]["id"], "n1"); + assert_eq!(doc["context_memories"][0]["context_of"], "m2"); + assert_eq!(doc["context_memories"][0]["context_of_rank"], 2); +} + +#[test] +fn recall_from_an_older_server_keeps_to_top_k() { + // A v0.12.2 server ignores neighborhood_mode and appends its context. + let server = MockServer::start(); + server.mock(|when, then| { + when.method(POST).path("/v1/memory/recall"); + then.status(200).json_body(json!({ + "memories": [ + item("m1", "first", 0.9), + item("m2", "second", 0.8), + item("n1", "neighbour one", 0.3), + item("n2", "neighbour two", 0.2) + ] + })); + }); + let assert = dk() + .args([ + "--url", + &server.base_url(), + "--format", + "json", + "memory", + "recall", + "bot", + "q", + "-k", + "2", + ]) + .assert() + .success() + .stderr(predicate::str::contains( + "2 neighbourhood context item(s) left out", + )); + let doc: Value = serde_json::from_slice(&assert.get_output().stdout).unwrap(); + let ids: Vec<&str> = doc["memories"] + .as_array() + .unwrap() + .iter() + .map(|m| m["id"].as_str().unwrap()) + .collect(); + assert_eq!(ids, ["m1", "m2"]); + assert_eq!(doc["context_memories"], json!([])); + + // Asked for inline, the overflow is the context the user wanted. + dk().args([ + "--url", + &server.base_url(), + "memory", + "recall", + "bot", + "q", + "-k", + "2", + "--neighborhood", + "inline", + ]) + .assert() + .success() + .stdout(predicate::str::contains("Found 2 memories")) + .stdout(predicate::str::contains("Context: 2 session neighbour")) + .stdout(predicate::str::contains("neighbour two")); +} + +// --------------------------------------------------------------------------- +// Strict admin routes and the key +// --------------------------------------------------------------------------- + +#[test] +fn key_create_sends_only_known_fields_and_explains_a_422() { + let server = MockServer::start(); + let m = server.mock(|when, then| { + when.method(POST).path("/admin/keys").json_body(json!({ + "name": "ci", "scope": "read", "expires_in_days": 7 + })); + then.status(422).json_body(json!({ + "error": "Failed to deserialize the JSON body into the target type: unknown field \ + `expires_in_days`, expected one of `name`, `scope`", + "code": "INVALID_REQUEST", + "status": 422 + })); + }); + dk().args([ + "--url", + &server.base_url(), + "keys", + "create", + "ci", + "--expires", + "7", + ]) + .assert() + .failure() + .stderr(predicate::str::contains( + "does not accept the field `expires_in_days`", + )); + m.assert(); +} + +#[test] +fn key_ids_are_encoded_in_paths() { + let server = MockServer::start(); + let m = server.mock(|when, then| { + when.method(GET).path("/admin/keys/a%2Fb"); + then.status(200).json_body(json!({"key_id": "a/b"})); + }); + dk().args(["--url", &server.base_url(), "keys", "get", "a/b"]) + .assert() + .success(); + m.assert(); +} + +#[test] +fn config_show_masks_the_key() { + let home = std::env::temp_dir().join(format!("dk-v0123-home-{}", std::process::id())); + std::fs::create_dir_all(&home).unwrap(); + let secret = "dk_live_0123456789abcdefSECRET9876"; + let assert = dk() + .env("HOME", &home) + .env("DAKERA_API_KEY", secret) + .env("DAKERA_URL", "https://dakera.example.com") + .args(["config"]) + .assert() + .success() + .stdout(predicate::str::contains("dk_…9876")) + .stdout(predicate::str::contains("TLS (https)")); + let out = assert.get_output(); + let all = format!( + "{}{}", + String::from_utf8_lossy(&out.stdout), + String::from_utf8_lossy(&out.stderr) + ); + assert!(!all.contains("0123456789abcdef"), "{all}"); + std::fs::remove_dir_all(&home).unwrap(); +} + +#[cfg(unix)] +#[test] +fn profile_add_writes_an_owner_only_file_and_a_readable_one_is_flagged() { + use std::os::unix::fs::PermissionsExt; + let home = std::env::temp_dir().join(format!("dk-v0123-perm-{}", std::process::id())); + std::fs::create_dir_all(&home).unwrap(); + dk().env("HOME", &home) + .args([ + "config", + "profile", + "add", + "prod", + "--url", + "https://d.example.com", + ]) + .assert() + .success(); + let path = home.join(".dakera").join("config.toml"); + let mode = std::fs::metadata(&path).unwrap().permissions().mode() & 0o777; + assert_eq!(mode, 0o600); + + std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o644)).unwrap(); + dk().env("HOME", &home) + .args(["config"]) + .assert() + .success() + .stderr(predicate::str::contains("readable by other users")); + std::fs::remove_dir_all(&home).unwrap(); +} + +#[test] +fn a_key_over_plain_http_to_a_remote_host_is_warned_about() { + dk().env("DAKERA_API_KEY", "dk_live_abcdefghijklmnop") + .args(["--url", "http://192.0.2.1:9", "config"]) + .assert() + .success() + .stderr(predicate::str::contains("sent unencrypted")); +} + +// --------------------------------------------------------------------------- +// dk document (Dakera v0.12.3, Dakera-AI/dakera#923) +// --------------------------------------------------------------------------- + +fn scratch_file(name: &str, bytes: &[u8]) -> std::path::PathBuf { + let dir = std::env::temp_dir().join(format!("dk-v0123-doc-{}", std::process::id())); + std::fs::create_dir_all(&dir).unwrap(); + let path = dir.join(name); + std::fs::write(&path, bytes).unwrap(); + path +} + +fn accepted(document_id: &str) -> Value { + json!({ + "job_id": "job_7_1", "document_id": document_id, "agent_id": "bot", + "status_url": "/v1/memory/documents/jobs/job_7_1?agent_id=bot" + }) +} + +#[test] +fn document_import_markdown_goes_to_the_text_route_and_waits() { + let server = MockServer::start(); + let file = scratch_file("guide.md", b"# Guide\n\nHello."); + let import = server.mock(|when, then| { + when.method(POST) + .path("/v1/memory/documents") + .json_body(json!({ + "agent_id": "bot", + "content": "# Guide\n\nHello.", + "media_type": "text/markdown", + "title": "Guide", + "tags": ["docs", "v1"], + "chunking": {"max_tokens": 300, "overlap_tokens": 30, "strategy": "fixed"} + })); + then.status(202).json_body(accepted("guide")); + }); + let job = server.mock(|when, then| { + when.method(GET) + .path("/v1/memory/documents/jobs/job_7_1") + .query_param("agent_id", "bot"); + then.status(200).json_body(json!({ + "id": "job_7_1", "job_type": "document_import", "status": "Completed", + "progress": 100, "message": "document imported: 2 chunks (2 written, 0 unchanged, 0 stale removed); chunks_stored 2/2", + "metadata": {"document_id": "guide"} + })); + }); + dk().args(["--url", &server.base_url(), "document", "import"]) + .arg(&file) + .args([ + "--agent", + "bot", + "--title", + "Guide", + "--tag", + "docs", + "--tag", + "v1", + "--chunk-tokens", + "300", + "--overlap", + "30", + "--strategy", + "fixed", + "--wait", + ]) + .assert() + .success() + .stdout(predicate::str::contains("Document 'guide' imported")) + .stdout(predicate::str::contains("2 chunks")); + import.assert(); + job.assert(); +} + +#[test] +fn document_import_pdf_uploads_then_imports_the_attachment() { + let server = MockServer::start(); + let file = scratch_file("manual.pdf", b"%PDF-1.7 test"); + let upload = server.mock(|when, then| { + when.method(POST) + .path("/v1/namespaces/_dakera_agent_bot/attachments") + .header("content-type", "application/pdf") + .body("%PDF-1.7 test"); + then.status(201) + .json_body(json!({"attachment_ref": "sha256:ab12", "size_bytes": 13})); + }); + let import = server.mock(|when, then| { + when.method(POST) + .path("/v1/namespaces/_dakera_agent_bot/attachments/sha256:ab12/document") + .json_body(json!({"agent_id": "bot", "document_id": "manual-v3", "decompose": true})); + then.status(202).json_body(accepted("manual-v3")); + }); + dk().args([ + "--url", + &server.base_url(), + "-f", + "json", + "document", + "import", + ]) + .arg(&file) + .args([ + "--agent", + "bot", + "--document-id", + "manual-v3", + "--decompose", + ]) + .assert() + .success() + .stdout(predicate::str::contains("\"job_id\": \"job_7_1\"")); + upload.assert(); + import.assert(); +} + +#[test] +fn document_import_needs_a_known_format() { + let file = scratch_file("data.bin", b"\x00\x01"); + dk().args(["--url", "http://127.0.0.1:1", "document", "import"]) + .arg(&file) + .args(["--agent", "bot"]) + .assert() + .code(5) + .stderr(predicate::str::contains("--media-type")); +} + +#[test] +fn document_import_reports_a_failed_job() { + let server = MockServer::start(); + let file = scratch_file("bad.html", b""); + server.mock(|when, then| { + when.method(POST).path("/v1/memory/documents"); + then.status(202).json_body(accepted("bad")); + }); + server.mock(|when, then| { + when.method(GET).path("/v1/memory/documents/jobs/job_7_1"); + then.status(200).json_body(json!({ + "id": "job_7_1", "status": "Failed", "progress": 2, + "message": "HTML nested deeper than 512 block elements", + "metadata": {}, "error": {"status": 422, "code": "UNPROCESSABLE_CONTENT"} + })); + }); + dk().args(["--url", &server.base_url(), "document", "import"]) + .arg(&file) + .args(["--agent", "bot", "--wait"]) + .assert() + .failure() + .stderr(predicate::str::contains("422 UNPROCESSABLE_CONTENT")); +} + +#[test] +fn document_routes_on_an_older_server_say_what_is_needed() { + let server = MockServer::start(); + server.mock(|when, then| { + when.method(GET).path("/v1/memory/documents"); + then.status(404).json_body(json!({ + "error": "No route for GET /v1/memory/documents", "code": "ROUTE_NOT_FOUND", "status": 404 + })); + }); + dk().args([ + "--url", + &server.base_url(), + "document", + "list", + "--agent", + "bot", + ]) + .assert() + .failure() + .stderr(predicate::str::contains("needs a Dakera v0.12.3 server")); +} + +#[test] +fn document_list_show_job_and_delete() { + let server = MockServer::start(); + server.mock(|when, then| { + when.method(GET) + .path("/v1/memory/documents") + .query_param("agent_id", "bot") + .query_param("limit", "1"); + then.status(200).json_body(json!({ + "documents": [{ + "document_id": "manual-v3", "title": "Field Manual", "media_type": "application/pdf", + "chunk_count": 212, "bytes": 1843311, "pages": 212, "created_at": 1, "updated_at": 2, + "sha256": "ab" + }], + "next_cursor": "manual-v3" + })); + }); + dk().args([ + "--url", + &server.base_url(), + "document", + "list", + "--agent", + "bot", + "--limit", + "1", + ]) + .assert() + .success() + .stdout(predicate::str::contains("Field Manual")) + .stdout(predicate::str::contains("--cursor manual-v3")); + + server.mock(|when, then| { + when.method(GET).path("/v1/memory/documents/manual-v3"); + then.status(200).json_body(json!({ + "document": {"document_id": "manual-v3", "title": "Field Manual", + "media_type": "application/pdf", "chunk_count": 1, "bytes": 9, + "created_at": 1, "updated_at": 2, "sha256": "ab"}, + "chunks": [{"id": "doc_x", "chunk_index": 0, "page_start": 3, "page_end": 4, + "heading_path": ["Field Manual", "Install"], "char_start": 0, "char_end": 80}] + })); + }); + dk().args([ + "--url", + &server.base_url(), + "document", + "show", + "manual-v3", + "--agent", + "bot", + ]) + .assert() + .success() + .stdout(predicate::str::contains("Field Manual › Install")) + .stdout(predicate::str::contains("3-4")); + + server.mock(|when, then| { + when.method(GET).path("/v1/memory/documents/jobs/job_9"); + then.status(200).json_body(json!({ + "id": "job_9", "status": "Running", "progress": 40, + "message": "chunks_stored 32/80", "metadata": {"document_id": "manual-v3"} + })); + }); + dk().args([ + "--url", + &server.base_url(), + "document", + "job", + "job_9", + "--agent", + "bot", + ]) + .assert() + .success() + .stdout(predicate::str::contains("Running")) + .stdout(predicate::str::contains("40%")); + + let del = server.mock(|when, then| { + when.method(DELETE) + .path("/v1/memory/documents/manual-v3") + .query_param("agent_id", "bot"); + then.status(200) + .json_body(json!({"document_id": "manual-v3", "deleted_chunks": 212})); + }); + dk().args([ + "--url", + &server.base_url(), + "document", + "delete", + "manual-v3", + "--agent", + "bot", + ]) + .assert() + .success() + .stdout(predicate::str::contains("212 chunk memories")); + del.assert(); +} + +#[test] +fn recall_cites_document_chunks() { + let server = MockServer::start(); + let mut chunk = item("doc_1", "Field Manual › Install\n\nRun the installer.", 0.9); + chunk["memory"]["metadata"] = json!({"_dakera_document": { + "id": "manual-v3", "title": "Field Manual", "media_type": "application/pdf", + "chunk_index": 4, "chunk_count": 9, "page_start": 39, "page_end": 40, + "heading_path": ["Field Manual", "Install"], "char_start": 0, "char_end": 9, "sha256": "ab" + }}); + let answer = json!({"memories": [chunk, item("m2", "unrelated", 0.4)]}); + server.mock(|when, then| { + when.method(POST).path("/v1/memory/recall"); + then.status(200).json_body(answer); + }); + dk().args([ + "--url", + &server.base_url(), + "memory", + "recall", + "bot", + "install", + ]) + .assert() + .success() + .stdout(predicate::str::contains("SOURCE")) + .stdout(predicate::str::contains("Field Manual, p. 39-40 › Install")); + let out = dk() + .args([ + "--url", + &server.base_url(), + "-f", + "json", + "memory", + "recall", + "bot", + "install", + ]) + .assert() + .success() + .get_output() + .stdout + .clone(); + let doc: Value = serde_json::from_slice(&out).unwrap(); + assert_eq!(doc["memories"][0]["document"]["document_id"], "manual-v3"); + assert_eq!(doc["memories"][0]["document"]["page_start"], 39); + assert!(doc["memories"][1].get("document").is_none()); +}