feat(spec): korg-ledger@v1 rev 1.1 — canonical flight-recorder contract

.github/workflows/ci.yml

@@ -113,6 +113,9 @@ jobs:
         with:
           node-version: "20"
 
+      - name: Conformance vectors match the generator (no drift, frozen v1.0 tips)
+        run: python3 spec/korg-ledger-v1/tools/generate_vectors.py
+
       - name: Python spec oracle reproduces the frozen vectors
         run: python3 spec/korg-ledger-v1/conformance.py
 

CHANGELOG.md

@@ -16,7 +16,42 @@ versioning follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
 
-_Nothing yet._
+### korg-ledger@v1 revision 1.1 — the canonical flight-recorder contract
+
+`spec/korg-ledger-v1/` is now **authoritative** for every korg document and
+implementation. No hash rule changed — every v1.0 vector reproduces its frozen
+tip. Upgrade notes: [`spec/korg-ledger-v1/MIGRATION.md`](spec/korg-ledger-v1/MIGRATION.md).
+
+**Fixed — causal links in real journals were never checked.** All three
+verifiers read `triggered_by` only at the top level, but every production writer
+stores it under `metadata.triggered_by`, so forward or dangling links in nested
+journals verified VALID. Rust, Python, and JS now resolve the parent from either
+shape (SPEC §9), reject non-integer and conflicting links, and emit
+byte-identical DAG errors (`seq Some(3)` in Rust is gone). Verdict splits on
+`triggered_by: true` (Python-only) and `1.5` (JS-only) are closed.
+
+**Added — redaction at persistence (`korg-redact@v1`, REDACTION.md).** Secrets
+(sensitive keys, bearer tokens, provider/GitHub/Slack/AWS/Google keys, JWTs, PEM
+private keys, URL credentials, `NAME_TOKEN=…` assignments) are replaced by
+`[REDACTED:<rule>]` markers *before* hashing in `LedgerWriter` (default on),
+`POST /api/agent/tool-call`, and korg-bridge. Python and Rust implementations are
+pinned by shared hand-authored cases and differentially fuzzed.
+`korg-verify --require-redacted` and `python -m korg_ledger lint` flag any
+persisted secret; `python -m korg_ledger rechain-redacted` migrates a leaked
+journal to a new, clean chain.
+
+**Added — import envelopes (`korg-import@v1`, IMPORT.md).** Allowlisted,
+atomic imports for sanitized Okto diagnostics (`okto.diagnostics@v1`) and
+ningen-shikkaku lifecycle metadata (`ningen-shikkaku.lifecycle@v1`), with
+projection-only exporters from Okto session/genome files and `[dazai]` daemon
+logs. Imported events use the new `import:<source>@<version>` actor prefix.
+
+**Changed.** `POST /api/agent/tool-call` answers **422** for a `triggered_by`
+that is not an existing, strictly-earlier `seq_id` (it used to persist it).
+`LedgerWriter` rejects non-`int` `triggered_by`. The duplicate vector set in
+`crates/korg-registry/tests/conformance/` is removed; all harnesses read
+`spec/korg-ledger-v1/`, now with nested, mixed-shape, malformed-link, and
+redaction vectors, generated and drift-checked by `tools/generate_vectors.py`.
 
 ---
 

adapters/korg-ledger-py/README.md

@@ -6,7 +6,18 @@ frozen spec at `spec/korg-ledger-v1/`, pinned to the same conformance vectors
 as the Rust and JavaScript references.
 
 `LedgerWriter` produces hash-chained `JournalEvent` JSONL that the Rust
-`korg-verify` binary validates byte-for-byte.
+`korg-verify` binary validates byte-for-byte. Every record is redacted under
+[`korg-redact@v1`](../../spec/korg-ledger-v1/REDACTION.md) **before** it is hashed
+or written (opt out only with `redaction=None`).
+
+    python -m korg_ledger lint ledger.jsonl                 # chain + causal DAG (flat & nested) + redaction
+    python -m korg_ledger envelope okto-session S.json --source-version 0.1.0 > env.json
+    python -m korg_ledger envelope dazai-log daemon.log --source-version 0.1.0 > env.json
+    python -m korg_ledger import env.json --ledger ledger.jsonl   # korg-import@v1, atomic on rejection
+    python -m korg_ledger rechain-redacted old.jsonl new.jsonl    # migrate a journal that leaked a secret
+
+See [`IMPORT.md`](../../spec/korg-ledger-v1/IMPORT.md) for the Okto and
+ningen-shikkaku envelope profiles.
 
     pip install -e adapters/korg-ledger-py
     pytest adapters/korg-ledger-py/tests -v

adapters/korg-ledger-py/src/korg_ledger/__init__.py

@@ -11,13 +11,17 @@
     GENESIS,
     HASH_FIELDS,
     canonicalize,
+    causal_parent,
     chain_hash,
+    json_int,
     verify_anchors,
     verify_chain,
     verify_dag,
 )
 from ._hlc import Hlc
-from .writer import CausalityError, LedgerWriter
+from .redaction import POLICY as REDACTION_POLICY
+from .redaction import redact, scan, verify_redaction
+from .writer import CausalityError, LedgerWriter, RedactionError
 
 __all__ = [
     "GENESIS",
@@ -26,6 +30,13 @@
     "Hlc",
     "LedgerWriter",
     "CausalityError",
+    "RedactionError",
+    "REDACTION_POLICY",
+    "redact",
+    "scan",
+    "verify_redaction",
+    "causal_parent",
+    "json_int",
     "canonicalize",
     "chain_hash",
     "verify_chain",

adapters/korg-ledger-py/src/korg_ledger/__main__.py

@@ -0,0 +1,151 @@
+"""``python -m korg_ledger`` — import envelopes and lint journals.
+
+    python -m korg_ledger envelope okto-session SESSION.json --source-version 0.1.0 > env.json
+    python -m korg_ledger envelope okto-genome genome.jsonl --source-version 0.1.0 > env.json
+    python -m korg_ledger envelope dazai-log daemon.log --source-version 0.1.0 > env.json
+    python -m korg_ledger import env.json --ledger .korg/ledger.jsonl [--triggered-by N]
+    python -m korg_ledger lint .korg/ledger.jsonl        # chain + causal DAG + redaction
+    python -m korg_ledger rechain-redacted old.jsonl new.jsonl   # migration: redacted copy, new chain
+
+Exit codes: 0 ok · 1 rejected / lint failure · 2 usage or parse error.
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from pathlib import Path
+
+from ._hash import verify_chain, verify_dag
+from .imports import (
+    OKTO_PROFILE,
+    SCHEMA,
+    ImportRejected,
+    dazai_log_to_envelope,
+    import_envelope,
+    okto_mutation_record,
+    okto_session_to_envelope,
+)
+from .redaction import rechain_redacted, verify_redaction
+from .writer import LedgerWriter
+
+
+def _json_objects(text: str) -> list:
+    text = text.strip()
+    if not text:
+        return []
+    if text.startswith("["):
+        return json.loads(text)
+    try:
+        return [json.loads(text)]  # a single (possibly pretty-printed) object
+    except json.JSONDecodeError:
+        return [json.loads(line) for line in text.splitlines() if line.strip()]
+
+
+def _cmd_envelope(a) -> int:
+    text = Path(a.file).read_text(encoding="utf-8")
+    if a.source == "okto-session":
+        objs = _json_objects(text)
+        if len(objs) != 1 or not isinstance(objs[0], dict):
+            print("expected exactly one Okto Session object", file=sys.stderr)
+            return 2
+        env = okto_session_to_envelope(objs[0], okto_version=a.source_version)
+    elif a.source == "okto-genome":
+        env = {
+            "schema": SCHEMA,
+            "profile": OKTO_PROFILE,
+            "source": {"name": "okto", "version": a.source_version},
+            "records": [okto_mutation_record(m) for m in _json_objects(text) if isinstance(m, dict)],
+        }
+    else:
+        env = dazai_log_to_envelope(text.splitlines(), version=a.source_version)
+    json.dump(env, sys.stdout, indent=2, sort_keys=True)
+    sys.stdout.write("\n")
+    return 0
+
+
+def _cmd_import(a) -> int:
+    env = json.loads(Path(a.envelope).read_text(encoding="utf-8"))
+    try:
+        seqs = import_envelope(LedgerWriter(a.ledger), env, triggered_by=a.triggered_by)
+    except ImportRejected as e:
+        print(f"import rejected ({len(e.errors)} problem(s)); nothing was written:", file=sys.stderr)
+        for err in e.errors[:50]:
+            print(f"  - {err}", file=sys.stderr)
+        return 1
+    print(f"imported {len(seqs) - 1} record(s) as seq {seqs[0]}..{seqs[-1]} (batch seq {seqs[0]})")
+    return 0
+
+
+def _cmd_lint(a) -> int:
+    events = _json_objects(Path(a.journal).read_text(encoding="utf-8"))
+    key = a.key.encode() if a.key else None
+    errors = verify_chain(events, key) + verify_dag(events) + verify_redaction(events)
+    if errors:
+        print(f"✗ {len(errors)} problem(s):")
+        for e in errors[:50]:
+            print(f"  - {e}")
+        return 1
+    print(f"✓ {len(events)} events: chain intact, causal DAG well-formed, redaction-clean (korg-redact@v1)")
+    return 0
+
+
+def _cmd_rechain(a) -> int:
+    events = _json_objects(Path(a.journal).read_text(encoding="utf-8"))
+    key = a.key.encode() if a.key else None
+    errors = verify_chain(events, key)
+    if errors and not a.force:
+        print("refusing to re-chain a journal that does not verify (pass --force to copy it anyway):")
+        for e in errors[:20]:
+            print(f"  - {e}")
+        return 1
+    out = rechain_redacted(events, key)
+    if Path(a.out).exists() and Path(a.out).stat().st_size:
+        print(f"refusing to overwrite non-empty {a.out}", file=sys.stderr)
+        return 2
+    Path(a.out).write_text("".join(json.dumps(e, separators=(",", ":")) + "\n" for e in out), encoding="utf-8")
+    old_tip = events[-1].get("entry_hash") if events else None
+    new_tip = out[-1]["entry_hash"] if out else None
+    print(f"wrote {len(out)} redacted events to {a.out}")
+    print(f"  old tip {old_tip}\n  new tip {new_tip}\n  record both tips (and anchor them) to keep the copy auditable")
+    return 0
+
+
+def main(argv=None) -> int:
+    p = argparse.ArgumentParser(prog="python -m korg_ledger", description=__doc__.split("\n\n")[0])
+    sub = p.add_subparsers(dest="cmd", required=True)
+
+    e = sub.add_parser("envelope", help="build a korg-import@v1 envelope from a sibling tool's native output")
+    e.add_argument("source", choices=["okto-session", "okto-genome", "dazai-log"])
+    e.add_argument("file")
+    e.add_argument("--source-version", required=True, help="version of the tool that produced FILE")
+    e.set_defaults(fn=_cmd_envelope)
+
+    i = sub.add_parser("import", help="validate an envelope and append it to a ledger (atomic on rejection)")
+    i.add_argument("envelope")
+    i.add_argument("--ledger", required=True)
+    i.add_argument("--triggered-by", type=int, default=None)
+    i.set_defaults(fn=_cmd_import)
+
+    lint = sub.add_parser("lint", help="verify chain + causal DAG + redaction cleanliness of a journal")
+    lint.add_argument("journal")
+    lint.add_argument("--key", default=None, help="HMAC key for keyed chains")
+    lint.set_defaults(fn=_cmd_lint)
+
+    r = sub.add_parser("rechain-redacted", help="write a redacted, re-chained copy of a journal (migration)")
+    r.add_argument("journal")
+    r.add_argument("out")
+    r.add_argument("--key", default=None, help="HMAC key: verifies the input and keys the new chain")
+    r.add_argument("--force", action="store_true", help="copy even if the input chain does not verify")
+    r.set_defaults(fn=_cmd_rechain)
+
+    a = p.parse_args(argv)
+    try:
+        return a.fn(a)
+    except (OSError, json.JSONDecodeError) as ex:
+        print(f"error: {ex}", file=sys.stderr)
+        return 2
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

adapters/korg-ledger-py/src/korg_ledger/_hash.py

@@ -121,27 +121,84 @@ def verify_chain(events: list, key: bytes | None = None) -> list:
     return errors
 
 
+def json_int(value):
+    """The integer a JSON value denotes, or None (korg-ledger@v1 §9.2).
+
+    A ledger integer is a JSON number with an integral value within ±(2^53-1).
+    Booleans are NOT integers (Python's ``bool`` subclasses ``int``; Rust and JS
+    treat ``true`` as a boolean), and an integral float such as ``2.0`` IS the
+    integer 2 (JS cannot tell ``2.0`` from ``2`` after parsing, so all three
+    verifiers must accept it to agree)."""
+    if isinstance(value, bool):
+        return None
+    if isinstance(value, int):
+        return value if abs(value) <= _SAFE_INT_MAX else None
+    if isinstance(value, float) and value.is_integer() and abs(value) <= _SAFE_INT_MAX:
+        return int(value)
+    return None
+
+
+def _seq_label(e: dict) -> str:
+    sid = json_int(e.get("seq_id"))
+    return "?" if sid is None else str(sid)
+
+
+def causal_parent(event: dict):
+    """Resolve an event's causal parent across the flat and nested shapes (§9.3).
+
+    Returns ``(parent, error)``: ``parent`` is the parent seq_id (int) or None
+    for a root; ``error`` is a message (without the ``seq N:`` prefix) when the
+    link is malformed. A flat record carries ``triggered_by`` at the top level, a
+    nested JournalEvent under ``metadata``; ``null`` and absence both mean root.
+    When both are present they must name the same parent."""
+    raw = []
+    if event.get("triggered_by") is not None:
+        raw.append(("top-level", event["triggered_by"]))
+    md = event.get("metadata")
+    if isinstance(md, dict) and md.get("triggered_by") is not None:
+        raw.append(("metadata", md["triggered_by"]))
+    if not raw:
+        return None, None
+    parsed = []
+    for _where, v in raw:
+        n = json_int(v)
+        if n is None:
+            return None, "triggered_by is not an integer"
+        parsed.append(n)
+    if len(parsed) == 2 and parsed[0] != parsed[1]:
+        return None, f"conflicting triggered_by (top-level {parsed[0]} vs metadata {parsed[1]})"
+    return parsed[0], None
+
+
 def verify_dag(events: list) -> list:
-    """Check the causal DAG: unique seq_ids and strictly-earlier triggered_by
-    links. Byte-for-byte equivalent to the JS ``verifyDag`` and the Rust
-    ``verify_dag``; ``triggered_by`` is read at the top level, so nested
-    JournalEvent records (which carry it under ``metadata``) get the
-    uniqueness check only — matching the other two implementations."""
+    """Check the causal DAG (korg-ledger@v1 §5 + §9): unique integer seq_ids, and
+    every causal parent — top-level ``triggered_by`` (flat) or
+    ``metadata.triggered_by`` (nested JournalEvent) — is a well-formed integer
+    that references an existing, strictly-earlier seq_id.
+
+    Error strings are byte-identical to the Rust ``verify_dag`` and the JS
+    ``verifyDag``; the conformance manifest pins them."""
     errors: list[str] = []
     if not isinstance(events, list):
         return ["events is not a list"]
     dicts = [e for e in events if isinstance(e, dict)]
-    seqs = [e.get("seq_id") for e in dicts if isinstance(e.get("seq_id"), int)]
+    seqs = [n for n in (json_int(e.get("seq_id")) for e in dicts) if n is not None]
     seqset = set(seqs)
     if len(seqset) != len(seqs):
         errors.append("duplicate seq_id present")
     for e in dicts:
-        tb = e.get("triggered_by")
-        if not isinstance(tb, int):
+        label = _seq_label(e)
+        tb, err = causal_parent(e)
+        if err is not None:
+            errors.append(f"seq {label}: {err}")
             continue
-        sid = e.get("seq_id")
+        if tb is None:
+            continue
+        sid = json_int(e.get("seq_id"))
         if tb not in seqset:
-            errors.append(f"seq {sid}: triggered_by {tb} does not exist")
-        elif isinstance(sid, int) and tb >= sid:
-            errors.append(f"seq {sid}: triggered_by {tb} is not strictly earlier")
+            errors.append(f"seq {label}: triggered_by {tb} does not exist")
+        elif sid is None:
+            errors.append(f"seq ?: triggered_by {tb} cannot be ordered (seq_id is not an integer)")
+        elif tb >= sid:
+            errors.append(f"seq {label}: triggered_by {tb} is not strictly earlier")
     return errors