Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .claude/rules/core-architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ joins the session of any sibling span on the same `trace_id`.

**Rule: both cache fields belong on both sides of every token sum and every price comparison** — an omission that recurs in `core/optimize/`. **(a) In aggregates**, sum cache reads AND cache writes in the same expression; `COALESCE(SUM(input_tokens + output_tokens + cache_tokens + cache_write_tokens), 0)` (the form in `optimize/runner.py`) is canonical for per-cluster and per-session totals — grep `SUM.*cache_tokens` when adding one. **(b) In pricing comparisons**, an actual-vs-alternative-model savings figure must price cache identically on both sides: a hand-rolled alt-price helper omitting cache-write while the actual `cost_usd` bills it inflates every derived saving by the candidate's full cache-write cost, and evades that grep because the SQL is correct and the Python pricing is not — the omission lives in a per-row helper, not a `SUM`. **Keep exactly one pricer:** route every alternative-side computation through `calculate_cost`, which covers all four token classes.

- **`alerts.py`** — `AlertEngine` (the kinds it can fire are the `AlertType` enum in `core/models.py`; read the enum, never a count kept here), `CooldownTracker` (in-memory, per agent+type, resets on restart), `AlertDispatcher` routing to the channel types the `_build_channel` factory knows (stdout, file, ntfy, webhook, Discord, Telegram — that factory is the live set). `AlertEngine.fire()` is the entry point for `SchemaValidator` and `DriftDetector`. Suppressed alerts are still persisted, just not dispatched. Thresholds are module-level constants, not config — read them for values: **retry loop** fires when the same tool call repeats often enough inside a trailing span window (`_RETRY_LOOP_THRESHOLD` / `_RETRY_LOOP_WINDOW`); **failure rate** when the error share over a trailing span window crosses a ceiling, re-evaluated only every Nth error to bound cost (`_FAILURE_RATE_THRESHOLD` / `_FAILURE_RATE_WINDOW` / `_FAILURE_RATE_CHECK_INTERVAL`); **session duration** past a default wall-clock ceiling (`_SESSION_DURATION_DEFAULT`). Stdout and file channels always include full detail regardless of `include_captured_content`.
- **`alerts.py`** — `AlertEngine` (the kinds it can fire are the `AlertType` enum in `core/models.py`; read the enum, never a count kept here), `CooldownTracker` (in-memory, per agent+type, resets on restart), `AlertDispatcher` routing to the channel types the `_build_channel` factory knows (stdout, file, ntfy, webhook, Discord, Telegram, Mast — that factory is the live set). `AlertEngine.fire()` is the entry point for `SchemaValidator` and `DriftDetector`. Suppressed alerts are still persisted, just not dispatched. Thresholds are module-level constants, not config — read them for values: **retry loop** fires when the same tool call repeats often enough inside a trailing span window (`_RETRY_LOOP_THRESHOLD` / `_RETRY_LOOP_WINDOW`); **failure rate** when the error share over a trailing span window crosses a ceiling, re-evaluated only every Nth error to bound cost (`_FAILURE_RATE_THRESHOLD` / `_FAILURE_RATE_WINDOW` / `_FAILURE_RATE_CHECK_INTERVAL`); **session duration** past a default wall-clock ceiling (`_SESSION_DURATION_DEFAULT`). Stdout and file channels always include full detail regardless of `include_captured_content`.

- **`drift.py`** — `DriftDetector`, Z-score based behavioral drift detection, fires at session end.

Expand Down
53 changes: 52 additions & 1 deletion docs/alerts.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,12 @@ Configure where alerts go in `.tj/config.toml`. Multiple channels work simultane
type = "ntfy"
topic = "my-agent-alerts"

# Mast pager app for iPhone and Android. Free, with or without an account (limits below).
# The key in the URL is the only credential.
[[alerts.channels]]
type = "mast"
channel_url = "https://mast.tissue.dev/mk_0123456789abcdef0123456789abcdef01234567"

# Discord webhook
[[alerts.channels]]
type = "discord"
Expand All @@ -56,6 +62,51 @@ path = "~/.tj/alerts.log"
type = "stdout"
```

### Mast

[Mast](https://tissue.systems/mast) is a pager app for iPhone and Android. Install it, make a
channel, and paste the channel URL into the config above. It works without an account, but a
channel made that way is lost if the app is reinstalled, and the only sign on this side is a
"No such channel." line in the log. Sign in to a free Tissue account if you want the channel
to survive that.

Both ways are free but capped. With no account you get 2 channels and 1,000 messages a month;
a free account gets 10 channels and 10,000 messages. Mast keeps accepting sends past the monthly
number, but if your agents alert a lot, check [Mast's limits](https://tissue.systems/mast/docs/limits/)
before relying on it.

`tj` posts a JSON body with `title`, `body`, `priority` and `key`. Mast documents those fields
on its [sending page](https://tissue.systems/mast/docs/send/), which is the place to look if
sends start failing.

`tj` maps severity onto three of Mast's four priorities: `info` sends `quiet`, `warning`
sends `normal`, `critical` sends `loud`. Nothing maps to the fourth, `page`. It repeats until
someone acks it and ignores mute and quiet hours, so you have to ask for it, which is what
you'd do on a channel that's meant to wake someone up:

```toml
[[alerts.channels]]
type = "mast"
channel_url = "https://mast.tissue.dev/mk_..."
priority = "page"
min_severity = "critical"
```

Each channel also has a ceiling, shown as **Highest priority** in the app, and a send that
asks for more than that is delivered at the ceiling. It's set when the channel is created. The
iOS app doesn't go below `loud`, the Android app uses whatever default priority you picked, and
a channel made from an API token starts at `loud`. So a channel created on Android with the
default of `Normal` will deliver `critical` alerts as `normal`, and `priority = "page"` won't
page anyone until you raise the ceiling on the phone. Worth checking before you rely on it.

Each send carries a dedupe key built from the same things the cooldown uses: the alert type,
the agent, and for `cost_budget_session` and `session_duration` the session too. If the same
alert fires again inside the channel's dedupe window (5 minutes unless you change it), Mast
adds to the count on the card already on the phone instead of sending a new one. Two sessions
over their own budget are different alerts, so they get separate cards. The default cooldown
is 60 seconds, so a repeat usually lands inside the window; if you raise `cooldown_seconds`
past the window, repeats arrive as new cards.

## Sensitive actions

Define which tool calls should trigger immediate alerts:
Expand Down Expand Up @@ -88,7 +139,7 @@ Cooldown state survives a daemon restart: `AlertEngine` rehydrates it from the p

## Content stripping

By default, alert payloads sent to external channels (Discord, Telegram, webhook, ntfy) have sensitive content stripped: `prompt_content`, `completion_content`, `tool_input`, `tool_output`. To include full content:
By default, alert payloads sent to external channels (Discord, Telegram, webhook, ntfy, Mast) have sensitive content stripped: `prompt_content`, `completion_content`, `tool_input`, `tool_output`. To include full content:

```toml
[alerts]
Expand Down
4 changes: 2 additions & 2 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ flowchart TD
Ingest["IngestPipeline\nSanitization · Session continuity · Attribute extraction"]

Ingest --> Cost["CostEngine\npricing.toml"]
Ingest --> Alerts["AlertEngine\n13 types · 6 channels"]
Ingest --> Alerts["AlertEngine\n13 types · 7 channels"]
Ingest --> Schema["SchemaValidator\nexplicit JSON Schema"]

Cost --> DB["DuckDB\nlocal · embedded"]
Expand Down Expand Up @@ -179,7 +179,7 @@ The alert engine evaluates rules after every span ingest. Alert types fall into

### Dispatch and cooldown

`AlertDispatcher` routes fired alerts to 6 channel types: stdout, file, ntfy, webhook, Discord, Telegram. `CooldownTracker` prevents alert storms by suppressing repeat alerts of the same type for the same agent within a configurable window. Suppressed alerts are still persisted to the database but not dispatched.
`AlertDispatcher` routes fired alerts to 7 channel types: stdout, file, ntfy, webhook, Discord, Telegram, Mast. `CooldownTracker` prevents alert storms by suppressing repeat alerts of the same type for the same agent within a configurable window. Suppressed alerts are still persisted to the database but not dispatched.

Content stripping removes `prompt_content`, `completion_content`, `tool_input`, `tool_output` from payloads sent to external channels unless `alerts.include_captured_content = true`. Stdout and file channels always receive the full payload.

Expand Down
6 changes: 5 additions & 1 deletion docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,10 @@ include_captured_content = false
type = "discord"
webhook_url = "https://discord.com/api/webhooks/..."

[[alerts.channels]]
type = "mast"
channel_url = "https://mast.tissue.dev/mk_..." # iPhone/Android pager app, free; see docs/alerts.md

[export]
[export.otlp]
endpoint = "" # set to forward spans to an external OTel backend
Expand Down Expand Up @@ -225,7 +229,7 @@ The four flags are independent: capture prompts without completions, or tool inp

**Where the gate runs.** All spans flow through `IngestPipeline.process()`, which calls `strip_captured_content()` once before any DB write. The receiver, the OTLP HTTP endpoint, and the Claude Code JSONL backfill all go through this single gate — there's no per-source duplication.

**Precedence with alert channels.** `alerts.include_captured_content = true` lets external alert channels (webhook, Discord, Telegram, ntfy) receive full payloads. If the corresponding `[capture]` flag is `false`, the content is already stripped at ingest, so the alert flag is moot — channels can't include what was never captured. Stdout and file alert channels always receive full payloads from the in-memory span before the DB strip, so they still see content during the brief in-process window.
**Precedence with alert channels.** `alerts.include_captured_content = true` lets external alert channels (webhook, Discord, Telegram, ntfy, Mast) receive full payloads. If the corresponding `[capture]` flag is `false`, the content is already stripped at ingest, so the alert flag is moot — channels can't include what was never captured. Stdout and file alert channels always receive full payloads from the in-memory span before the DB strip, so they still see content during the brief in-process window.

**What this means for downstream analyzers.**

Expand Down
2 changes: 1 addition & 1 deletion docs/nemoclaw-integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ The NemoClaw observer listens on the OpenShell Gateway WebSocket for sandbox enf
| Syscall denied | `syscall_denied` | critical |
| Inference rerouted | `inference_rerouted` | warning |

These alerts flow through the standard alert pipeline — cooldown, dispatch to configured channels (ntfy, Discord, Telegram, webhook), and persistence to the database.
These alerts flow through the standard alert pipeline — cooldown, dispatch to configured channels (ntfy, Mast, Discord, Telegram, webhook), and persistence to the database.

## Configuration

Expand Down
18 changes: 18 additions & 0 deletions tests/unit/test_cmd_policy.py
Original file line number Diff line number Diff line change
Expand Up @@ -157,6 +157,24 @@ def test_alerts_row_summarises_channel_count_and_flags():
assert "path=/tmp/alerts.log" in ch1.setting


def test_channel_summary_redacts_the_mast_channel_key():
# The URL carries the channel key, so it must not show up in the table.
summary = _channel_summary(AlertChannelConfig(
type="mast",
channel_url="https://mast.tissue.dev/mk_0123456789abcdef",
))
assert "type=mast" in summary
assert "channel_url=<set>" in summary
assert "mk_0123456789abcdef" not in summary


def test_channel_summary_shows_a_mast_priority_override():
summary = _channel_summary(AlertChannelConfig(
type="mast", channel_url="https://mast.tissue.dev/mk_x", priority="page",
))
assert "priority=page" in summary


def test_drift_overrides_default_detects_change():
assert _drift_overrides_default(DriftConfig()) is False
assert _drift_overrides_default(DriftConfig(token_threshold=3.0)) is True
Expand Down
18 changes: 18 additions & 0 deletions tests/unit/test_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -246,6 +246,24 @@ def test_alerts_channels_parsed(self):
assert len(config.alerts.channels) == 2
assert config.alerts.channels[1].topic == "my-topic"

def test_alerts_mast_channel_parsed(self):
raw = {
"alerts": {
"channels": [
{
"type": "mast",
"channel_url": "https://mast.tissue.dev/mk_abc",
"priority": "page",
},
],
}
}
config = _parse(raw)
channel = config.alerts.channels[0]
assert channel.type == "mast"
assert channel.channel_url == "https://mast.tissue.dev/mk_abc"
assert channel.priority == "page"

def test_default_alert_channel_is_stdout(self):
config = _parse({})
assert len(config.alerts.channels) == 1
Expand Down
Loading
Loading