Skip to content

Latest commit

 

History

History
65 lines (49 loc) · 3.42 KB

operations.md

File metadata and controls

65 lines (49 loc) · 3.42 KB

Operations

Check an install

tracelet doctor

It checks the public URL, email, the data directory, the write-ahead log, Postgres and ClickHouse connectivity and versions, and pending migrations. Each line is ok, warn or FAIL, with a fix for anything that is not ok. It exits non-zero on a failure, so it works as a deploy check. tracelet doctor --json prints the same results for scripts.

Back up and restore

tracelet backup --out tracelet-2026-11-03.tar.gz
tracelet restore tracelet-2026-11-03.tar.gz

A backup is one compressed file with every Postgres and ClickHouse table and a manifest of row counts. Postgres is copied inside one consistent snapshot. Restore runs migrations, refuses a database that already has data, loads every table, checks the row counts against the manifest and advances id sequences. Backups contain personal data and encrypted secrets: store them like the database itself, and keep TRACELET_SECRET_KEY with them, because restored secrets need it.

Metrics

Prometheus metrics are at /metrics. The ones worth alerting on:

Metric Meaning
tracelet_dropped_total{kind,reason} Items rejected or lost, with the reason: quota, too_large, malformed, backpressure, unknown_key, filtered, poison (a record that kept failing), corrupt (unreadable write-ahead log data), delivery_failed (an alert that could not be sent)
tracelet_wal_lag_bytes Accepted data not yet written to storage. It should return to zero.
tracelet_wal_bytes Bytes held in the write-ahead log, against TRACELET_WAL_MAX_BYTES
tracelet_flush_total{table,outcome} Storage writes that succeeded or failed
tracelet_symbolicated_total{outcome} JavaScript events and whether a source map applied

/live answers while the process runs. /health also checks Postgres, ClickHouse and that the write-ahead log is under 90% full; use it for readiness.

The write-ahead log

Tracelet answers an SDK only after the data is written to a local write-ahead log in TRACELET_DATA_DIR. A processor then writes it to ClickHouse and Postgres and marks it done. If storage is down, data waits in the log and nothing is lost; when the log reaches TRACELET_WAL_MAX_BYTES, ingest answers with a retry-later status (which the SDKs honour) and counts the rejections as backpressure. Put the data directory on durable disk.

The operator console

Staff accounts see Admin in the account menu. It shows every organization with its usage, the tracelet doctor checks, write-ahead log size and lag, and lets staff grant or revoke staff access. Every grant and revoke writes an audit row. The first account created on an instance is staff; that account grants staff to others from the console.

Spend caps (Tracelet Cloud only)

Billing is off unless TRACELET_BILLING=stripe; a self-hosted instance never meters or caps anything. With billing on, each organization has a monthly spend cap, $0 by default, so only the free tier is used until an owner raises it in Settings, Organization, Usage. Usage counts unique events, errors and replay sessions received since the first of the month (UTC), including projects deleted during the month. Once the cap is reached, ingest for that organization is rejected with the counted quota reason, owners and billing members get one email, and ingest resumes when the cap is raised or the month ends. The cap is re-checked every minute and right after it changes.