Give the edges' query latency series buckets up to 60 s (T-625)

[chasers]

TL;DR: Query latency now has buckets from 50 ms to 60 s. A query p99 no longer reads a flat ~9.7 s whenever the tail passes 2.5 s.

Tracker: T-625.

Why

• All HTTP edges shared one set of latency bounds: 5 ms, 25 ms, 100 ms, 250 ms, 500 ms, 1 s, 2.5 s, 10 s.
• These were tuned for insert acks (250 ms to 1 s).
• A VictoriaMetrics query runs up to max_query_duration_ms (30 s). An API query runs to timeoutMs (60 s by default).
• Above 1 s there were only two bounds: 2.5 s and 10 s.
• So histogram_quantile(0.99, ...) read any query tail past 2.5 s as about 9.7 s.
• Nothing past 10 s was visible.
• On the sandbox (2026-10-02), the VictoriaMetrics edge p99 sat at 8.4-9.8 s. In one window 37 of 150 queries were between 2.5 s and 10 s.

What changed

• New @query_latency_buckets: 50, 100, 250, 500, 750 ms, 1, 1.5, 2, 3, 5, 7.5, 10, 15, 20, 30, 60 s.
• These series use them:
  • smolquery_api_request_microseconds_bucket{route="query"}
  • smolquery_clickhouse_request_microseconds_bucket{kind="query"}
  • smolquery_victoriametrics_request_microseconds_bucket{kind="query"}
• Every other route and kind keeps the old bounds.
• timed/4 takes the bounds from its caller. The caller picks them by route or kind (latency_buckets/1).
• The HELP texts, docs/clickhouse.md and a docs/deployment.md upgrade note describe the change.

Tests

• ✅ New test: a 3.4 s request through each query series lands at 5 s and above, and not at 3 s or below.
• ✅ No query series emits the old le="2500000".
• ✅ Insert (API) and write (VictoriaMetrics) keep the old bounds.
• ✅ Mutation check: with query series back on the old bounds, the new test fails.

Review

A Fable review found nothing serious. It confirmed that the selection is a closed set, that rendering handles two bound sets in one family, and that no test, dashboard or doc in the repo hardcodes the old query bounds. Fixed in Review of T-625: ...:

• 30 s is only the VictoriaMetrics cap. API queries run to 60 s by default, so the query bounds gain 60 s.
• The upgrade note now says a sum by (le) across routes or kinds mixes bound sets for good, not only during a roll.
• The test title.

Checks

• ✅ mix precommit: 3,079 tests pass
• ✅ mix ci
• ✅ mix dialyzer

Watch out

• ⚠️ Each query series has 16 le values instead of 8 (+8 series per query label value).
• ⚠️ During a rolling upgrade, old and new pods report different le sets. A sum by (le) over both is rough until the old pods are gone and one rate window has passed.
• ⚠️ A sum by (le) across routes or kinds now mixes two bound sets for good. Filter or group by route or kind first.
• ⚠️ Below 50 ms the query series have no bound. ClickHouse catalog reads and SELECT 1 all land in le="50000". The goal here is the tail, so this is intended.
• ⚠️ A ClickHouse query past 60 s counts only in +Inf, so a quantile reads it as 60 s.
• ⚠️ This fixes the measurement only.
  The real tail (query-tier CPU) is a separate fix.
• ⚠️ T-625's sandbox check is open: replay the dashboard at 8 in flight; p99 should read between 7.5 s and 15 s.

🤖 Generated with Claude Code