sortie

Declarative load-test plans on top of Nighthawk.

Nighthawk is an excellent load generator and a poor test harness. It takes one set of options, runs one benchmark, and prints one result. sortie adds the layer around it that a benchmarking campaign actually needs: a plan file describing several scenarios, rate shapes over time, fan-out across a pool of Nighthawk instances, and thresholds that turn a run into a pass or a fail with an exit code CI can read.

It drives Nighthawk's existing gRPC control plane — NighthawkService and, for fan-out, NighthawkDistributor — so the load generation itself is entirely Nighthawk's, unmodified.

version: v1

pools:
  - name: local
    services: ["127.0.0.1:8443"]

defaults:
  pool: local
  target: http://127.0.0.1:8080/
  concurrency: "2"

thresholds:
  - "counter:benchmark.http_5xx == 0"

scenarios:
  - name: ramp
    executor:
      type: ramping-rate
      rate: 500
      ramp_time: 30s
      duration: 120s
    thresholds:
      - "latency_2xx.p95 < 50ms"
      - "latency_2xx.p99 < 200ms"

$ sortie run plan.yaml
  ok   ramp  (ramp to 500 rps over 30s, then hold for 90s, pool "local", 2m1s)
       127.0.0.1:8443: 52500 requests in 2m0s
         ok  counter:benchmark.http_5xx == 0  actual 0           (pool)
         ok  latency_2xx.p95 < 50ms           actual 2.598527ms  (per-backend)
         ok  latency_2xx.p99 < 200ms          actual 8.114688ms  (per-backend)

PASS  1/1 executions passed

Build

bazel build //:sortie
bazel test //...

Bazel is the only supported build. The engine's protos are generated by Bazel from engine/api into packages under github.com/bpalermo/sortie/engine/api/..., which no Go module publishes, so plain go build cannot resolve them. go.mod exists for Gazelle and editor tooling and lists the third-party modules only.

Change dependencies through rules_go's SDK — bazel run @rules_go//go -- get <module>@<version> — then bazel mod tidy. Not go mod tidy, which cannot resolve the Bazel-only proto packages.

bazel run //:gazelle regenerates BUILD files after adding or renaming a Go file. CI fails if it leaves a diff.

The engine's Envoy dependency is pinned by commit; bazel/bump-envoy.sh <commit> moves the pin and everything that has to move with it (see AGENTS.md).

Commands

Command	What it does
sortie run <plan.yaml>	Run the plan and report a verdict. --json for CI, -o FILE to write the report to a file (stdout then gets the text summary), --results-stream FILE to append each execution's result to a file as it finishes, --progress 5s to narrate each backend's run on stderr.
sortie validate <plan.yaml>	Check the plan without running anything.
sortie compile <plan.yaml>	Print the CommandLineOptions it would send to each backend.

Exit code is 0 when every threshold held, 1 when one failed or an execution errored, and 2 when the plan or command line was invalid.

Pools

A pool names the Nighthawk backends a scenario runs on.

pools:
  # sortie drives each instance itself and merges the results.
  - name: local
    services: ["10.0.0.11:8443", "10.0.0.12:8443"]

  # A nighthawk_distributor fans the request out on sortie's behalf.
  - name: fleet
    distributor: "10.0.0.1:8442"
    targets: ["10.0.0.11:8443", "10.0.0.12:8443"]

The distributor forwards one ExecutionRequest unchanged to every target, so sortie sends it the per-target share rather than the aggregate. rate therefore means the same thing on both paths — what the target receives — but on this one it must divide exactly by targets x concurrency, because a single request leaves nowhere to put a remainder.

Nighthawk ships no distributor binary. NighthawkDistributor exists in its sources as a library and is marked experimental (envoyproxy/nighthawk#369), but no released binary hosts it and nighthawk_service does not. Using a distributor pool means building a host for that service yourself. The direct services: pool is the path that works out of the box.

pools found through DNS

pools:
  - name: nodes
    dns: nightly-sortie-engine-nodes.loadtest.svc.cluster.local:8443

sortie run resolves the host's A and AAAA records once, when it starts, and drives every address it got at that port for the whole run. A Kubernetes headless Service answers with one address per ready pod, so a DaemonSet of engines behind one is a pool of one backend per node that needs no editing when nodes come and go.

The addresses are sorted, so the backend order is a property of the addresses and not of the order the name server answered in, and the report says what the name resolved to: it is the only record of which nodes a run drove. A name that answers with nothing is retried for up to 30 seconds, because a headless Service lists a pod only once it is ready; nothing after that is an error before any load is generated, with exit code 2. With several such pools each name has its own 30 seconds, and one that has answered is not asked again. A node that joins during a run gets no load, and one that leaves fails the execution visibly rather than silently shrinking the pool, without taking the other nodes' results with it. validate and compile accept the pool without resolving it.

Executors

Type	Shape
constant-rate	A fixed aggregate rate for duration.
ramping-rate	Linear from zero to rate over ramp_time, then holds rate for the rest of duration. Nighthawk's nighthawk.linear-ramping-rate-limiter-plugin.
staircase	One constant-rate step per entry in stages.

open_loop: true selects Nighthawk's open-loop mode, where the rate limiter never compensates for a client that cannot keep pace, so saturation shows up as pool overflow rather than as latency.

between requests a worker waits

A worker thread that has nothing to send waits until the next request is due. That is sortie's choice, not the engine's: left to itself the engine spins, which releases each request closest to its time and costs a whole core per worker at any rate. Measured on one worker sending 60 rps, the engine's idle strategies cost SPIN 970 millicores, POLL 390, SLEEP 150 and WAIT 26, and sortie asks for WAIT.

The reason is what the alternative does to the numbers. A load generator runs beside the thing it measures, usually at tens or hundreds of requests per second, and often under a CPU limit. Spinning there gets the engine throttled, and it then reports its own scheduling delay as the target's latency. A user running 8 targets at 60 rps per engine measured 1.7 cores per engine with SLEEP and 0.11 with WAIT, with every request sent on both and the same latencies.

Three things to know about it. The cost still grows with the number of workers: each wakes at least every 5 ms to see whether the run should end, about 11 millicores per idle worker. A worker that is woken late ends the run without the requests that came due while it slept, so a run may send fewer than rate x duration, and never more. What has been seen is one request short, 999 of 1000, in about one short run in four on a busy machine, and none short in a user's two 15-minute runs; that is an observation, not a bound, since nothing limits how late an overloaded machine wakes a worker. Write a threshold on a total count as a range with some slack, not an equality. And waiting has only been measured at low rates. For a very high rate on a machine with cores to spare, where a worker is rarely idle and a late wake-up would show, ask for the engine's default:

scenarios:
  - name: flood
    executor: {type: constant-rate, rate: 50000, duration: 1m}
    nighthawk_template:
      sequencer_idle_strategy: {value: SPIN}

A strategy named in the template is always left alone.

rate is aggregate, and that has consequences

rate is what the target receives. Nighthawk's own --rps is per worker thread, so sortie divides by backends x concurrency before sending it.

Two things follow, both deliberate:

• A rate no integer per-worker --rps can express is an error, not a rounding. rate: 101 with concurrency: "2" is refused. A load generator that silently produces a different rate than the plan asked for is worse than one that refuses.
• concurrency: "auto" cannot be combined with a rate, because the worker count is decided on the backend and sortie cannot divide by a number it does not know.

per_backend makes rate each backend's

executor: {type: constant-rate, rate: 60, duration: 8h, per_backend: true}

With per_backend: true, rate (and each stage's rate) is what every backend generates rather than the pool's total, so the load scales with the pool instead of being spread over it. It is written for a DNS pool of one engine per node: 60 rps per node stays 60 rps per node however many nodes the name resolves to. The divisibility rule then concerns rate / concurrency alone, and the report labels the rate rps per backend. On a distributor pool it is what every target receives anyway, since the distributor forwards one request unchanged to each.

staircase stages are separate runs

Nighthawk cannot change the rate of a run already in flight — the UpdateRequest RPC exists in api/client/service.proto but the service rejects it. Each stage is therefore its own execution: connections are re-established at every boundary, and each stage is reported and judged separately. A staircase of hundreds of stages is a run of hours, so each stage's verdict is printed, and with --results-stream recorded, when the stage ends rather than when the run does: see a verdict as each execution finishes.

A stage that fails does not stop the ones after it, with one exception: when a stage is refused because an engine is at its execution cap (see Weighted targets), the scenario's remaining stages are not attempted. They are listed in the report as SKIP ("not_run": true in the JSON) with the stage that was refused, and the plan's other scenarios still run. The refused stage itself has "refused": "execution_cap" in the JSON, on each of its executions and on the entries of backend_errors for the engines that refused: those backends were not lost, and none of this needs the error text to be read.

Weighted targets

A scenario can drive several targets at once, each with a share of its rate. This is the shape of a soak that draws a weighted target per request.

scenarios:
  - name: mesh
    executor: {type: constant-rate, rate: 60, duration: 8h, open_loop: true}
    targets:
      - {name: orders,  url: "http://orders.shop.svc:8080/",  weight: 6}
      - {name: catalog, url: "http://catalog.shop.svc:8080/", weight: 3}
      - {name: search,  url: "http://search.shop.svc:8080/",  weight: 1}
    thresholds:
      - "counter:benchmark.http_5xx == 0"

Each entry becomes its own execution, labelled <scenario>/<name>, with its own connection pool, rate limiter, counters and latency histograms. They start together on every backend in the pool and are reported and judged one by one, so a threshold on the scenario has to hold for every target, and a failing target is named. targets replaces target; every other field of the scenario applies to each target alike, except the request a target describes for itself (below). defaults may carry either form; a scenario that names a target or a list of its own takes neither from it.

A target may carry its own method, headers, body or body_file, so reads and writes run side by side on the same schedule:

    targets:
      - {name: orders, url: "http://orders.shop.svc:8080/", weight: 6}
      - name: submit
        url: "http://orders.shop.svc:8080/submit"
        weight: 1
        method: POST
        headers: ["content-type: application/json"]
        body: '{"sku": 1}'

Each of the four that a target sets replaces the scenario's (after defaults), and one it leaves out is the scenario's: headers replaces the whole list rather than adding to it, and body and body_file replace whichever of the two the scenario had. A target cannot clear what the scenario sets, so what only some targets need belongs on those targets. The other targets are sent exactly what they were sent before: a target that describes no request compiles to the same engine options whatever its neighbours say. The fields are held to the rules of the target's scheme, so a udp:// target takes a body alone and a tcp:// one no method or headers; with the method on the HTTP target rather than on the scenario, one scenario can drive an HTTP, a TCP and a UDP target.

Two things differ from a per-request weighted draw, both deliberate:

• Each target is paced on its own. A target's rate is a fixed rate x weight / sum(weights), kept by its own rate limiter, not a random share that only averages out to it. Every target gets its rate throughout the run, and one slow target cannot starve the others of their slots. A window short enough to hold a handful of requests can still contain some targets and not others.
• Shares are rounded. A share is rounded to the nearest whole request per second and is never less than 1, so the shares may not sum to rate: 100 split 1:1:1 is 33 + 33 + 33. The report shows the rate each target ran at. Each share is then divided over backends and workers by the usual rule, so it must be a multiple of concurrency.

The backend has to accept one execution per target at a time: nighthawk_service --max-concurrent-executions N, default 1. The chart sets engine.maxConcurrentExecutions (default 16). A backend at its limit refuses the start, and the scenario does not run on part of its slots: as soon as one start is refused, the executions that did start are cancelled on every backend, and the scenario fails at once with an error that names the backend, its limit and the setting to raise. The slots are not reserved ahead of the start, so for that moment the targets that got one do send load. Behind a distributor the refusal is recognised and reported the same way, but only once the distributor answers, and the executions that started there cannot be stopped: a distributor forwards no cancellation. Every execution has its own worker threads, so a ten-target scenario at concurrency: "2" runs twenty. An engine that allows more than one execution logs at its own level throughout: the log level is one setting per process, so a verbosity passed through nighthawk_template is not applied there. Concurrent executions are tested over HTTP/1; the other protocols and modes have not been run concurrently.

A staircase scenario with targets runs its stages in order, with every target's execution of a stage running together.

gRPC

A scenario with a grpc block generates gRPC load instead of plain HTTP. It implies protocol: http2 and method: POST; the request message is the scenario's body or, for a serialized protobuf, body_file (a path relative to the plan file).

scenarios:
  - name: unary
    target: http://grpc.example.test:8080/helloworld.Greeter/SayHello
    body_file: hello_request.bin
    grpc: {mode: unary}
    executor: {type: constant-rate, rate: 1000, duration: 60s}
    thresholds:
      - "counter:benchmark.grpc_error == 0"
      - "latency_grpc_ok.p99 < 20ms"

  - name: streaming
    target: http://grpc.example.test:8080/echo.Echo/Chat
    body_file: chat_message.bin
    concurrency: "2"
    grpc: {mode: bidi-stream, streams: 20, max_inflight_per_stream: 256}
    executor: {type: constant-rate, rate: 4000, duration: 60s}
    thresholds:
      - "counter:benchmark.stream_deferred == 0"
      - "message_latency.p99 < 10ms"

Mode	What it does
unary	One gRPC call per request, scored by grpc-status: a failed RPC never counts as http_2xx. Counters benchmark.grpc_error and benchmark.grpc_status.<code>; statistic latency_grpc_ok.
bidi-stream	streams long-lived bidirectional streams per backend; the message is sent round-robin at the executor's rate and each one is timed against its echo (benchmark_stream.message_latency). The server must echo one message per message, in order. A stream holding max_inflight_per_stream unanswered messages drops the next scheduled send into benchmark.stream_deferred -- the saturation signal.

In bidi-stream the engine spreads a backend's share of the rate over its own workers, so sortie sends each backend its aggregate rather than a per-worker rate. The split is still made in per-worker units -- the plan's rate must be a multiple of the scenario's concurrency, which has to be a number, as must streams -- so 300 rps over two backends with four workers is 152 and 148. sortie compile shows the result.

WebSocket

scenarios:
  - name: chat
    target: http://gateway.example.test/ws   # the upgrade request's URL, not ws://
    body: '{"type":"ping"}'
    concurrency: "2"
    websocket:
      streams: 40                   # connections per backend; default 20
      max_inflight_per_stream: 16   # default 256
      binary: false                 # text frames unless true
    executor: {type: constant-rate, rate: 2000, duration: 60s}
    thresholds:
      - "benchmark_stream.message_latency.p99 < 50ms"
      - "counter:benchmark.stream_deferred == 0"

Each backend upgrades streams HTTP/1.1 connections (through Envoy's own upgrade path, so an Envoy in front with upgrade_configs: [{upgrade_type: websocket}] is exactly what gets tested) and sends the message on them round-robin at its share of the executor's rate -- the rate is pool-wide and divided between backends, as always -- each timed against its echo. The server is expected to echo what it receives; sortie prefixes every message with a sequence number to match echoes, so an endpoint that answers something else counts in benchmark.stream_unexpected_message. Everything said about bidi-stream applies: the rate is a backend's aggregate, streams and the rate must be multiples of concurrency, a full connection drops the next send into benchmark.stream_deferred, and the counters and statistic carry the same names (benchmark.stream_*, benchmark_stream.message_latency), plus stream_upgrade_rejected and stream_protocol_errors. Every stream is a connection: sortie sets the engine's connection cap to the streams per worker unless the plan sets connections. Implies protocol: http1 and method: GET; mutually exclusive with grpc. nighthawk_test_server's websocket-echo filter is the target the e2e test uses.

TLS

An https:// target needs nothing more than that: the engine verifies nothing and presents no certificate, so a self-signed target works out of the box. tls turns on what a real one needs:

defaults:
  target: https://gateway.example.test/
  tls:
    ca_file: certs/ca.pem           # verify the target against these roots
    cert_file: certs/client.pem     # present this certificate when asked (mTLS)
    key_file: certs/client-key.pem

Paths are relative to the plan file. sortie reads the files and sends them to the backends inline, so a backend needs no access to them -- which also means the plaintext channel to a backend (see Pools) carries the client key. What sortie prints (sortie compile) and what the engine logs replace the key with a note of its size; the options proto itself carries it. SNI follows the target's host, or a Host header; ALPN follows protocol. Anything beyond this -- cipher suites, a SPIFFE validator, a different transport socket altogether -- goes through nighthawk_template: its tls_context gets the same SNI and ALPN treatment, its transport_socket replaces the whole socket.

TCP

scenarios:
  - name: raw
    target: tcp://edge.example.test:9000     # tcps:// for a TLS-terminating listener
    body: "ping\n"
    concurrency: "2"
    tcp:
      connections: 4          # per worker; default 1
      expect_echo: true       # default; false for a sink that answers nothing
      max_messages_per_connection: 600   # a new connection every 600 messages; default 0, never
    executor: {type: constant-rate, rate: 2000, duration: 60s}
    thresholds:
      - "benchmark_tcp.message_latency.p99 < 20ms"
      - "counter:benchmark.tcp_deferred == 0"
      - "counter:benchmark.tcp_connection_closed == 0"
      - "counter:benchmark.tcp_echo_mismatch == 0"

For Envoy's tcp_proxy and TLS-terminating listeners in front of non-HTTP services. The target's scheme selects the mode (an explicit port is required); tcp tunes it. Each worker keeps connections connections open and writes the message, as is, on them round-robin at the executor's rate -- per worker, as for HTTP, so the rate is divided by backends x workers as usual. With expect_echo a write completes when those bytes come back on the same connection -- matched in order, which is what a TCP connection gives -- and that round trip is the latency (benchmark_tcp.message_latency). Without it the write completes at once and nothing is read or timed: a write only queues bytes locally. tls works with tcps://. method, headers, protocol, grpc and websocket are errors with a tcp target.

expect_echo needs an exact echo: the message's bytes, nothing before, between or after them. There is no framing to tell replies apart, so a target that prefixes or rewrites its reply cannot be matched -- and cut into message-sized pieces, such a stream lines up with the message every so often by chance, which would count as an echo and be timed against a send it has nothing to do with. So the first message's worth of bytes on a connection that is not the message (or that arrives with nothing outstanding) ends the matching on that connection: it counts in benchmark.tcp_echo_mismatch, the connection's outstanding messages are tcp_inflight_lost and are not timed, and the connection is closed and reopened as above. A latency is therefore only ever recorded for an echo before which everything its connection returned was exactly what was sent; against a target that is not an echo benchmark_tcp.message_latency stays empty rather than filling with numbers that look real, tcp_messages_received stays at zero, and most of the run is tcp_unavailable, since each reopened connection lasts one reply and the backoff grows to a second. The engine logs an error naming the cause, once per worker, and the text report puts a warning under the backend's line. Mismatches do not fail an execution by themselves: add "counter:benchmark.tcp_echo_mismatch == 0" to the thresholds for that, or set expect_echo: false for a target that is not meant to echo.

A connection that closes is reopened. Whether the peer closed or reset it or a connect failed, the worker connects again after a backoff: 10 ms, doubling with each consecutive retry up to 1 s, and back to 10 ms once the connection has gone 2 s without needing one. So a target that is rolled is picked up again within a second of coming back, and a dead one is tried once a second per connection, not in a loop. The run stays open-loop meanwhile: messages that come due while a connection is away are dropped and counted in benchmark.tcp_unavailable, not held back and sent in a burst, and messages unanswered on the connection when it closed are benchmark.tcp_inflight_lost. tcp_connection_closed counts the closes, tcp_reconnects the times a connection came back.

max_messages_per_connection makes a run keep setting connections up: once a connection has sent that many messages it is replaced. The replacement is opened while the old connection goes on sending, and takes over when it has connected, so rotating neither drops nor delays a message -- and a connection carries the configured number plus whatever came due while its replacement connected (none, or a few). The old connection is sent nothing more, is given up to the scenario's timeout for its outstanding echoes (still missing then: tcp_inflight_lost), and is closed. A replacement that fails to connect is retried with the same backoff while the old connection keeps serving. At 100 messages a second per connection, 600 is a new connection every 6 s. There is no time-based lifetime; at a constant rate the message count is one.

Every successful connect -- the first ones, reconnects, rotations -- is timed from connect() to connected, TLS handshake included, in benchmark_tcp.connect_latency. To read connection setup from the counters: tcp_connections_opened is every connect that succeeded (the workers' first connections + tcp_reconnects + tcp_connections_rotated) and tcp_connect_failures every attempt that did not (refused, reset, or not connected within timeout), so the success rate is opened / (opened + failures).

Counters: benchmark.tcp_connections_opened, tcp_connect_failures, tcp_reconnects, tcp_connections_rotated, tcp_messages_sent, tcp_messages_received, tcp_deferred, tcp_unavailable, tcp_connection_closed, tcp_echo_mismatch, tcp_inflight_lost, tcp_drain_incomplete, tcp_write_blocked. Envoy's echo network filter on nighthawk_test_server is the target the e2e test uses.

UDP

scenarios:
  - name: dns-like
    target: udp://edge.example.test:5300
    body_file: query.bin
    concurrency: "2"
    udp:
      max_inflight: 64     # default 256
      timeout: 500ms       # a datagram unanswered this long is lost; default 1s
    executor: {type: constant-rate, rate: 4000, duration: 60s}
    thresholds:
      - "benchmark_udp.message_latency.p99 < 10ms"
      - "rate:benchmark.udp_lost < 1"

For Envoy's udp_proxy. One datagram socket per worker, connected to the target; each scheduled request sends the message as one datagram -- per worker, as for HTTP -- prefixed with a sequence number, and completes when a datagram carrying that number comes back (datagrams reorder and vanish, so the match is by number, not order), with the round trip as benchmark_udp.message_latency. A datagram unanswered for timeout is lost and counted in benchmark.udp_lost -- that, not a connection error, is what a proxy drops. method, headers, protocol, grpc, websocket and tcp are errors with a udp target; a body is required. Counters: benchmark.udp_datagrams_sent, udp_datagrams_received, udp_lost, udp_unexpected (a duplicate, a late echo, or not an echo), udp_send_errors, udp_deferred, udp_unavailable. nighthawk_test_server's udp-echo listener filter is the target the e2e test uses.

Anything this schema does not model

nighthawk_template on a scenario carries Nighthawk options straight through. sortie overlays only the fields it owns — requests_per_second, duration, execution_id, the rate-limiter plugin, and whatever the scenario's own fields set — so everything else reaches the backend as written.

scenarios:
  - name: passthrough
    executor: {type: constant-rate, rate: 100, duration: 30s}
    nighthawk_template:
      max_requests_per_connection: 1000
      burst_size: 5

This mirrors how upstream Nighthawk's adaptive load controller (not carried in engine/) takes a nighthawk_traffic_template, and it means sortie does not have to grow a field for every Nighthawk flag — transport sockets, request-source plugins, tunnelling and user-defined output plugins are all reachable without one.

Live metrics

The report arrives when a run ends. To watch a run while it is running, give the plan a stats block and the engine flushes its metrics to a statsd server:

stats:
  flush_interval: 5s
  statsd:
    address: "10.96.14.7:8125"   # an OpenTelemetry collector's statsd receiver

A stats block can sit at three levels, and the most specific one wins whole: the top of the plan, then defaults.stats, then a scenario's own. Each replaces the one before it rather than merging with it, so a scenario with no block gets defaults.stats when there is one and the plan's only when there is not. The report at the end is unaffected.

What goes out is everything the engine keeps in its Envoy stats store: the benchmark.* counters thresholds are written against, Envoy's own cluster counters (upstream_cx_total, upstream_rq_total, and so on), and every latency sample as a statsd timer in milliseconds. Names are

<prefix>.<scenario>.<backend>.cluster.<worker>.benchmark.http_2xx                   counter
<prefix>.<scenario>.<backend>.cluster.<worker>.benchmark_http_client.latency_2xx   timer, ms

prefix defaults to sortie. The scenario's label is lowercased and reduced to [a-z0-9_], with a / becoming a level of its own, so Live Metrics is sortie.live_metrics and stage 2 of a staircase named ramp is sortie.ramp.stage_2. That is what tells scenarios, stages and targets apart on a dashboard.

<backend> is the engine the metric came from. Every backend of a pool runs the same execution, and without it they would all write the same series, which a statsd server would show as one backend's worth. With it there is a series per engine. Counters are per worker: sum over workers for a backend's total, and over backends for the pool's, which is what the report's totals are.

What names the engine is stats.backend, and there are two answers:

stats:
  backend: name          # or address, the default
  statsd:
    address: "10.96.14.7:8125"

• address, the default: the engine's address without the port, sanitized the same way, so 10.0.0.11:8443 is 10_0_0_11, or host and port together when two engines share a host. Should two still read alike after that, as two IPv6 addresses can, each gets its position in the pool appended (_b0, _b1), so no two backends ever share a series. This is for pools sortie dispatches to itself, listed or found through DNS. Behind a distributor every target is sent the same options, so their metrics keep the prefix without a backend and land on one series.
• name: the name the engine was started with, nighthawk_service --backend-name, sanitized the same way (Node-A.example is node_a_example). An address is only as stable as what it belongs to. A pod's changes with every install and every restart, so each run of a DaemonSet of engines writes a whole new set of series into the store, and yesterday's run and today's cannot be laid over each other. A node's name does not change. sortie knows a backend only by its address, so it sends a placeholder (%BACKEND%) where the name goes and each engine puts its own in. That works behind a distributor too, whose targets then have a series each.

Two things follow from sortie not knowing the names. An engine started without one refuses the execution rather than emit the placeholder as a metric name, and the report shows that backend's error. And nothing can check that two engines of a pool end up named differently: two with one name write one series. "Differently" means after the name is reduced to [a-z0-9_], which is many-to-one: nodes called node-a and node.a are both node_a. So the names have to stay distinct once punctuation and case are gone, which most naming schemes satisfy and none guarantees. The chart supplies a name with engine.backendNameFrom: node (the node's name, right for a DaemonSet) or pod (the pod's name, unique among a Deployment's replicas but new whenever a pod is replaced), and engine.backendName is a literal for a single engine; it cannot check the result either. A name must be ASCII.

Three limits, each deliberate:

• The statsd address is an IP, not a name. Envoy's statsd sink does not resolve names, and a name would fail on the backend with the plan already dispatched, so the plan is refused instead. In Kubernetes use the Service's cluster IP.
• UDP only. Envoy's TCP statsd sink addresses a cluster of the bootstrap, and the engine writes its bootstrap itself.
• No OpenTelemetry sink. Envoy's OTLP sink aborts the engine on its first flush here: its gRPC client must be used on the thread that created it, and the engine creates sinks on its main thread and flushes them on another. To reach an OpenTelemetry collector, send statsd to its statsd receiver.

stats.sinks takes other Envoy stats sinks, written as they would be in an Envoy bootstrap, and passes them through, so the prefix scheme above does not apply to them. It reaches the sinks whose configuration is an envoy.config.metrics.v3 message -- dog_statsd, statsd, hystrix, metrics_service -- because sortie decodes the plan's typed_config and links those types only. A sink configured by an extension's own message, such as graphite_statsd, fails when the plan is parsed. The OpenTelemetry sink is refused there too, and so is the engine's own adapter: name the Envoy sink and sortie wraps it. A StatsdSink or DogStatsdSink written out there follows the same rules as the statsd block: an IP and a port, and no tcp_cluster_name. Its prefix is yours to write, and %BACKEND% in it is replaced with the engine's name as above, whatever stats.backend says.

Because the statsd prefix is built from sanitized labels, two scenarios can land on one: names that differ only in case or punctuation, or a scenario named ramp/stage-1 beside a staircase named ramp. A plan in which two statsd-reporting executions would share a prefix is refused; rename one, or give them different prefix values.

Thresholds

Thresholds are <metric> <op> <value> strings. Operators are <, <=, >, >=, ==, !=. Plan-level thresholds and scenario-level thresholds are additive; both must hold.

Selector	Meaning
latency_2xx.p95	A percentile of a statistic. Fractional percentiles like p99.9 work.
latency_2xx.mean	mean, min, max, pstdev or count.
counter:benchmark.http_5xx	A counter's absolute value.
rate:benchmark.http_2xx	A counter divided by the execution duration — the achieved rate.

Statistic and counter names match exactly or by dotted suffix, so latency_2xx resolves benchmark_http_client.latency_2xx. An ambiguous suffix is an error rather than an arbitrary pick.

Duration-valued metrics need duration values (500ms); counters and rates need plain numbers. Mixing them is an error, not a comparison against nanoseconds.

A counter that never incremented is absent from Nighthawk's output rather than present and zero, so a missing counter reads as zero — counter:benchmark.http_5xx == 0 passes on a clean run. A threshold naming a statistic that does not exist fails, because it went unchecked rather than being satisfied.

a run goes the distance

A failed request does not end a run. The engine's own defaults stop an execution the moment it counts one 4xx, 5xx, failed connection or reset, which suits a benchmark that means nothing once the target misbehaves. sortie turns them off: an execution lasts its duration, counts what goes wrong, and the thresholds judge it. One 503 three hours into a soak is a number in the report, not the end of that target's load.

A plan that does want an early stop asks for it, and what it asks for is left alone:

nighthawk_template:
  failure_predicates: {"benchmark.http_5xx": 100}   # stop past 100 of them

An execution the engine stops this way fails, and still reports what it had counted up to then.

a lost backend

Backends are independent. One that cannot be reached, or goes away during a run -- a node rebooting under an engine-per-node pool -- fails the execution and is named in the report, and the other backends carry on and are reported and judged as usual. Counter and rate thresholds are then judged against the survivors' totals, so a threshold on a total count will show the gap; the execution fails either way. Only when no backend returns anything is there nothing to report.

A backend can also be lost without a sound: a node that freezes or drops off the network sends nothing at all, and a connection can wait on it for ever. So every backend has a deadline, counted from dispatch: the execution's duration, plus every wait the plan asks of the engine after it -- the timeout, and the drain window of a gRPC stream, WebSocket, TCP or UDP run -- plus two minutes, and counted from its scheduled start when the plan gives it one. One that has not answered by then is cancelled and reported as silent, and the run ends with the others' results instead of hanging with none. The deadline is wall time, so a sortie that was itself stopped for a while -- its node frozen -- can wake past it with the results waiting: an answer that is there, or arrives within ten seconds of the deadline being noticed, is taken as the result. The same margin is applied to what a backend says of itself: a result that claims to have run far longer than planned -- a node frozen halfway and thawed later -- fails that backend too. Its numbers stay in the report, marked as not those of the plan.

failure classes

These are the counters a failed request shows up in. The finer ones refine the two older ones, stream_resets and pool_connection_failure, without changing them. They are not a partition: do not add them up to count failed requests. A unary gRPC call that fails is in grpc_error and also in whatever stopped it -- stream_resets for a reset, its HTTP status class for a non-2xx response.

Counter	A request that
benchmark.http_4xx, benchmark.http_5xx	got a complete response with that status
benchmark.grpc_error	was a unary gRPC call that ended with a status other than OK
benchmark.stream_resets	had its stream reset, in any way
benchmark.stream_resets_before_headers	was reset before the final response headers arrived; an informational response such as 103 Early Hints does not count as having arrived
benchmark.stream_resets_incomplete_body	was reset after the headers, before the body or trailers finished
benchmark.stream_resets_<reason>	was reset for that reason: Envoy's StreamResetReason in snake case, such as connection_termination, remote_reset, protocol_error
benchmark.pool_connection_failure	never got a connection, because connecting failed
benchmark.pool_failure_local_connection_failure, benchmark.pool_failure_remote_connection_failure	the same, by which side failed
benchmark.pool_failure_timeout	never got a connection, because connecting timed out
benchmark.pool_overflow	was refused by the client's own pool; in open loop this is saturation
benchmark.http_inflight_lost	was sent, or was queued for a connection, and had no outcome when the run was over: see below

For plain HTTP and unary gRPC requests the two phase counters sum to stream_resets, and so do the reason counters. The streaming modes (bidi-stream, WebSocket) count a reset of one of their long-lived streams in stream_resets alone. pool_failure_timeout is outside pool_connection_failure, which never included it.

A run does not end the instant its duration does. The requests that are in flight at that moment -- sent and unanswered, or still queued for a connection -- are waited for, for up to the scenario's timeout, and are counted and timed as what they turn out to be. That wait is not part of the reported execution duration, so it changes no rate. A request that is still without an outcome when the timeout has passed is counted in benchmark.http_inflight_lost, and in nothing else: it was not reset, and it has no status. With that counter no request that was issued is missing from the report. A run that is cancelled, or that a failure predicate ends, does not wait: its in-flight requests are counted as lost at once. A run that a termination predicate ends does wait, so its counters can pass the predicate's threshold by the requests that were in flight. The engine does not fail a run for http_inflight_lost; a threshold does, if the plan has one.

A zero-failure soak names every class, which costs nothing: a counter that never incremented reads as zero.

thresholds:
  - "counter:benchmark.http_4xx == 0"
  - "counter:benchmark.http_5xx == 0"
  - "counter:benchmark.grpc_error == 0"
  - "counter:benchmark.stream_resets == 0"
  - "counter:benchmark.pool_connection_failure == 0"
  - "counter:benchmark.pool_failure_timeout == 0"
  - "counter:benchmark.pool_overflow == 0"
  - "counter:benchmark.http_inflight_lost == 0"

Leave pool_overflow out and an open-loop run in which the client refused every request itself passes with nothing sent. Leave http_4xx out only if the target is meant to answer with one.

Three things are not failure classes, and why:

• A disconnect after a complete response. A server that closes the connection once the whole response has arrived -- an HTTP/3 server going away between requests does this -- is not a reset. The request has already been counted under its status, and none of Envoy's codecs raise a reset on a stream that is complete in both directions. It shows only in Envoy's connection counters, upstream_cx_destroy_remote. So stream_resets_incomplete_body is exactly the real mid-body cut.
• A response timeout. The engine has no per-request one: a request waits for its response until the run ends and for timeout after that, and is then in http_inflight_lost. timeout on a scenario bounds connecting and that final wait.
• A DNS failure. The target is resolved once, when an execution starts. A name that does not resolve fails the execution with an error, before any request.

The report prints a backend's non-zero failure counters beside its request count, and lists them under failures in JSON. The JSON also carries, per execution, results -- each backend's benchmark.* counters, elapsed time and statistics -- totals, the counters summed over the pool, and backend_errors for any backend that did not finish cleanly.

sent is how many requests were put on a connection to the target, answered or not: on each entry of results for that backend, and on the execution as their sum -- for a weighted scenario, one number per target. It is Envoy's upstream_rq_total, kept beside totals and not in them. A request counts from the moment it is given a connection, so one turned away by a full queue (pool_overflow) or still waiting for a connection when the run ended is not in it, and one whose connection broke while it was being written is. It is therefore the number to set against the target's own count of what it received, and an upper bound on it. The text report adds , N sent to a backend's line only when N is not the request count beside it. HTTP and unary gRPC only: the stream, TCP and UDP modes report what they sent in their own counters. It is absent when zero.

Each execution is placed in time, so a failed stage can be matched to what else was happening. started_at is when sortie began dispatching it and ended_at when its last backend had answered or been given up on, which is started_at plus elapsed_ms: RFC 3339, UTC, to the millisecond, by the clock of the machine sortie runs on. The text report gives the start on each execution's first line. The two bracket the load rather than time it: connecting to the backends happens inside them, and so does the wait of an execution the plan gave a scheduled start. Closer to when the load began is each backend's own started_at in results: when the first of its workers started its rate limiter's clock, by the engine's clock. That is just before a request is asked for, so it is not proof that one was sent; it is absent when the engine gave no timestamp.

statistics is each statistic that recorded anything, by id, with count, mean, pstdev, min, max, p50, p90, p99 and p99.9: in nanoseconds when unit is ns, plain numbers when it is raw. The percentiles are resolved as thresholds resolve them, so a p99 in the report is the one a threshold on that statistic was judged against. They are per backend only, for the reason given above: a pool-wide percentile cannot be had from the backends' own.

a verdict as each execution finishes

The report is written when the run ends. A run of hours should not have to end, or end well, before a stage that failed in its first hour is known, so every execution leaves two records the moment it finishes.

One is a line on stderr, beside the running line that announced it, with or without --progress:

  running steps/stage-40 (400 rps for 1m0s) on 10.0.0.1:8443, 10.0.0.2:8443
  PASS steps/stage-40 (scenario steps, 1m0.412s)
  running steps/stage-41 (410 rps for 1m0s) on 10.0.0.1:8443, 10.0.0.2:8443
  FAIL steps/stage-41 (scenario steps, 1m0.398s): latency_2xx.p95 < 50ms actual 12ms, 81ms

PASS or FAIL, the label, the scenario and the elapsed time; a failure adds what failed it, separated by ; : the execution's own error, each backend that did not finish cleanly, and each threshold that did not hold with the values observed (one per backend for a latency threshold, in the pool's order). Thresholds that held are left to the report. It is always one line, so grep '^ FAIL ' over a pod's log finds every failed stage so far.

The other is --results-stream FILE: each finished execution is appended to the file as one line of JSON (JSON Lines). A line is exactly the object that execution is in executions of the --json report -- same keys, same values, started_at and ended_at included -- so whatever reads the report's executions reads the stream's lines, and the stream of a run that finished holds the report's executions in order:

{"label":"steps/stage-41","scenario":"steps","pool":"nodes","rate":410,"duration_ms":60000,"elapsed_ms":60398,"started_at":"2026-10-07T12:40:07.118Z","ended_at":"2026-10-07T12:41:07.516Z","pass":false,"backends":["10.0.0.1:8443","10.0.0.2:8443"],"thresholds":[{"expr":"latency_2xx.p95 \u003c 50ms","scope":"per-backend","pass":false,"actual":"12ms, 81ms"}],"totals":{...},"results":[...]}

Each line is written whole, in one write, and synced to disk as its execution finishes, so a reader following the file can parse every line that ends in a newline, and what was written survives the pod. The targets of a weighted scenario finish together and are written together, in plan order. The file is appended to, never truncated: a run that is retried, or the next run given the same file, adds its lines after the ones already there, and started_at tells them apart. Delete or rename the file between runs to keep one run in it. A line that cannot be written is reported on stderr and the run carries on, as is a line that was written but could not be synced to disk; a file that cannot be opened is refused before any load, as are - and anything that is stdout or stderr under another name, since stdout carries the report, the report's own file under any name, and a file that has content this user cannot read back, since its end cannot be checked. A file that ends in part of a line, left by a run that died or by a write that failed half way, is not repaired: the part is closed off as a line of its own, which does not parse, and the lines after it are whole. A reader skips a line it cannot parse. The stream does not replace the report, which is still written at the end, and it has no overall pass: that is known only then.

percentiles resolve to the next histogram bucket

Nighthawk returns an HdrHistogram's own buckets, not round percentiles. A request for p95 resolves to the first bucket at or above 0.95, which is the same rule nighthawk_client applies to its own human-readable output.

what aggregates across a pool, and what does not

Counters sum, and counter and rate thresholds are judged against the pool total.

Latency statistics do not aggregate. A pool-wide p95 cannot be recovered from per-backend p95s, which is why Nighthawk's own sink service merges Outputs by appending results rather than combining them. sortie evaluates a percentile or statistic threshold against every backend individually and requires all of them to hold. For a single-backend pool this is exactly the obvious behaviour.

Limitations

What sortie drives, against what Envoy does, and how each piece was designed is in docs/parity.md: HTTP/1-3, gRPC, TLS, WebSocket, raw TCP and UDP are all in.

• No mid-run updates. A run's rate cannot be changed once started (UpdateRequest is declared in the engine's API and rejected). It can be stopped: interrupting sortie run cancels every backend's execution in a services: pool and the run is reported as cancelled, not evaluated. A distributor pool is the exception: the distributor RPC is abandoned and its targets run to their configured duration, because nothing here hosts a distributor that forwards cancellations.
• Progress is opt-in, and advisory. sortie run --progress 5s asks each backend for a snapshot of its run that often and prints a line per execution and backend on stderr: <label> <backend> <elapsed> http_2xx N ... mean X max Y. A snapshot carries the live counters and, for each latency statistic, its count, mean, standard deviation, min and max. It carries no percentiles, so the line has no p99: a percentile needs a copy of every worker's histograms (about 3 MiB each, 14 per worker), and those copies are what once made a snapshot cost hundreds of megabytes that the engine did not give back. A snapshot now allocates no histogram, and an engine's memory is the same with --progress as without. The engine's API can still be asked for full statistics per snapshot (StartRequest.progress_statistics), at that price; sortie does not ask. A snapshot is not free of CPU, though: each one asks every worker of every execution for its summaries. A user running 8 concurrent executions per engine measured about 0.6 CPU-seconds per engine per snapshot, which is 19 millicores averaged over a 30 s interval and 10 over 60 s. Under the WAIT idle strategy that is a visible share of an engine's whole cost, so pick the interval with it in mind. The verdict comes from the final response only. Backends behind a distributor report nothing until they finish.
• An engine's memory is its histograms. Every worker of every execution holds about 37 MiB of latency histograms for as long as it runs, and assembling the final report copies them. Measured on one engine running 8 executions at once, one worker each, 64 rps in total against a local target (an unoptimized x86-64 build): 123 MiB idle, 421 MiB while running, with or without --progress 1s, and a peak of 722 MiB for the instant the final reports are assembled. A user running 8 concurrent executions at 60 rps in total reported about 300 MiB steady and about 600 MiB at that peak. Budget for the peak, not the steady state: a memory limit below it kills the engine as the run ends. After a run the engine keeps part of that resident (340-400 MiB in the measurement above) until the next one reuses it.
• No scripting. Nighthawk's RequestSource yields independent requests and never sees responses, so there is no session flow — no login, capture a token, reuse it. Scenarios are stateless load.
• Scenarios run one after another. Only the targets of one scenario run at the same time (see Weighted targets); two scenarios never overlap. nighthawk_service runs as many executions at once as --max-concurrent-executions allows, one by default, and refuses the rest.

Running it in Kubernetes

Two multi-arch images and a Helm chart are published to Quay on every push to main:

quay.io/sortie/sortie               linux/amd64, linux/arm64
quay.io/sortie/engine               linux/amd64, linux/arm64
oci://quay.io/sortie/chart-sortie   the chart

Commits before 2026-10-04 were published to ghcr.io/bpalermo/sortie, ghcr.io/bpalermo/sortie/engine and oci://ghcr.io/bpalermo/sortie/charts/sortie, and stay there; nothing newer is pushed to them. Where things are published is one setting, bazel/registry.bzl.

The engine image carries nighthawk_service (its entrypoint), nighthawk_test_server and nighthawk_client, built from engine/ in this repository at the same commit as sortie. Each architecture is built natively and the two are stitched into one index. With engine.enabled=true the chart also deploys nighthawk_service behind a Service named <release>-sortie-engine on port 8443, for a plan's services: to name.

The chart runs a plan as a Job, or as a CronJob with cronJob.enabled=true. The plan is held in a ConfigMap and mounted read-only, so changing a run does not mean republishing anything. The Job's exit code is the verdict: a breached threshold fails it, and a malformed plan fails it differently.

One loader per node

To load every node's proxy from that node -- a service-mesh soak -- run the engine as a DaemonSet and let the plan find its pods:

# The engines first, on every node, and wait for them all.
helm install nightly oci://quay.io/sortie/chart-sortie --version <chart version> \
  --namespace loadtest --values values.yaml --set job.enabled=false
kubectl --namespace loadtest rollout status daemonset/nightly-sortie-engine

# Then the run.
helm upgrade nightly oci://quay.io/sortie/chart-sortie --version <chart version> \
  --namespace loadtest --values values.yaml --set-file plan=plan.yaml

Two steps, because the plan's name is resolved once, when the run starts, and the first non-empty answer is taken: a Job installed together with the DaemonSet would drive whichever engine pods happened to be ready first and leave the other nodes out for the whole soak.

# values.yaml
engine:
  enabled: true
  kind: DaemonSet                       # behind a headless Service, <release>-sortie-engine-nodes
  backendNameFrom: node                 # live metrics are keyed by node, not by pod IP
  priorityClassName: loadgen            # a class you create; see below
  tolerations: [{operator: Exists}]
  podAnnotations: {mesh.example.com/inject: "true"}
  resources:
    requests: {cpu: 500m, memory: 256Mi}
podAnnotations: {mesh.example.com/inject: "true"}   # the sortie pod itself

# plan.yaml
version: v1
pools:
  - name: nodes
    dns: nightly-sortie-engine-nodes.loadtest.svc.cluster.local:8443
stats:
  backend: name                         # the engine's node; see Live metrics
  statsd:
    address: "10.96.14.7:8125"
scenarios:
  - name: soak
    pool: nodes
    target: http://orders.shop.svc.cluster.local:8080/
    executor:
      type: constant-rate
      rate: 60
      duration: 8h
      per_backend: true
      open_loop: true
    thresholds:
      - "counter:benchmark.http_5xx == 0"

To watch the soak node by node, the plan sends live metrics with backend: name and the engines take their names from their nodes (engine.backendNameFrom: node). The series are then sortie.soak.<node>.cluster..., the same ones on every run; named by address they would be keyed by pod IP and start over with every install.

The same series on every run means each run's numbers follow the last run's on it. The engine counts every run from zero, but what it sends is what a counter gained since the last flush, so a receiver that keeps a running total adds the next run to it: a series it has kept does not drop at a run boundary. It starts again from zero only when the receiver had expired the series between runs, or was itself restarted. Either can happen, so read counters across runs with rate() or increase(), which take a drop as a reset, and do not take a total to be one run's count.

The engine is the pod that opens the connections, so it is the one a mesh has to inject; engine.podAnnotations and engine.podLabels are for that, and the same pair at the top level is for the sortie pod, which a mesh treats like any other.

A DaemonSet pod cannot move, so when its node fills up it is evicted rather than rescheduled and the run loses that node. engine.priorityClassName names a PriorityClass above the workloads under test, which makes one of them the eviction victim instead. The class is cluster-scoped and not the chart's to own, so create it yourself.

A CronJob has the same property for free: by the time its schedule fires, the engines have long been up.

The report is the pod's log unless you keep it. report.path with a report.volume writes it to a file on a volume of your choosing, as JSON by default, and the readable summary still goes to the log:

report:
  path: /var/run/sortie/report.json
  volume:
    persistentVolumeClaim: {claimName: soak-reports}
podSecurityContext: {runAsNonRoot: true, fsGroup: 65532}

report.stream with a file name, say results.jsonl, adds --results-stream: each execution's result is appended to that file, beside the report on the same volume, as the execution finishes (see a verdict as each execution finishes). For a run of hours it is the part of the report that exists if the pod does not live to write the rest.

The next run on the same volume and path overwrites the report, and appends to the stream. The volume is mounted at the file's directory, so give the file a directory of its own: a path directly under / or /etc, or at or under /sortie or /etc/sortie, is refused.

The Job and the engine come up together, so sortie waits up to 30 seconds for each backend to accept connections before the run starts; a backend that is not there by then fails the run with the address it tried. test/chart/run.sh installs the chart on a kind cluster with both images built from the checkout and checks both verdicts; CI runs it on every change.

The Job's name carries a digest of its rendered pod template, because a Job's spec.template is immutable: with a stable name, helm upgrade with a changed plan would fail with field is immutable rather than run it. With the suffix an upgrade creates a new Job and Helm removes the previous one. Changing nothing therefore re-applies the same Job rather than re-running it; to run an unchanged plan again, delete the Job. Changing only backoffLimit or ttlSecondsAfterFinished does not rename it -- those are mutable on a Job, so they patch the existing run instead of starting a new one.

job.ttlSecondsAfterFinished defaults to an hour, and deleting the Job is also what re-runs it: once the TTL has removed it, the next helm upgrade finds it missing and creates it again, generating load even if the upgrade changed nothing about the run. Set it to null to keep finished Jobs until something deletes them.

Upgrading a release while a run is in flight replaces the Job. The old pod is sent SIGTERM, on which sortie cancels its backends' executions and waits for their partial responses (seconds), and the old run is reported as cancelled, not evaluated. Nothing orders the replacement after that, though: Kubernetes can schedule the new pod while the old one is still cancelling, and a backend runs one execution at a time, so the new Job can find it busy and fail -- with backoffLimit: 0 it does not retry. Upgrading between runs avoids the race; so does a backoffLimit of 1 or 2, at the cost of re-running a plan that failed for a real reason.

The ConfigMap is named after the plan's digest for the same reason in reverse. A stable name would be updated in place, and a CronJob's Job that was created before an upgrade but had not started yet would mount the new plan while reporting itself as the old one. Each plan gets its own object, so a Job can only mount the plan it was created for.

helm install nightly oci://quay.io/sortie/chart-sortie \
  --version <chart version> --set-file plan=plan.yaml

The default plan in values.yaml names no backend that exists. That is deliberate: a plausible-looking default would generate load against whatever happened to answer.

Provenance

Images are signed with cosign, keyless, through GitHub's OIDC identity — there is no key to store or rotate. Signatures cover the index and every per-arch manifest, so pulling by an architecture-specific digest is covered too:

cosign verify \
  --certificate-identity-regexp '^https://github\.com/bpalermo/sortie/\.github/workflows/publish\.yml@refs/heads/main$' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  quay.io/sortie/sortie@sha256:...

cosign verify has no --recursive, so that checks one digest. From a checkout, bazel run //bazel/cosign:verify_image -- quay.io/sortie/sortie@sha256:... verifies the index and every manifest under it with the cosign this repository pins, and is what the publish workflow runs.

A published chart pins both images by digest: the driver's, injected at package time from the push target, and the engine's, the index the same workflow run pushed and signed just before. So a chart can only ever reference the images built alongside it, and those are the ones whose signatures were verified. A chart built from a checkout names the engine by the commit's tag instead.

An image's digest names its content and nothing else: no label in the image records the commit, so a commit that does not change an image publishes the digest the one before it did, and a consumer who pins by digest has nothing to re-pin. The engine image is the usual case, since most commits leave the engine alone. The driver's digest moves with every commit, because the binary reports its own version. Which commit published a digest is answered outside the image: every commit's dev-<commit> tag names what that commit published, and the certificate of the signature names the commit that first published it (cosign verify prints it as githubWorkflowSha). A digest that is already signed is not signed again.

The chart itself is signed the same way, as a single manifest:

bazel run //bazel/cosign:verify_image -- --single quay.io/sortie/chart-sortie@sha256:...

Verification needs cosign 3 or newer. Signatures are sigstore bundles attached to the manifest they sign as OCI 1.1 referrers; there is no sha256-<digest> tag. A cosign 2 client does not find them and reports no signatures found, which is indistinguishable from an unsigned image, so check the version before concluding anything from that. (Images on ghcr.io, from before the move, carry their signatures on sha256-<digest> tags instead, because that registry does not serve the Referrers API.)

The plan schema

The schema is a protobuf definition, api/sortie/plan/v1/plan.proto. Field names in a plan file are the field names there, and most constraints are declared inline with protovalidate so a rule sits next to the field it governs.

Plans are parsed with protoyaml, which reports violations with a line and column and the offending line:

plan.yaml:11:7 scenarios[0].executor: the ramping-rate executor requires a ramp_time
  11 |       type: ramping-rate
  11 | ......^

Unknown fields are rejected, so a misspelled key fails the run rather than being silently ignored.

Two kinds of rule cannot live in the schema and stay in Go: those spanning messages (a scenario naming a pool declared elsewhere in the file) and those depending on dispatch (a rate divisible by backends x concurrency, which needs the pool). Threshold expressions are also checked at load time, since their grammar belongs to sortie rather than to protobuf.

Durations follow protobuf's own JSON mapping — decimal seconds ending in s, so 90s rather than Go's 1m30s.

Attribution

engine/ is a fork of envoyproxy/nighthawk, Apache License 2.0; its LICENSE is kept in that directory and its README says what was changed. The rest of this repository is under the root LICENSE.

Protos

The engine's API protos live in engine/api and are compiled by Bazel for both C++ and Go, so the control plane and the engine always agree on one definition. Nothing is vendored and no generated code is checked in, so the bindings cannot go stale against the .proto files they came from.

Envoy's types come from the envoy_api module rather than from the go-control-plane Go module, because that is where the generated Nighthawk bindings get theirs. Two targets carrying the same Go import path would put a second copy of every Envoy message in the binary and panic at init on duplicate registration. The same applies to google/rpc/status.proto, whose Go code is taken from the genproto module gRPC-Go already links.

Linting

aspect_rules_lint runs buildifier over the hand-written Starlark as an ordinary test target, so bazel test //... covers it.

Go is analysed by rules_go's nogo (//:nogo), which runs as part of compilation, so a vet finding fails the build rather than a separate job. aspect_rules_lint ships no Go linter, which is why the two are split.

nogo on a Go 1.27 SDK needs golang.org/x/tools >= v0.48.0, and go_deps resolves one x/tools across every module by MVS, so this repository's go.mod is what the analyzers are built against. //bazel/deps:deps_test fails if that floor is breached, because the symptom otherwise is an export-data error that never mentions x/tools.