diff --git a/cmd/benchmarking/egress/README.md b/cmd/benchmarking/egress/README.md new file mode 100644 index 0000000000..ceea5f69f8 --- /dev/null +++ b/cmd/benchmarking/egress/README.md @@ -0,0 +1,225 @@ +# benchmark-egress + +`benchmark-egress` is the workload of the egress scale test. It holds many +CONNECT tunnels open through the egress gateway at a fixed pace and reports what +each tunnel saw, so that a run can price what one tunnel costs the gateway. + +## 1. What the benchmark measures + +The benchmark sizes one `atenet-egress` replica for a target fleet: A actors, +each reaching C destinations. It reports what one tunnel and one request cost +the gateway, so that the replica count for a fleet follows from fleet size, +node shape, and these costs. + +Three terms: + +- **Tunnel.** One kept-alive CONNECT from one actor to one destination + Service, opened by the actor's first request and reused by every later one. + It is the unit the gateway's cost scales with: a fleet of A actors over C + Services holds A × C tunnels. +- **Blended.** The window from the first actor's start until the last actor + has opened its last tunnel. New tunnels are being set up while tunnels + already open carry repeat requests, so the gateway's CPU mixes setup with + ordinary requests. +- **Settled.** The period, after the blended window, in which every tunnel + has been reused and every request reuses one. Nothing is set up; the + gateway's cost is per request. + +The settled period gives the cost of one request directly. The blended window +gives the cost of setting up one tunnel once its requests are priced at the +settled rate. + +A cell is one fleet size and one protocol. It reports: + +- **Settled req/s per core.** Throughput per gateway core once every tunnel + exists. Its inverse is the CPU cost of one request. +- **Gateway CPU per tunnel opened.** CPU milliseconds to set up one tunnel, the + cost of its requests taken out. +- **Envoy heap per tunnel.** KiB of heap one held tunnel adds. +- **Settled p50, p99, p99.9 and success rate.** Latency of repeat requests as + the actor sees it, and the share that got HTTP 200. Flat across fleet sizes + means the gateway was not saturated while the costs were read. + +The costs price held tunnels, not tunnels that reconnect. Section 4 lists the +checks a cell must pass before its numbers count. + +Per-core and per-tunnel figures do not depend on the node size or the replica +count. Memory sizes as tunnel count times heap per tunnel, against one node's +memory. CPU sizes as request rate times CPU per request, plus the rate of +tunnel opens times CPU per tunnel opened, against one node's cores. The +replica count for a fleet is the larger of the two. + +## 2. Topology + +```mermaid +flowchart LR + D[driver] --> A[actor] + A --> G[egress gateway] + G --> S0[Service 0] + G --> S1[Service 1] + G --> S2[Service 2] + S0 --> T[target Deployment] + S1 --> T + S2 --> T + G --> M[collector] + M --> P[Prometheus] + D --> P +``` + +- **`benchmark-egress actor`** runs inside each Actor. The driver controls it + through the atenet router on port 80: `/readyz`, `/start`, `/stats`, + `/stop`. Once started, it holds one CONNECT tunnel per endpoint through the + gateway and sends a `GET` on each at a fixed pace, and it exports what it + measures as OpenTelemetry metrics over OTLP, through the same gateway, to the + collector named by its environment. The actor's egress policy must name every + endpoint hostname and the collector. The actor's README has + [the workflow](internal/cmd/actor/README.md#1-what-the-actor-does) and + [the metrics](internal/cmd/actor/README.md#2-what-the-actor-reports). +- **Egress gateway.** One `atenet-egress` replica in its sdsmint deployment, + the system under test. It carries one CONNECT tunnel per actor and + destination, plus one per actor to the collector, terminates the actor's TLS + on HTTPS runs, and checks the egress policy on every request. On HTTPS runs + the actor trusts the gateway's MITM CA through `SSL_CERT_FILE`, and the + gateway re-originates TLS to the target and must trust the target CA. +- **Endpoints.** C ClusterIP Services, `egress-target-0` to + `egress-target-`, in namespace `egress-tests-targets`. Each is a + distinct hostname, so each is its own tunnel. +- **`benchmark-egress target`** runs as one Deployment behind all C Services. + It answers every `GET /` with a 200 and a small fixed body, over plain HTTP + and, given a certificate, HTTPS. Each Service maps to its own backend port, + set with `--listen-ports` and `--tls-listen-ports`. The driver issues the + HTTPS certificate for `*.egress-tests-targets.svc.cluster.local` from a + throwaway CA that is name-constrained to + `egress-tests-targets.svc.cluster.local`. +- **Collector and Prometheus.** An OpenTelemetry collector that receives the + actors' OTLP on port 4317 and serves it as a Prometheus exposition, and a + Prometheus that scrapes it and the gateway node's cAdvisor. Both belong to + the benchmark's deployment, not to the actor. The actor only needs the + collector's address. +- **Driver.** Sets up the run, starts the A actors, posts `/start`, polls + `/stats` for the gates, records the instants that bound each window, and + reads every result from Prometheus with the queries of section 4. + +## 3. Where to read next + +| Where | What it holds | +|---|---| +| [The actor's README](internal/cmd/actor/README.md) | What the actor does and reports: the contract a driver drives it through. | +| [The `contract` package](internal/contract/doc.go) | The Go form of that contract, shared by the actor, the target and the driver. | +| `benchmark-egress target` | The topology above is all it needs: one Deployment behind every Service, one backend port each, and the certificate it serves. | +| The deploy script and the telemetry stack | In `benchmarking/egress-tunnels/`, a later PR. | +| The driver | In this directory, a later PR. | + +## 4. What a run computes + +The driver is the next PR. This section is the contract it implements. + +The driver polls every actor's `/stats` for the gates and records four +instants: `t0`, the first `/start`; `t1`, when every actor's `opened` is C, +the end of the blended window; `t2`, when every actor's `rounds` is at least +two and `newConns` is flat between two polls, the start of the settled window; +`t3`, `t2` plus the cell's duration, when it posts `/stop`. Every result is a +difference between two instants, read from Prometheus with the instant query +API, `time=` set to the later instant and the earlier one as an `@` anchor, so +the query is about the instants and not about when the driver asked. The +driver reads at `t` plus a lag of one export interval, one scrape interval and +a margin, so the samples at `t` exist. + +Series selector `S` is `{job="benchmark-egress-actor", cell="", run=""}`. +The merge across actors happens in the query: + +| Rule | Series | +|---|---| +| `sum` of `@`-differences | `requests_total`, `request_errors_total`, `connections_opened_total`, `endpoints_opened_total`, `latency_seconds_count`, `latency_seconds_sum` | +| `sum by (le)` of `@`-differences, then `histogram_quantile` | `latency_seconds_bucket`, `opening_seconds_bucket` | +| `max` | `opening_done_seconds`, `longest_gap_seconds` | +| `min` | `rounds` | +| `count by (actor)` | presence: every actor's series exist at both instants | + +| Result | Query at `time=t_end` | +|---|---| +| Requests in a window | `sum(requests_total S) - (sum(requests_total S @ t_start) or vector(0))` | +| Success rate | the same pair with `result="success"`, over the total | +| Settled p50, p99, p99.9 | `histogram_quantile(q, sum by (le) (latency_seconds_bucket S) - (sum by (le) (latency_seconds_bucket S @ t_start) or sum by (le) (latency_seconds_bucket S) * 0))` | +| Blended end | per actor, its `/start` time plus `opening_done_seconds`; the latest is `t1` | +| Gateway CPU in a window | `sum(container_cpu_usage_seconds_total{pod=~"atenet-egress.*", container=~"envoy|ext-proc|sdsmint"})` as the same `@`-difference, from the cAdvisor scrape of the gateway's node | +| Presence | `count(count by (actor) (requests_total S)) == A` at `t2` and `t3` | + +`rate` and `increase` are not used for results: they extrapolate over the +range and lose the first sample. They serve only the collector's own counters. + +The results of a cell come from two windows. The blended window runs from the +first `/start` to the latest `openingDone`, each actor's shifted by its own +`/start` time. An actor's `openingDone` counts from its `/start`, so each +actor's window also covers its `offsetMs`. The settled window starts once +every actor has opened, every actor's `rounds` has reached two, checked per +actor before the sum, and `newConns` is flat between two polls. Two actors and +two endpoints, 100 ms apart, both started at the same time, actor 2 with a +50 ms offset and a slow second open: + +```mermaid +sequenceDiagram + autonumber + participant A1 as Actor 1 + participant A2 as Actor 2 + participant S0 as Service 0 + participant S1 as Service 1 + rect rgb(240, 248, 255) + A1->>+S0: open (t=0ms) + S0-->>-A1: done (t=20ms) + A2->>+S0: open (t=50ms, offset) + S0-->>-A2: done (t=70ms) + A1->>+S1: open (t=100ms) + S1-->>-A1: done (t=120ms) + A2->>+S1: open, SLOW (t=150ms) + Note over A1,S1: Repeat requests begin while the slow open is still pending + A1->>S0: repeat (t=200ms) + A2->>S0: repeat (t=250ms) + A1->>S1: repeat (t=300ms) + S1-->>-A2: latest openingDone (t=320ms) + end + Note over A1,S1: Settled, after the blended window + A2->>S1: repeat (t=350ms) + A1->>S0: repeat (t=400ms) + A2->>S0: repeat (t=450ms) + A1->>S1: repeat (t=500ms) +``` + +Both actors' first repeats fall inside the blended window, because it ends +when the slowest actor finishes its last open. The offset puts actor 2's sends +halfway between actor 1's, so the two together send one request every 50 ms +rather than two every 100 ms; without it every send of actor 2 would coincide +with one of actor 1's. The gateway's CPU in that window paid for four tunnel +setups and for every request in it; the settled rate prices every request, the +first on each tunnel included, and what is left is the setups. `t1` is the +edge the driver reads the gateway's CPU against, and the `@`-difference of +`requests_total` over `t0..t1` is the window's request count; +`requestsAtOpeningDone`, summed from `/stats`, is the count at each actor's +own opening end, for the opening alone. + +| Result | Source | How it is computed | +|---|---|---| +| Settled req/s per gateway core | actors + gateway | Requests completed in the settled window divided by the gateway pod's CPU-seconds in the same window, both as `@`-differences over `t2..t3` from the same Prometheus. | +| Gateway CPU per tunnel opened | actors + gateway | The blended window's gateway CPU, minus every request in it priced at the settled per-request cost, divided by the tunnels opened. What remains is setup alone, independent of pacing, on the assumption that a request on an open tunnel costs the same in either window. Computed after the settled window, since it needs the settled rate. | +| Envoy heap | gateway | The maximum of `server.memory_allocated` in the cell; per-tunnel cost is the slope across fleet sizes. | +| Settled p50, p99, p99.9 | actors | `histogram_quantile` on the settled window's bucket differences, reported as interpolated with a bucket resolution of 12%. | +| Success rate | actors | `result="success"` over all `requests_total` in the window. | +| Run validity | actors + gateway | `endpoints_opened_total` equals A × C and `connections_opened_total` does not move over `t2..t3` (a first request that failed counts its endpoint as opened and dials its tunnel one period later, so the two counters need not be equal); `max(longest_gap_seconds)` stays under the stream idle timeout; `dns_seconds_count` and `tls_seconds_count` do not move over `t2..t3`; presence is A at `t2` and `t3`; `cluster.mitm_internal.upstream_cx_total` equals A × (C + 1) and stays flat; `http.egress_connect.downstream_rq_idle_timeout`, `downstream_cx_idle_timeout` and `downstream_cx_protocol_error` do not move; the collector's `otelcol_receiver_refused_metric_points` does not move. | + +Three limits of this design are accepted and stated here rather than worked +around: + +- **Instant reads lag.** A sample at `t` holds what the actor exported up to + one export interval before `t`, scraped up to one scrape interval later. In + a difference between two instants of a steady window the lag is the same at + both ends and cancels; measured on GKE, a 300 s settled window agreed with + exact cuts within 0.1% of requests. Where the rate changes between the + instants, at the start, the stop, or a restart, the error is up to export + plus scrape interval per end. +- **A restart inside a window invalidates the cell.** A `requests` that went + down between two `/stats` polls, or a 404, means the actor restarted. Its new + instance starts from zero under the same `actor`; the old instance's series + stop and expire from the collector after its expiration time. The driver + marks the cell invalid rather than reasoning about the overlap. +- **No maximum.** The histogram carries no exact maximum; the largest latency + is known to one bucket step. diff --git a/cmd/benchmarking/egress/internal/cmd/actor/README.md b/cmd/benchmarking/egress/internal/cmd/actor/README.md new file mode 100644 index 0000000000..c8f7bb9cfd --- /dev/null +++ b/cmd/benchmarking/egress/internal/cmd/actor/README.md @@ -0,0 +1,205 @@ +# benchmark-egress actor + +`benchmark-egress actor` is the workload inside each Actor of the egress scale +test; the [overview](../../../README.md) has what the benchmark measures and the +topology the actor runs in. + +## 1. What the actor does + +The actor is idle or running. It boots idle and sends nothing until `/start`: +every actor restores from a golden snapshot, and a loop started at boot would +have run in the golden actor, which has no egress policy. A restarted actor is +idle again with nothing kept. + +| Route | Idle | Running | +|---|---|---| +| `GET /readyz` | 200 | 200 | +| `POST /start` | 204 | 409 | +| `GET /stats` | 404 | 200, [section 2](#2-what-the-actor-reports) | +| `POST /stop` | 200 with the last loop's final counters, 404 if none | 200 with the final counters once every goroutine has exited and the last export is out | + +Every answer carries `Benchmark-Egress-Protocol: 1`; the driver requires that +value, and the header is what lets a later contract be refused. + +`POST /start` takes one JSON object, at most 1 MiB; unknown fields are ignored. + +| Field | Meaning | 400 when | +|---|---|---| +| `endpoints` | C, the number of endpoints. | Missing, under 1, or over 1024. | +| `intervalMs` | Pacing. Endpoint *i* opens *i* intervals after `/start`; each endpoint is then requested once per period of C × `intervalMs`. Zero sends every endpoint's requests back to back. | Negative. | +| `offsetMs` | Delay before endpoint 0 opens; endpoint *i* opens `offsetMs` + *i* × `intervalMs` after `/start`. Lets a driver give each actor its own phase, so a fleet started at once does not send in lockstep; see below for how a driver picks it. | Negative, or more than C × `intervalMs`. | +| `scheme` | `http` or `https`; empty means `http`. | Any other value. | +| `requestTimeoutMs` | Bound on one request, handshake included; zero means 5000. | Negative. | +| `actor` | This actor's name, `/`. An attribute on every metric the loop exports; the fleet's series are told apart by it. | Missing. | +| `cell` | The cell this loop belongs to. An attribute on every metric. | Missing. | +| `run` | The run the cell belongs to. An attribute on every metric; a rerun of a cell gets a new `run`. | Missing. | + +```json +{"endpoints": 1000, "intervalMs": 100, "scheme": "https", + "actor": "benchmark-workloads/egress-0017", "cell": "1000x1000-https", "run": "20261022-1"} +``` + +204 means the goroutines exist, not that any request has been sent. A 400 +carries a one-line reason. `/start` does not probe the endpoints or the +collector: a missing egress policy or Service shows up as errors in `/stats`, +not as a failed `/start`, and a collector that cannot be reached shows up as +absent series in Prometheus. The instruments are created at `/start` and +released at `/stop`, so every counter counts from `/start` and a later `/start` +begins from zero under a new instance id. + +The loop then runs one goroutine per endpoint: + +1. Wait offset + *i* × interval, so C handshakes do not reach the gateway at + once. +2. Send the first `GET` to the endpoint. When it completes, success or not, the + endpoint counts as opened. When the last endpoint opens, the actor stamps + `openingDone`. +3. Send one `GET` per period on a ticker started when the open starts. At most + one request is in flight per endpoint. A request that outlasts the period + delays only its own endpoint, whose next request starts as soon as the slow + one returns. +4. Stop on `POST /stop`, which waits for every goroutine, flushes the last + export, and answers with the final counters of section 2. A request cut off + by the stop is not counted. The actor keeps those counters until the next + `/start`, so a `/stop` whose answer was lost is asked again and answered + again. + +The client keeps one connection per endpoint and never closes it for idling, +so a redial, one more in `newConns`, means the gateway cut the tunnel or a +request on it timed out. The gateway resets a tunnel that stays quiet for its stream idle +timeout, 5 minutes by default, so keep the per-endpoint period, C × +`intervalMs` between two requests on the same endpoint, under it. + +Only an actor's phase modulo `intervalMs` matters to the fleet: once started +it sends at `/start` + `offsetMs` + k × `intervalMs`. A driver gives actor k +of A `offsetMs` = ⌊k × `intervalMs` / A⌋, which spreads the phases evenly; a +uniform draw from [0, `intervalMs`) does about as well but needs its seed +recorded. Four actors with `intervalMs` 100: + +| Offsets | Fleet sends in each interval | +|---|---| +| 0, 0, 0, 0 | 4 at 0 ms | +| 0, 25, 50, 75 | 1 at 0, 25, 50, 75 ms | + +The offset lives on the actor because the router's fan-out of `/start` sets +each actor's real phase, and the driver does not control it. + +With these rules an actor sends one request per interval throughout the run, +and holds C tunnels. + +## 2. What the actor reports + +The actor reports on two paths. `GET /stats` answers one small JSON object, +the counters cumulative since `/start`, for a driver's gates and progress: +poll it, it consumes nothing. Everything a result is computed from, the same +counters and the four latency distributions, the actor exports as +OpenTelemetry metrics over OTLP to the collector, once per export interval, and +the driver reads them from Prometheus. The actor keeps no window state: it has +no cut to serve and nothing to replay, so a lost export costs one interval of +staleness and nothing else. + +### 2.1 `GET /stats` and `POST /stop` + +Durations are Go `time.Duration` values, nanoseconds. The per-endpoint list +comes only with `?endpoints=1`, on either route. + +| Field | What it holds | Used for | +|---|---|---| +| `scheme` | `http` or `https`. | The scheme in use. | +| `elapsed` | Time since `/start`. | Rates between two polls. | +| `requests`, `successes` | Requests finished; HTTP 200 with no error is a success. | Progress; a `requests` that went down means a restart. | +| `newConns` | Requests sent on a freshly dialed connection. | One per endpoint is the tunnel. Any more is a reconnect after the gateway cut a tunnel or a request on it timed out. | +| `opened` | Endpoints whose first request has completed. | Progress of this actor's opens; the gate on C. | +| `openingDone` | Elapsed when the last endpoint opened; zero until then. | Added to this actor's `/start` time, the latest across actors ends the blended window. | +| `requestsAtOpeningDone` | `requests` at that same moment; zero until then. | This actor's request count for its own opening, section 4 of the [overview](../../../README.md#4-what-a-run-computes). | +| `rounds` | Requests completed by the least-visited endpoint. | Two means every tunnel has been reused; the settled gate. | +| `longestGap` | Longest time between two consecutive requests on one endpoint. | Whether any tunnel went quiet long enough to be reset. | +| `errors` | Failures by class, such as `timeout`, `connection reset`, `HTTP 403`; one entry per class seen. | One class per failure mode. | +| `endpoints` | Per endpoint, by number: `requests`, `errors`, `longestGap`. `?endpoints=1` only. | Naming a failing or quiet tunnel. | + +At C = 1024 a `/stats` is about 250 bytes and the endpoint list about 60 KB +more. + +```json +{ + "scheme": "https", "elapsed": 612000000000, + "requests": 61440, "successes": 61437, "newConns": 1024, "opened": 1024, + "openingDone": 102400000000, "requestsAtOpeningDone": 1536, + "rounds": 60, "longestGap": 102600000000, + "errors": {"timeout": 3} +} +``` + +### 2.2 Metrics + +The loop records with these instruments. Names are chosen so that the +collector's Prometheus exporter serves exactly the series the queries of +section 4 of the [overview](../../../README.md#4-what-a-run-computes) read: +a counter gains `_total`, a unit of seconds gains `_seconds`, a histogram +gains `_bucket`, `_sum` and `_count`. + +| Instrument | Kind, unit | Attributes | Prometheus series | What it holds | +|---|---|---|---|---| +| `requests` | counter | `result` = `success` or `error` | `requests_total` | Requests finished, by outcome. | +| `request_errors` | counter | `class` | `request_errors_total` | Failures by class, as `errors` in `/stats`. | +| `connections_opened` | counter | | `connections_opened_total` | Requests sent on a freshly dialed connection, as `newConns`. | +| `endpoints_opened` | counter | | `endpoints_opened_total` | Endpoints whose first request completed, as `opened`. | +| `rounds` | gauge | | `rounds` | Requests completed by the least-visited endpoint. | +| `opening_done` | gauge, s | | `opening_done_seconds` | Seconds from `/start` to the last open; zero until then. | +| `longest_gap` | gauge, s | | `longest_gap_seconds` | Longest gap between two requests on one endpoint. | +| `opening` | histogram, s | | `opening_seconds_*` | Successful first requests: tunnel setup latency. | +| `latency` | histogram, s | | `latency_seconds_*` | Successful repeat requests: the settled quantiles, no handshakes in them. | +| `dns`, `tls` | histogram, s | | `dns_seconds_*`, `tls_seconds_*` | Lookup and handshake times, when a request did one. Both stop moving once every tunnel is up. | + +Every point carries `actor`, `cell` and `run` from `/start`. The resource is +`service.name` = `benchmark-egress-actor`, the actor's default unless +`OTEL_RESOURCE_ATTRIBUTES` sets another, which the exporter writes as `job`, +and `service.instance.id`, which the actor draws at `/start`, so a restarted +actor is a new `instance` and never continues the old series. The per-endpoint +rows are not exported: at C = 1024 across a thousand actors they would be three +million series. They stay on `/stats?endpoints=1` for a person debugging one +actor. + +The histograms share one set of bounds, 20 per decade from 1 µs to 63.1 s, +157 values, the same in every actor and every image, so `sum by (le)` across +the fleet is meaningful. Neighboring bounds differ by about 12%, which sets the +resolution: a quantile is within one bucket step of the true value, and two +runs whose quantiles fall within one step are indistinguishable. There is no +exact maximum; the overflow bucket is the last. + +One actor exports 646 series, about 90 KB of protobuf per interval. A fleet of +a thousand at the default interval is 646 thousand series and 130 thousand +points per second into the collector, and one more CONNECT tunnel per actor +through the gateway. + +### 2.3 Export + +The actor is configured by the standard OpenTelemetry environment, set on the +ActorTemplate: + +| Variable | Value | Meaning | +|---|---|---| +| `OTEL_EXPORTER_OTLP_ENDPOINT` | `http://:4317` | Where to push, OTLP over gRPC, plaintext. Unset means the loop exports nothing and `/start` still works. | +| `OTEL_RESOURCE_ATTRIBUTES` | `service.name=benchmark-egress-actor` | The `job` the series are stored under. Optional: this value is the actor's default, and the variable overrides it. | +| `OTEL_METRIC_EXPORT_INTERVAL` | `5000` | Milliseconds between exports; the default of the SDK is 60000, too coarse for a window of a few minutes. | +| `OTEL_METRIC_EXPORT_TIMEOUT` | `5000` | A failed export must not delay the next. | + +Temporality is cumulative, the SDK's default, which is what the collector's +Prometheus exporter expects. The push goes through the egress gateway like +every other byte the actor sends, so the actor's egress policy needs one more +rule, cleartext HTTP to the collector on its port: + +```yaml +- http: + hostnames: + - telemetry-meter.benchmarking.svc.cluster.local + - telemetry-meter.benchmarking.svc + - telemetry-meter.benchmarking + ports: + numbers: [4317] +``` + +Both the full name and the short forms are needed: the sandbox's resolver +completes the short name to the full one, which the DNS relay forwards only +when a rule names it, while the gateway matches the request's authority, which +is the short name the SDK dialed. A rule without `ports` admits port 80 only. diff --git a/cmd/benchmarking/egress/internal/cmd/actor/actor.go b/cmd/benchmarking/egress/internal/cmd/actor/actor.go new file mode 100644 index 0000000000..23f186f005 --- /dev/null +++ b/cmd/benchmarking/egress/internal/cmd/actor/actor.go @@ -0,0 +1,548 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package actor + +import ( + "cmp" + "context" + "crypto/tls" + "crypto/x509" + "encoding/json" + "errors" + "fmt" + "io" + "log/slog" + "maps" + "net" + "net/http" + "net/http/httptrace" + "slices" + "strings" + "sync" + "time" + + "github.com/spf13/cobra" + sdkmetric "go.opentelemetry.io/otel/sdk/metric" + + "go.ate.dev/substrate/cmd/benchmarking/egress/internal/contract" +) + +// Command is the workload of the egress scale test. Once told to start, +// it opens every endpoint, one per interval, then on one goroutine per +// endpoint sends it an HTTP GET once per period of C intervals, counts what +// happened under one lock, and exports the counts and the latencies as OTel +// metrics over OTLP. +// +// The loop starts on POST /start rather than at boot: every actor restores +// from a golden snapshot taken after the template's warmup, so a loop started +// at boot would run in the golden actor, which has no egress policy, and be +// frozen into every actor's snapshot, failed requests included. +func Command() *cobra.Command { + var listen string + cmd := &cobra.Command{ + Use: "actor", + Short: "Hold one tunnel per endpoint open through the gateway and count what happened", + Args: cobra.NoArgs, + RunE: func(cmd *cobra.Command, _ []string) error { + return runActor(cmd.Context(), listen) + }, + } + cmd.Flags().StringVar(&listen, "listen", ":80", "Address to serve the readyz, start, stats and stop routes on.") + return cmd +} + +// runActor serves the actor's routes on listen until ctx ends. +func runActor(ctx context.Context, listen string) error { + srv := &http.Server{ + Addr: listen, + Handler: newServer().handler(), + ReadHeaderTimeout: 10 * time.Second, + } + go func() { + <-ctx.Done() + shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + _ = srv.Shutdown(shutdownCtx) + }() + + slog.Info("benchmark-egress actor listening", "addr", listen) + if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) { + return fmt.Errorf("serving: %w", err) + } + return nil +} + +// maxBodyBytes bounds what the actor reads, from a /start body or from an +// endpoint's response. +const maxBodyBytes = 1 << 20 + +// server serves the actor routes. At most one loop runs at a time. +type server struct { + // endpointURL maps an endpoint number and scheme to the URL the loop + // requests. Tests point it at local servers. + endpointURL func(i int, scheme string) string + // rootCAs verifies HTTPS endpoints; nil uses the system roots, which + // SSL_CERT_FILE extends with the gateway's MITM CA. Tests set their own. + rootCAs *x509.CertPool + // dial opens the loop's connections; nil uses a plain net.Dialer. Tests + // send the in-cluster endpoint names to local listeners through it. + dial dialFunc + // readers, when set, is called at each /start for readers to add to the + // loop's MeterProvider. Tests read the instruments through it. + readers func() []sdkmetric.Reader + + mu sync.Mutex + run *loopRun // nil when no loop is running + // final is the last stopped loop's Stats, endpoints included, kept until + // the next /start so a repeated /stop answers it again. + final *contract.Stats +} + +func newServer() *server { + return &server{endpointURL: contract.EndpointURL} +} + +func (s *server) handler() http.Handler { + mux := http.NewServeMux() + mux.HandleFunc("GET "+contract.ReadyzRoute, func(w http.ResponseWriter, _ *http.Request) { + w.Write([]byte("ok\n")) + }) + mux.HandleFunc("POST "+contract.StartRoute, s.handleStart) + mux.HandleFunc("GET "+contract.StatsRoute, s.handleStats) + mux.HandleFunc("POST "+contract.StopRoute, s.handleStop) + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set(contract.ProtocolHeader, contract.ProtocolVersion) + mux.ServeHTTP(w, r) + }) +} + +func (s *server) handleStart(w http.ResponseWriter, r *http.Request) { + var req contract.StartRequest + if err := json.NewDecoder(io.LimitReader(r.Body, maxBodyBytes)).Decode(&req); err != nil { + http.Error(w, "decoding start request: "+err.Error(), http.StatusBadRequest) + return + } + if err := req.Validate(); err != nil { + http.Error(w, err.Error(), http.StatusBadRequest) + return + } + + s.mu.Lock() + defer s.mu.Unlock() + if s.run != nil { + http.Error(w, "a loop is already running; POST "+contract.StopRoute+" first", http.StatusConflict) + return + } + urls := make([]string, req.Endpoints) + for i := range urls { + urls[i] = s.endpointURL(i, req.URLScheme()) + } + var readers []sdkmetric.Reader + if s.readers != nil { + readers = s.readers() + } + run, err := startLoop(r.Context(), req, urls, s.rootCAs, s.dial, readers) + if err != nil { + http.Error(w, "starting loop: "+err.Error(), http.StatusInternalServerError) + return + } + s.run, s.final = run, nil + w.WriteHeader(http.StatusNoContent) +} + +// withEndpoints reports whether the request asked for the per-endpoint +// counts. +func withEndpoints(r *http.Request) bool { + return r.URL.Query().Get("endpoints") == "1" +} + +// view is a kept Stats as a request asked for it: the per-endpoint slice +// stays only with ?endpoints=1. The kept value is not touched. +func view(s contract.Stats, endpoints bool) contract.Stats { + if !endpoints { + s.Endpoints = nil + } + return s +} + +// runOr404 returns the loop, or answers 404 and returns nil. +func (s *server) runOr404(w http.ResponseWriter) *loopRun { + s.mu.Lock() + run := s.run + s.mu.Unlock() + if run == nil { + http.Error(w, contract.NoLoopMessage, http.StatusNotFound) + } + return run +} + +func (s *server) handleStats(w http.ResponseWriter, r *http.Request) { + run := s.runOr404(w) + if run == nil { + return + } + st, ok := run.snapshot(withEndpoints(r)) + if !ok { + http.Error(w, contract.NoLoopMessage, http.StatusNotFound) + return + } + writeJSON(w, st) +} + +func (s *server) handleStop(w http.ResponseWriter, r *http.Request) { + // Held across the wait so a /start cannot slip in before the loop exits. + s.mu.Lock() + defer s.mu.Unlock() + if s.run == nil { + if s.final == nil { + http.Error(w, contract.NoLoopMessage, http.StatusNotFound) + return + } + writeJSON(w, view(*s.final, withEndpoints(r))) + return + } + st := s.run.stop() + s.run, s.final = nil, &st + writeJSON(w, view(st, withEndpoints(r))) +} + +func writeJSON(w http.ResponseWriter, v any) { + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(v) +} + +// loopRun is one started loop and what it has measured: the counters under +// its lock, mirrored into its Recorder as they change. +type loopRun struct { + cancel context.CancelFunc + wg sync.WaitGroup // one goroutine per endpoint + started time.Time + transport *http.Transport + rec *Recorder + + mu sync.Mutex + // stopped is set by stop once the goroutines are gone, so a read that + // raced the stop answers 404. + stopped bool + // stats holds the counters; Elapsed is derived at snapshot. + stats contract.Stats + last []time.Time // start of each endpoint's most recent request + // rounds is Stats.Rounds kept current, for the gauge: the least-visited + // endpoint's request count. + rounds int64 +} + +// newLoopRun is a loop over req's endpoints with a fresh Recorder and no +// goroutines yet; startLoop runs it. +func newLoopRun(ctx context.Context, req contract.StartRequest, readers []sdkmetric.Reader) (*loopRun, error) { + rec, err := NewRecorder(ctx, req, readers...) + if err != nil { + return nil, err + } + return &loopRun{ + started: time.Now(), + rec: rec, + stats: contract.Stats{Scheme: req.URLScheme(), Endpoints: make([]contract.Endpoint, req.Endpoints)}, + last: make([]time.Time, req.Endpoints), + }, nil +} + +func startLoop(ctx context.Context, req contract.StartRequest, urls []string, roots *x509.CertPool, dial dialFunc, readers []sdkmetric.Reader) (*loopRun, error) { + timeoutMs := req.RequestTimeoutMs + if timeoutMs == 0 { + timeoutMs = contract.DefaultRequestTimeoutMs + } + r, err := newLoopRun(ctx, req, readers) + if err != nil { + return nil, err + } + ctx, cancel := context.WithCancel(context.Background()) + r.cancel, r.transport = cancel, newTransport(roots, dial) + client := &http.Client{ + Transport: r.transport, + // Count a redirect as the endpoint's answer rather than following it + // somewhere the egress policy may not allow. + CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }, + } + timeout := time.Duration(timeoutMs) * time.Millisecond + interval := time.Duration(req.IntervalMs) * time.Millisecond + offset := time.Duration(req.OffsetMs) * time.Millisecond + period := interval * time.Duration(len(urls)) + for i, u := range urls { + r.wg.Go(func() { r.endpoint(ctx, client, i, u, timeout, offset+interval*time.Duration(i), period) }) + } + return r, nil +} + +// dialFunc opens one connection, as net.Dialer.DialContext does. +type dialFunc = func(ctx context.Context, network, addr string) (net.Conn, error) + +// newTransport keeps one connection alive per endpoint and verifies HTTPS +// endpoints against roots. It ignores proxy environment variables: the +// sandbox's egress capture is the only proxy under test. +func newTransport(roots *x509.CertPool, dial dialFunc) *http.Transport { + if dial == nil { + dial = (&net.Dialer{Timeout: 30 * time.Second, KeepAlive: 30 * time.Second}).DialContext + } + return &http.Transport{ + DialContext: dial, + TLSClientConfig: &tls.Config{RootCAs: roots}, + ForceAttemptHTTP2: false, // HTTP/1.1 over either scheme: one request per connection at a time + MaxIdleConns: 0, // no total cap, so C endpoints keep C connections + MaxIdleConnsPerHost: 1, // one request in flight per endpoint, so one per endpoint is enough + // No IdleConnTimeout: a kept-alive connection the client closed for + // idling would redial like one the gateway reset, and the gateway's + // stream idle timeout is what the run measures. + IdleConnTimeout: 0, + // No TLSHandshakeTimeout: the request's own timeout bounds the + // handshake, so a stalled one is counted as a timeout like any other. + } +} + +// endpoint waits offset, opens endpoint i, then requests it once per period +// until ctx ends. The opens are one interval apart across endpoints, so C +// handshakes do not reach the gateway at once. At most one request is in +// flight per endpoint, and the period is counted from the start of the open: +// a request that outlasts the period delays only this endpoint, whose next +// request starts as soon as the slow one returns. +func (r *loopRun) endpoint(ctx context.Context, client *http.Client, i int, u string, timeout, offset, period time.Duration) { + if !pause(ctx, offset) { + return + } + var tick <-chan time.Time + if period > 0 { + ticker := time.NewTicker(period) + defer ticker.Stop() + tick = ticker.C + } + r.request(ctx, client, i, u, timeout) + if ctx.Err() != nil { + return // stopped mid-open: not an open + } + r.mu.Lock() + r.stats.Opened++ + r.rec.EndpointOpened(context.Background()) + if r.stats.Opened == int64(len(r.stats.Endpoints)) { + r.stats.OpeningDone = time.Since(r.started) + r.stats.RequestsAtOpeningDone = r.stats.Requests + r.rec.Gauges(context.Background(), r.rounds, r.stats.OpeningDone.Seconds(), r.stats.LongestGap.Seconds()) + } + r.mu.Unlock() + for ctx.Err() == nil { + if tick != nil { + select { + case <-ctx.Done(): + return + case <-tick: + } + } + r.request(ctx, client, i, u, timeout) + } +} + +// pause waits d, or until ctx ends, and reports whether to go on. +func pause(ctx context.Context, d time.Duration) bool { + if d <= 0 { + return ctx.Err() == nil + } + select { + case <-ctx.Done(): + return false + case <-time.After(d): + return true + } +} + +// requestTrace collects what httptrace reports about one request. The dial, +// and with it the DNS callbacks, can still be running on another goroutine +// after a timed-out request returns, hence the lock. +type requestTrace struct { + mu sync.Mutex + gotConn bool + newConn bool + dnsStart time.Time + dns time.Duration + didDNS bool + tlsStart time.Time + tls time.Duration + tlsStarted bool + didTLS bool +} + +func (t *requestTrace) clientTrace() *httptrace.ClientTrace { + return &httptrace.ClientTrace{ + GotConn: func(info httptrace.GotConnInfo) { + t.mu.Lock() + defer t.mu.Unlock() + t.gotConn, t.newConn = true, !info.Reused + }, + DNSStart: func(httptrace.DNSStartInfo) { + t.mu.Lock() + defer t.mu.Unlock() + t.dnsStart = time.Now() + }, + DNSDone: func(httptrace.DNSDoneInfo) { + t.mu.Lock() + defer t.mu.Unlock() + t.dns, t.didDNS = time.Since(t.dnsStart), true + }, + TLSHandshakeStart: func() { + t.mu.Lock() + defer t.mu.Unlock() + t.tlsStart, t.tlsStarted = time.Now(), true + }, + TLSHandshakeDone: func(_ tls.ConnectionState, err error) { + t.mu.Lock() + defer t.mu.Unlock() + if err == nil { + t.tls, t.didTLS = time.Since(t.tlsStart), true + } + }, + } +} + +func (r *loopRun) request(ctx context.Context, client *http.Client, i int, u string, timeout time.Duration) { + reqCtx, cancel := context.WithTimeout(ctx, timeout) + defer cancel() + trace := &requestTrace{} + req, err := http.NewRequestWithContext(httptrace.WithClientTrace(reqCtx, trace.clientTrace()), http.MethodGet, u, nil) + if err != nil { + r.record(i, 0, 0, err, trace) + return + } + + start := time.Now() + r.mu.Lock() + if last := r.last[i]; !last.IsZero() { + gap := start.Sub(last) + r.stats.LongestGap = max(r.stats.LongestGap, gap) + r.stats.Endpoints[i].LongestGap = max(r.stats.Endpoints[i].LongestGap, gap) + } + r.last[i] = start + r.mu.Unlock() + status := 0 + resp, err := client.Do(req) + if err == nil { + status = resp.StatusCode + _, err = io.Copy(io.Discard, io.LimitReader(resp.Body, maxBodyBytes)) + resp.Body.Close() + } + latency := time.Since(start) + if ctx.Err() != nil { + // Stopped mid-request: the failure is the stop, not the egress path. + return + } + r.record(i, latency, status, err, trace) +} + +func (r *loopRun) record(i int, latency time.Duration, status int, err error, trace *requestTrace) { + trace.mu.Lock() + newConn := trace.gotConn && trace.newConn + didDNS, dns := trace.didDNS, trace.dns + tlsStarted, didTLS, tlsTime := trace.tlsStarted, trace.didTLS, trace.tls + trace.mu.Unlock() + + r.mu.Lock() + defer r.mu.Unlock() + s, ctx := &r.stats, context.Background() + first := s.Endpoints[i].Requests == 0 + s.Requests++ + s.Endpoints[i].Requests++ + if s.Endpoints[i].Requests-1 == r.rounds { + // Only the least-visited endpoint can raise the minimum. + r.rounds = slices.MinFunc(s.Endpoints, func(a, b contract.Endpoint) int { return cmp.Compare(a.Requests, b.Requests) }).Requests + } + if newConn { + s.NewConns++ + r.rec.ConnectionOpened(ctx) + } + r.rec.Gauges(ctx, r.rounds, s.OpeningDone.Seconds(), s.LongestGap.Seconds()) + if didDNS { + r.rec.Record(ctx, contract.DNS, dns.Seconds()) + } + if didTLS { + r.rec.Record(ctx, contract.TLS, tlsTime.Seconds()) + } + class := classify(err, status) + if class != "" && tlsStarted && !didTLS && !strings.HasPrefix(class, "tls: ") { + // A failure inside the handshake, such as the gateway closing a + // connection whose SNI no rule allows, is told apart from the same + // error in an HTTP exchange. + class = "tls handshake: " + class + } + r.rec.Request(ctx, class == "") + if class == "" { + s.Successes++ + if first { + r.rec.Record(ctx, contract.Opening, latency.Seconds()) + } else { + r.rec.Record(ctx, contract.Latency, latency.Seconds()) + } + return + } + if s.Errors == nil { + s.Errors = make(map[string]int64) + } + s.Errors[class]++ + s.Endpoints[i].Errors++ + r.rec.RequestError(ctx, class) +} + +// snapshotLocked copies the counters so far, with Elapsed derived. The +// caller holds r.mu. +func (r *loopRun) snapshotLocked(endpoints bool) contract.Stats { + s := r.stats + s.Elapsed, s.Rounds = time.Since(r.started), r.rounds + s.Errors = maps.Clone(s.Errors) + if endpoints { + s.Endpoints = slices.Clone(s.Endpoints) + } else { + s.Endpoints = nil + } + return s +} + +// snapshot answers GET /stats: the counters alone. ok is false once the +// loop is stopped. +func (r *loopRun) snapshot(endpoints bool) (contract.Stats, bool) { + r.mu.Lock() + defer r.mu.Unlock() + if r.stopped { + return contract.Stats{}, false + } + return r.snapshotLocked(endpoints), true +} + +// stop cancels the loop, waits for every endpoint's goroutine to exit, and +// returns the final Stats, endpoints included. It then flushes the last +// export and releases the Recorder; a read that raced the stop answers 404. +func (r *loopRun) stop() contract.Stats { + r.cancel() + r.wg.Wait() + r.transport.CloseIdleConnections() + r.mu.Lock() + r.stopped = true + s := r.snapshotLocked(true) + r.mu.Unlock() + // On its own context: a client that hung up must not leave the provider + // running. Bounded at twice the template's export timeout: a collector + // that is gone must not hold the driver's /stop for good. + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + if err := errors.Join(r.rec.ForceFlush(ctx), r.rec.Shutdown(ctx)); err != nil { + slog.Warn("final export failed", "err", err) + } + return s +} diff --git a/cmd/benchmarking/egress/internal/cmd/actor/actor_test.go b/cmd/benchmarking/egress/internal/cmd/actor/actor_test.go new file mode 100644 index 0000000000..4d485c4b0e --- /dev/null +++ b/cmd/benchmarking/egress/internal/cmd/actor/actor_test.go @@ -0,0 +1,1324 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package actor + +import ( + "bytes" + "context" + "crypto/tls" + "crypto/x509" + "encoding/json" + "errors" + "fmt" + "io" + "log" + "net" + "net/http" + "net/http/httptest" + "net/url" + "slices" + "strings" + "sync" + "sync/atomic" + "testing" + "testing/synctest" + "time" + + sdkmetric "go.opentelemetry.io/otel/sdk/metric" + "go.opentelemetry.io/otel/sdk/metric/metricdata" + + "go.ate.dev/substrate/cmd/benchmarking/egress/internal/contract" + "go.ate.dev/substrate/cmd/benchmarking/egress/internal/targetcert" +) + +// target is an endpoint that counts the TCP connections opened to it. +type target struct { + *httptest.Server + conns atomic.Int64 + requests atomic.Int64 +} + +func newTarget(t *testing.T, h http.HandlerFunc) *target { + t.Helper() + return newTargetConnState(t, h, nil) +} + +// newTargetConnState is newTarget with onState called on every connection +// state change, after the connection count. +func newTargetConnState(t *testing.T, h http.HandlerFunc, onState func(net.Conn, http.ConnState)) *target { + t.Helper() + tg := &target{} + tg.Server = httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + tg.requests.Add(1) + h(w, r) + })) + tg.Config.ConnState = func(c net.Conn, s http.ConnState) { + if s == http.StateNew { + tg.conns.Add(1) + } + if onState != nil { + onState(c, s) + } + } + tg.Start() + t.Cleanup(tg.Close) + return tg +} + +func okHandler(w http.ResponseWriter, _ *http.Request) { w.Write([]byte("ok")) } + +// quietLog discards a test server's own error log. +var quietLog = log.New(io.Discard, "", 0) + +// host is the target's host:port. +func (tg *target) host() string { return tg.Listener.Addr().String() } + +// fixture is the actor's handler with a reader on every loop it starts, so +// a test reads the instruments the actor exports beside the counters it +// answers. +type fixture struct { + http.Handler + mu sync.Mutex + readers []*finalReader // one per /start, the current loop's last +} + +func newFixture(s *server) *fixture { + f := &fixture{} + s.readers = func() []sdkmetric.Reader { + r := &finalReader{ManualReader: sdkmetric.NewManualReader()} + f.mu.Lock() + f.readers = append(f.readers, r) + f.mu.Unlock() + return []sdkmetric.Reader{r} + } + f.Handler = s.handler() + return f +} + +// metrics is the current loop's instruments: collected now while it runs, +// as of its Shutdown once it is stopped. +func (f *fixture) metrics(t *testing.T) metricdata.ResourceMetrics { + t.Helper() + f.mu.Lock() + r := f.readers[len(f.readers)-1] + f.mu.Unlock() + rm, err := r.read(context.Background()) + if err != nil { + t.Fatal(err) + } + return rm +} + +// finalReader is a manual reader that keeps the collection of its Shutdown, +// which the loop calls once its goroutines have exited, so a test reads the +// instruments as they stood when the final Stats were answered. +type finalReader struct { + *sdkmetric.ManualReader + mu sync.Mutex + final *metricdata.ResourceMetrics +} + +func (r *finalReader) Shutdown(ctx context.Context) error { + var rm metricdata.ResourceMetrics + err := r.ManualReader.Collect(ctx, &rm) + r.mu.Lock() + r.final = &rm + r.mu.Unlock() + return errors.Join(err, r.ManualReader.Shutdown(ctx)) +} + +func (r *finalReader) read(ctx context.Context) (metricdata.ResourceMetrics, error) { + r.mu.Lock() + final := r.final + r.mu.Unlock() + if final != nil { + return *final, nil + } + var rm metricdata.ResourceMetrics + err := r.ManualReader.Collect(ctx, &rm) + return rm, err +} + +// serve returns the actor with endpoint i mapped to hosts[i], a host:port, +// in place of the in-cluster Service names. The URL keeps the scheme the +// loop asks for. +func serve(hosts ...string) *fixture { + return serveTLS(nil, hosts...) +} + +// serveTLS is serve, verifying HTTPS endpoints against roots. +func serveTLS(roots *x509.CertPool, hosts ...string) *fixture { + s := newServer() + s.rootCAs = roots + s.endpointURL = func(i int, scheme string) string { return scheme + "://" + hosts[i] + "/" } + return newFixture(s) +} + +// serveTargets is serve over n fresh targets, one per endpoint. Endpoints +// that share one host would share the transport's single idle connection per +// host and redial on every request. +func serveTargets(t *testing.T, n int) *fixture { + t.Helper() + hosts := make([]string, n) + for i := range hosts { + hosts[i] = newTarget(t, okHandler).host() + } + return serve(hosts...) +} + +// newTLSTarget is newTarget serving HTTPS with httptest's certificate, which +// the returned pool trusts. +func newTLSTarget(t *testing.T, h http.HandlerFunc) (*target, *x509.CertPool) { + t.Helper() + tg := &target{} + tg.Server = httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + tg.requests.Add(1) + h(w, r) + })) + tg.Config.ConnState = func(_ net.Conn, s http.ConnState) { + if s == http.StateNew { + tg.conns.Add(1) + } + } + tg.Config.ErrorLog = quietLog // failed handshakes are what some tests expect + tg.StartTLS() + t.Cleanup(tg.Close) + roots := x509.NewCertPool() + roots.AddCert(tg.Certificate()) + return tg, roots +} + +func post(t *testing.T, h http.Handler, path string, body any) *httptest.ResponseRecorder { + t.Helper() + var buf bytes.Buffer + if body != nil { + if err := json.NewEncoder(&buf).Encode(body); err != nil { + t.Fatal(err) + } + } + rec := httptest.NewRecorder() + h.ServeHTTP(rec, httptest.NewRequest(http.MethodPost, path, &buf)) + return rec +} + +func get(h http.Handler, path string) *httptest.ResponseRecorder { + rec := httptest.NewRecorder() + h.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, path, nil)) + return rec +} + +// decodeStats reads a 200 answer's Stats. +func decodeStats(t *testing.T, what string, rec *httptest.ResponseRecorder) *contract.Stats { + t.Helper() + if rec.Code != http.StatusOK { + t.Fatalf("%s = %d %s", what, rec.Code, rec.Body) + } + var s contract.Stats + if err := json.Unmarshal(rec.Body.Bytes(), &s); err != nil { + t.Fatalf("%s: decoding %s: %v", what, rec.Body, err) + } + return &s +} + +// The poll and the stop of a running loop, each with the per-endpoint counts. +const ( + statsPath = contract.StatsRoute + "?endpoints=1" + stopPath = contract.StopRoute + "?endpoints=1" +) + +// start posts /start with req, identified. +func start(t *testing.T, h http.Handler, req contract.StartRequest) *httptest.ResponseRecorder { + t.Helper() + return post(t, h, contract.StartRoute, identified(req)) +} + +// readStats polls the counters. +func readStats(t *testing.T, h http.Handler) *contract.Stats { + t.Helper() + return decodeStats(t, "GET /stats", get(h, statsPath)) +} + +// stopLoop stops the loop and returns its final Stats. +func stopLoop(t *testing.T, h http.Handler) *contract.Stats { + t.Helper() + return decodeStats(t, "POST /stop", post(t, h, stopPath, nil)) +} + +// runLoop starts a loop with req, lets it run until cond holds on the live +// counters (or fails after 10s), then stops it and returns the final Stats. +func runLoop(t *testing.T, h http.Handler, req contract.StartRequest, cond func(*contract.Stats) bool) *contract.Stats { + t.Helper() + if rec := start(t, h, req); rec.Code != http.StatusNoContent { + t.Fatalf("POST /start = %d %s", rec.Code, rec.Body) + } + waitFor(t, h, cond) + return stopLoop(t, h) +} + +// waitFor polls the counters until cond holds, or fails after 10s. +func waitFor(t *testing.T, h http.Handler, cond func(*contract.Stats) bool) { + t.Helper() + deadline := time.Now().Add(10 * time.Second) + for { + s := readStats(t, h) + if cond(s) { + return + } + if time.Now().After(deadline) { + t.Fatalf("condition not reached; last stats %+v", s) + } + time.Sleep(5 * time.Millisecond) + } +} + +func atLeast(n int64) func(*contract.Stats) bool { + return func(s *contract.Stats) bool { return s.Requests >= n } +} + +// histogram is the named distribution's data point in the current loop's +// instruments; ok is false when the loop recorded nothing in it. +func histogram(t *testing.T, f *fixture, name contract.Distribution) (p metricdata.HistogramDataPoint[float64], ok bool) { + t.Helper() + for _, sm := range f.metrics(t).ScopeMetrics { + for _, m := range sm.Metrics { + if m.Name != string(name) { + continue + } + h, isHist := m.Data.(metricdata.Histogram[float64]) + if !isHist || len(h.DataPoints) != 1 { + t.Fatalf("%s is %+v, want a histogram with one point", name, m.Data) + } + return h.DataPoints[0], true + } + } + return p, false +} + +// samples is the count of a distribution in the current loop, 0 when absent. +func samples(t *testing.T, f *fixture, name contract.Distribution) uint64 { + t.Helper() + p, _ := histogram(t, f, name) + return p.Count +} + +func TestKeepAliveReusesOneConnectionPerEndpoint(t *testing.T) { + t.Parallel() + targets := []*target{newTarget(t, okHandler), newTarget(t, okHandler), newTarget(t, okHandler)} + var hosts []string + for _, tg := range targets { + hosts = append(hosts, tg.host()) + } + h := serve(hosts...) + + s := runLoop(t, h, contract.StartRequest{Endpoints: len(hosts)}, func(s *contract.Stats) bool { return s.Rounds >= 10 }) + if s.Scheme != contract.SchemeHTTP || samples(t, h, contract.TLS) != 0 || samples(t, h, contract.DNS) != 0 { + t.Errorf("a start with no scheme against IP hosts ran over %q with %d TLS handshakes and %d DNS lookups, want http with neither", s.Scheme, samples(t, h, contract.TLS), samples(t, h, contract.DNS)) + } + + if s.Successes != s.Requests || len(s.Errors) != 0 { + t.Errorf("successes %d of %d requests, errors %v", s.Successes, s.Requests, s.Errors) + } + for i, tg := range targets { + if got := tg.conns.Load(); got != 1 { + t.Errorf("target %d saw %d connections, want 1 kept alive", i, got) + } + } + if s.NewConns != int64(len(targets)) { + t.Errorf("NewConns = %d, want %d (one per endpoint)", s.NewConns, len(targets)) + } + if opening, latency := samples(t, h, contract.Opening), samples(t, h, contract.Latency); opening != uint64(len(targets)) || int64(opening+latency) != s.Successes { + t.Errorf("opening %d and latency %d samples for %d successes; want one opening per endpoint and the rest latency", opening, latency, s.Successes) + } + for i, e := range s.Endpoints { + if e.Requests < 10 { + t.Errorf("endpoint %d got %d requests, want at least the %d rounds the loop was stopped at", i, e.Requests, 10) + } + } +} + +func TestStopDuringOpeningDoesNotCountTheCancelledOpen(t *testing.T) { + t.Parallel() + first := newTarget(t, okHandler) + blocked := newTarget(t, func(w http.ResponseWriter, r *http.Request) { + <-r.Context().Done() // holds the second open until the client gives up + }) + h := serve(first.host(), blocked.host()) + + s := runLoop(t, h, contract.StartRequest{Endpoints: 2}, func(s *contract.Stats) bool { return s.Opened >= 1 }) + if s.Opened != 1 { + t.Errorf("Opened = %d after stopping during the second open, want 1", s.Opened) + } + if s.Rounds != 0 || s.OpeningDone != 0 || s.Endpoints[1].Requests != 0 { + t.Errorf("Rounds = %d, OpeningDone = %v, endpoint 1 requests = %d during the opening, want 0, 0 and 0", s.Rounds, s.OpeningDone, s.Endpoints[1].Requests) + } +} + +func TestRequestTimeoutIsClassified(t *testing.T) { + t.Parallel() + release := make(chan struct{}) + slow := newTarget(t, func(w http.ResponseWriter, r *http.Request) { + select { + case <-release: + case <-r.Context().Done(): + } + }) + t.Cleanup(func() { close(release) }) + h := serve(slow.host()) + + s := runLoop(t, h, contract.StartRequest{Endpoints: 1, RequestTimeoutMs: 20}, atLeast(2)) + + if s.Errors["timeout"] != s.Requests || s.Successes != 0 { + t.Errorf("errors %v, successes %d for %d requests; want every request a timeout", s.Errors, s.Successes, s.Requests) + } + if n := samples(t, h, contract.Latency); n != 0 { + t.Errorf("failed requests recorded %d latency samples, want 0", n) + } + if s.Endpoints[0].Errors != s.Requests { + t.Errorf("endpoint errors = %d, want %d", s.Endpoints[0].Errors, s.Requests) + } +} + +func TestNon200IsAnError(t *testing.T) { + t.Parallel() + ok := newTarget(t, okHandler) + failing := newTarget(t, func(w http.ResponseWriter, _ *http.Request) { + http.Error(w, "egress denied", http.StatusServiceUnavailable) + }) + h := serve(ok.host(), failing.host()) + + s := runLoop(t, h, contract.StartRequest{Endpoints: 2}, atLeast(10)) + + if s.Errors["HTTP 503"] == 0 { + t.Errorf("errors = %v, want HTTP 503 counted", s.Errors) + } + if s.Endpoints[0].Errors != 0 || s.Endpoints[1].Errors != s.Endpoints[1].Requests { + t.Errorf("endpoints = %+v, want only the second one failing", s.Endpoints) + } +} + +func TestStartStopLifecycle(t *testing.T) { + t.Parallel() + tg := newTarget(t, okHandler) + h := serve(tg.host()) + req := contract.StartRequest{Endpoints: 1} + + if rec := get(h, contract.StatsRoute); rec.Code != http.StatusNotFound { + t.Errorf("GET /stats before start = %d, want 404", rec.Code) + } + if rec := post(t, h, contract.StopRoute, nil); rec.Code != http.StatusNotFound { + t.Errorf("POST /stop before start = %d, want 404", rec.Code) + } + + first := runLoop(t, h, req, atLeast(5)) + time.Sleep(20 * time.Millisecond) + // A request the stop cut short can still reach the target after /stop + // returns; nothing else may. + if extra := tg.requests.Load() - first.Requests; extra > 1 { + t.Errorf("target got %d requests the stopped loop did not count, want at most the one cut short by the stop", extra) + } + + if rec := start(t, h, req); rec.Code != http.StatusNoContent { + t.Fatalf("restart = %d %s", rec.Code, rec.Body) + } + if rec := start(t, h, req); rec.Code != http.StatusConflict { + t.Errorf("second /start while running = %d, want 409", rec.Code) + } + second := stopLoop(t, h) + // The target saw every request of both runs (plus any cut short by a + // stop), so a second run that started from zero counts no more than the + // target saw after the first run. + if sinceRestart := tg.requests.Load() - first.Requests; second.Requests > sinceRestart { + t.Errorf("restarted loop counted %d requests, more than the %d sent since the restart: the old counts were kept", second.Requests, sinceRestart) + } +} + +func TestStartRejectsBadRequests(t *testing.T) { + t.Parallel() + h := newServer().handler() + const named = `"actor":"benchmark-workloads/egress-0","cell":"cell-1","run":"run-1"` + for _, tc := range []struct{ name, body string }{ + {"not json", `not json`}, + {"empty object", `{}`}, + {"invalid", `{"endpoints":1,"scheme":"ftp",` + named + `}`}, + {"no identity", `{"endpoints":1}`}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + rec := httptest.NewRecorder() + h.ServeHTTP(rec, httptest.NewRequest(http.MethodPost, contract.StartRoute, strings.NewReader(tc.body))) + if rec.Code != http.StatusBadRequest { + t.Errorf("POST /start %s = %d, want 400", tc.body, rec.Code) + } + }) + } + if rec := get(h, contract.ReadyzRoute); rec.Code != http.StatusOK { + t.Errorf("GET /readyz = %d, want 200", rec.Code) + } +} + +func TestDefaultURLsAreTheEndpointServices(t *testing.T) { + t.Parallel() + s := newServer() + for _, i := range []int{0, 9, 255} { + for _, scheme := range []string{contract.SchemeHTTP, contract.SchemeHTTPS} { + if got, want := s.endpointURL(i, scheme), contract.EndpointURL(i, scheme); got != want { + t.Errorf("endpoint %d over %s = %q, want %q", i, scheme, got, want) + } + } + } + if s.rootCAs != nil { + t.Error("the actor's roots are set by default; they must stay nil so SSL_CERT_FILE applies") + } + if s.dial != nil { + t.Error("the actor dials through a hook by default; it must stay nil so a plain dialer is used") + } +} + +func TestHTTPSFailuresAreClassified(t *testing.T) { + t.Parallel() + bundle, err := targetcert.Generate(time.Now()) + if err != nil { + t.Fatal(err) + } + endpointCert, err := tls.X509KeyPair(bundle.Cert, bundle.Key) + if err != nil { + t.Fatal(err) + } + endpointRoots := x509.NewCertPool() + endpointRoots.AppendCertsFromPEM(bundle.CA) + // Serves the endpoints' certificate, which names no IP address. + wrongName := &target{Server: httptest.NewUnstartedServer(http.HandlerFunc(okHandler))} + wrongName.TLS = &tls.Config{Certificates: []tls.Certificate{endpointCert}} + wrongName.Config.ErrorLog = quietLog + wrongName.StartTLS() + t.Cleanup(wrongName.Close) + + untrusted, _ := newTLSTarget(t, okHandler) + plain := newTarget(t, okHandler) + + // Accepts, reads the ClientHello so closing sends a FIN rather than a + // reset, and closes: a peer that refuses the handshake, as the gateway + // does for an SNI no rule allows. + closer, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { closer.Close() }) + go func() { + for { + conn, err := closer.Accept() + if err != nil { + return + } + _, _ = conn.Read(make([]byte, 4096)) + conn.Close() + } + }() + + for _, tc := range []struct { + name string + roots *x509.CertPool + host string + want string + }{ + {"untrusted CA", x509.NewCertPool(), untrusted.host(), "tls: unknown authority"}, + {"hostname mismatch", endpointRoots, wrongName.host(), "tls: hostname mismatch"}, + {"plain HTTP server", x509.NewCertPool(), plain.host(), "tls: not a TLS server"}, + {"peer closes during handshake", x509.NewCertPool(), closer.Addr().String(), "tls handshake: "}, + } { + t.Run(tc.name, func(t *testing.T) { + h := serveTLS(tc.roots, tc.host) + s := runLoop(t, h, contract.StartRequest{Endpoints: 1, Scheme: contract.SchemeHTTPS}, atLeast(2)) + var matched int64 + for class, n := range s.Errors { + if strings.HasPrefix(class, tc.want) { + matched += n + } + } + if matched != s.Requests || s.Successes != 0 || samples(t, h, contract.TLS) != 0 { + t.Errorf("errors %v, %d successes, %d handshakes for %d requests; want every request classed %q", s.Errors, s.Successes, samples(t, h, contract.TLS), s.Requests, tc.want) + } + }) + } +} + +// dialTo returns a dial function that connects host:port to addrs[host] +// instead, so a loop can request the in-cluster endpoint names against local +// listeners. TLS still verifies the name the loop asked for. +func dialTo(addrs map[string]string) dialFunc { + return func(ctx context.Context, network, addr string) (net.Conn, error) { + host, _, err := net.SplitHostPort(addr) + if err != nil { + return nil, err + } + real, ok := addrs[host] + if !ok { + return nil, fmt.Errorf("no local listener for %s", addr) + } + return (&net.Dialer{}).DialContext(ctx, network, real) + } +} + +// targetCertServer is an HTTPS target serving the endpoints' certificate. +func targetCertServer(t *testing.T, bundle *targetcert.Bundle) *target { + t.Helper() + cert, err := tls.X509KeyPair(bundle.Cert, bundle.Key) + if err != nil { + t.Fatal(err) + } + tg := &target{Server: httptest.NewUnstartedServer(http.HandlerFunc(okHandler))} + tg.Config.ConnState = func(_ net.Conn, s http.ConnState) { + if s == http.StateNew { + tg.conns.Add(1) + } + } + tg.TLS = &tls.Config{Certificates: []tls.Certificate{cert}} + tg.StartTLS() + t.Cleanup(tg.Close) + return tg +} + +// TestEndpointsByNameOverBothSchemes runs the loop as the actor does in the +// cluster: the default endpoint URLs, eight endpoints, and over HTTPS the +// target's own certificate verified under each endpoint's name. +func TestEndpointsByNameOverBothSchemes(t *testing.T) { + t.Parallel() + const c = 8 + bundle, err := targetcert.Generate(time.Now()) + if err != nil { + t.Fatal(err) + } + roots := x509.NewCertPool() + if !roots.AppendCertsFromPEM(bundle.CA) { + t.Fatal("no CA in the bundle") + } + for _, scheme := range []string{contract.SchemeHTTP, contract.SchemeHTTPS} { + t.Run(scheme, func(t *testing.T) { + t.Parallel() + addrs := make(map[string]string, c) + targets := make([]*target, c) + for i := range c { + if scheme == contract.SchemeHTTPS { + targets[i] = targetCertServer(t, bundle) + } else { + targets[i] = newTarget(t, okHandler) + } + addrs[contract.EndpointHost(i)] = targets[i].host() + } + s := newServer() + s.rootCAs = roots + s.dial = dialTo(addrs) + h := newFixture(s) + + const interval = 2 * time.Millisecond + st := runLoop(t, h, contract.StartRequest{Endpoints: c, Scheme: scheme, IntervalMs: int64(interval / time.Millisecond)}, + func(s *contract.Stats) bool { return s.Rounds >= 3 }) + + if st.Scheme != scheme || st.Opened != c || st.Rounds < 3 { + t.Errorf("Scheme = %q, Opened = %d, Rounds = %d, want %s, %d and at least 3", st.Scheme, st.Opened, st.Rounds, scheme, c) + } + if st.Successes != st.Requests || len(st.Errors) != 0 { + t.Errorf("successes %d of %d requests, errors %v", st.Successes, st.Requests, st.Errors) + } + if st.NewConns != c { + t.Errorf("NewConns = %d, want one per endpoint (%d)", st.NewConns, c) + } + wantTLS := uint64(0) + if scheme == contract.SchemeHTTPS { + wantTLS = c + } + if n := samples(t, h, contract.TLS); n != wantTLS { + t.Errorf("TLS handshakes = %d, want %d", n, wantTLS) + } + for i, tg := range targets { + if got := tg.conns.Load(); got != 1 { + t.Errorf("endpoint %d saw %d connections, want 1 kept alive", i, got) + } + } + // Every endpoint is requested once per period of C intervals; a gap + // is about one period, never anywhere near Elapsed. + if period := c * interval; st.LongestGap < period/2 || st.LongestGap > 5*time.Second { + t.Errorf("LongestGap = %v, want about one period (%v) and well under Elapsed", st.LongestGap, period) + } + }) + } +} + +// TestResetConnectionIsRedialedWithoutAnError pins how a tunnel the gateway +// reset shows up: the loop redials, net/http retries the GET, and NewConns +// grows by one while Errors stays empty. +func TestResetConnectionIsRedialedWithoutAnError(t *testing.T) { + t.Parallel() + var cutOnce atomic.Bool + cut := newTargetConnState(t, okHandler, func(c net.Conn, s http.ConnState) { + // Closes the server side once the first response is out, as the + // gateway does to an idle tunnel. + if s == http.StateIdle && cutOnce.CompareAndSwap(false, true) { + c.Close() + } + }) + steady := []*target{newTarget(t, okHandler), newTarget(t, okHandler)} + h := serve(steady[0].host(), cut.host(), steady[1].host()) + + s := runLoop(t, h, contract.StartRequest{Endpoints: 3, IntervalMs: 2}, func(s *contract.Stats) bool { return s.Rounds >= 2 }) + + if s.Successes != s.Requests || len(s.Errors) != 0 { + t.Errorf("successes %d of %d requests, errors %v; want the reset absorbed", s.Successes, s.Requests, s.Errors) + } + if s.NewConns != 4 { + t.Errorf("NewConns = %d, want 4: one per endpoint plus the redial", s.NewConns) + } + if got := cut.conns.Load(); got != 2 { + t.Errorf("the cut endpoint saw %d connections, want 2", got) + } + for i, tg := range steady { + if got := tg.conns.Load(); got != 1 { + t.Errorf("steady endpoint %d saw %d connections, want 1", i, got) + } + } +} + +// pipeNet serves a handler over in-memory connections, so a loop can run +// inside a synctest bubble, where a real socket would not count as blocked. +type pipeNet struct { + conns chan net.Conn + done chan struct{} + once sync.Once + srv *http.Server +} + +func newPipeNet(h http.Handler) *pipeNet { + p := &pipeNet{conns: make(chan net.Conn), done: make(chan struct{}), srv: &http.Server{Handler: h, ErrorLog: quietLog}} + go p.srv.Serve(p) + return p +} + +func (p *pipeNet) dial(ctx context.Context, _, _ string) (net.Conn, error) { + client, server := net.Pipe() + select { + case p.conns <- server: + return client, nil + case <-ctx.Done(): + return nil, ctx.Err() + } +} + +func (p *pipeNet) Accept() (net.Conn, error) { + select { + case c := <-p.conns: + return c, nil + case <-p.done: + return nil, net.ErrClosed + } +} + +func (p *pipeNet) Close() error { + p.once.Do(func() { close(p.done) }) + return nil +} + +func (p *pipeNet) Addr() net.Addr { return pipeAddr{} } + +type pipeAddr struct{} + +func (pipeAddr) Network() string { return "pipe" } +func (pipeAddr) String() string { return "pipe" } + +// startPipeLoop serves the actor over pn, with endpoint i named endpoint-i, +// and starts a loop with req. It returns the handler and a counters reader +// that waits for the bubble to settle first. +func startPipeLoop(t *testing.T, pn *pipeNet, req contract.StartRequest) (*fixture, func() *contract.Stats) { + t.Helper() + s := newServer() + s.dial = pn.dial + s.endpointURL = func(i int, scheme string) string { return fmt.Sprintf("%s://endpoint-%d/", scheme, i) } + h := newFixture(s) + if rec := start(t, h, req); rec.Code != http.StatusNoContent { + t.Fatalf("POST /start = %d %s", rec.Code, rec.Body) + } + return h, func() *contract.Stats { + synctest.Wait() + return readStats(t, h) + } +} + +// TestPacingAndGapsAreExact runs the loop on a fake clock with instant +// endpoints, so the schedule can be checked to the millisecond: endpoint i +// opens i intervals in, each endpoint is then requested once per period of C +// intervals, and so every gap is one period. At each step the instruments +// hold one opening per opened endpoint and the rest latency. +func TestPacingAndGapsAreExact(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + pn := newPipeNet(http.HandlerFunc(okHandler)) + defer pn.srv.Close() + const c, interval = 4, 100 * time.Millisecond + const period = c * interval + start := time.Now() + h, stats := startPipeLoop(t, pn, contract.StartRequest{Endpoints: c, IntervalMs: int64(interval / time.Millisecond)}) + for _, tc := range []struct { + at time.Duration + requests, opened, rounds int64 + openingDone, longestGap time.Duration + why string + }{ + {interval / 2, 1, 1, 0, 0, 0, "endpoint 0 opens at once, the others wait their turn"}, + {period - interval/2, c, c, 1, (c - 1) * interval, 0, "one open per interval; the last completes at (C-1) intervals"}, + {2*period - interval/2, 2 * c, c, 2, (c - 1) * interval, period, "each endpoint's second request comes one period after its first"}, + {3*period - interval/2, 3 * c, c, 3, (c - 1) * interval, period, "steady state: every gap is one period"}, + } { + time.Sleep(tc.at - time.Since(start)) + st := stats() + if st.Requests != tc.requests || st.Opened != tc.opened || st.Rounds != tc.rounds || st.OpeningDone != tc.openingDone || st.LongestGap != tc.longestGap { + t.Errorf("at %v: requests %d, opened %d, rounds %d, opening done %v, longest gap %v; want %d, %d, %d, %v, %v (%s)", + tc.at, st.Requests, st.Opened, st.Rounds, st.OpeningDone, st.LongestGap, tc.requests, tc.opened, tc.rounds, tc.openingDone, tc.longestGap, tc.why) + } + opening, latency := samples(t, h, contract.Opening), samples(t, h, contract.Latency) + if int64(opening) != tc.opened || int64(opening+latency) != st.Successes { + t.Errorf("at %v: %d opening and %d latency samples for %d successes; want one opening per opened endpoint (%d) and the rest latency", + tc.at, opening, latency, st.Successes, tc.opened) + } + } + synctest.Wait() + for i, e := range readStats(t, h).Endpoints { + if e.LongestGap != period { + t.Errorf("endpoint %d: longest gap %v, want one period (%v)", i, e.LongestGap, period) + } + } + stopLoop(t, h) + }) +} + +// TestOffsetShiftsEveryEndpoint pins that offsetMs delays the whole schedule +// and nothing else: endpoint 0 opens at the offset, the last endpoint at the +// offset plus C-1 intervals, and every gap is still one period. +func TestOffsetShiftsEveryEndpoint(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + pn := newPipeNet(http.HandlerFunc(okHandler)) + defer pn.srv.Close() + const c, interval, offset = 4, 100 * time.Millisecond, 50 * time.Millisecond + const period = c * interval + start := time.Now() + h, stats := startPipeLoop(t, pn, contract.StartRequest{Endpoints: c, IntervalMs: int64(interval / time.Millisecond), OffsetMs: int64(offset / time.Millisecond)}) + for _, tc := range []struct { + at time.Duration + requests, opened, rounds int64 + openingDone, longestGap time.Duration + why string + }{ + {offset / 2, 0, 0, 0, 0, 0, "nothing opens before the offset"}, + {offset + interval/2, 1, 1, 0, 0, 0, "endpoint 0 opens at the offset"}, + {offset + period - interval/2, c, c, 1, offset + (c-1)*interval, 0, "the last open completes at the offset plus C-1 intervals"}, + {offset + 2*period - interval/2, 2 * c, c, 2, offset + (c-1)*interval, period, "each repeat comes one period after the open it follows"}, + } { + time.Sleep(tc.at - time.Since(start)) + st := stats() + if st.Requests != tc.requests || st.Opened != tc.opened || st.Rounds != tc.rounds || st.OpeningDone != tc.openingDone || st.LongestGap != tc.longestGap { + t.Errorf("at %v: requests %d, opened %d, rounds %d, opening done %v, longest gap %v; want %d, %d, %d, %v, %v (%s)", + tc.at, st.Requests, st.Opened, st.Rounds, st.OpeningDone, st.LongestGap, tc.requests, tc.opened, tc.rounds, tc.openingDone, tc.longestGap, tc.why) + } + } + for i, e := range stats().Endpoints { + if e.LongestGap != period { + t.Errorf("endpoint %d: longest gap %v, want one period (%v)", i, e.LongestGap, period) + } + } + stopLoop(t, h) + }) +} + +// TestSlowEndpointDelaysOnlyItself pins that a request hanging until its +// timeout on one endpoint leaves the other endpoints' gaps at one period, and +// that the hung endpoint's own gap is the timeout: its next request starts as +// soon as the slow one returns. +func TestSlowEndpointDelaysOnlyItself(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + pn := newPipeNet(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Host == "endpoint-0" { + <-r.Context().Done() + return + } + okHandler(w, r) + })) + defer pn.srv.Close() + const c, interval, timeout = 3, 50 * time.Millisecond, 300 * time.Millisecond + const period = c * interval + h, stats := startPipeLoop(t, pn, contract.StartRequest{Endpoints: c, IntervalMs: int64(interval / time.Millisecond), RequestTimeoutMs: int64(timeout / time.Millisecond)}) + + time.Sleep(time.Second + interval/2) + st := stats() + for i := 1; i < c; i++ { + if got := st.Endpoints[i].LongestGap; got != period { + t.Errorf("endpoint %d: longest gap %v, want one period (%v) with endpoint 0 hung", i, got, period) + } + if st.Endpoints[i].Errors != 0 { + t.Errorf("endpoint %d: %d errors, want 0", i, st.Endpoints[i].Errors) + } + } + if st.Endpoints[0].LongestGap != timeout || st.LongestGap != timeout { + t.Errorf("endpoint 0 longest gap %v, overall %v; want the timeout (%v) for both", st.Endpoints[0].LongestGap, st.LongestGap, timeout) + } + if st.Errors["timeout"] == 0 || st.Errors["timeout"] != st.Endpoints[0].Errors { + t.Errorf("errors %v, endpoint 0 errors %d; want every timeout on endpoint 0", st.Errors, st.Endpoints[0].Errors) + } + if st.Opened != c || st.OpeningDone != timeout { + t.Errorf("Opened = %d, OpeningDone = %v; want %d and %v: the hung open counts once it times out, and it completes last", st.Opened, st.OpeningDone, c, timeout) + } + stopLoop(t, h) + }) +} + +// TestSlowOpensDoNotDelayOpenedEndpoints pins the schedule when opens are +// slow, as they are while a gateway works through a fleet's handshakes: an +// endpoint already opened is requested one period after its open while the +// others are still opening, and a slow open stretches only its own +// endpoint's first gap, to the open's latency. +func TestSlowOpensDoNotDelayOpenedEndpoints(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + const c, interval, openLatency = 4, 100 * time.Millisecond, 500 * time.Millisecond + const period = c * interval + var mu sync.Mutex + opened := map[string]bool{} + pn := newPipeNet(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + first := !opened[r.Host] + opened[r.Host] = true + mu.Unlock() + if first && r.Host != "endpoint-0" { + select { + case <-time.After(openLatency): + case <-r.Context().Done(): + return + } + } + okHandler(w, r) + })) + defer pn.srv.Close() + start := time.Now() + h, stats := startPipeLoop(t, pn, contract.StartRequest{Endpoints: c, IntervalMs: int64(interval / time.Millisecond)}) + + // endpoint 0 opened at once; its second request is due at one period, + // while endpoints 1 to 3 are still inside their first. + time.Sleep(period + interval/2) + st := stats() + if st.Opened != 1 || st.Endpoints[0].Requests != 2 || st.Endpoints[0].LongestGap != period || st.Requests != 2 { + t.Errorf("at %v: opened %d, endpoint 0 %d requests with gap %v, %d requests in all; want 1, 2, %v, 2: the opened endpoint keeps its period while the others open", + time.Since(start), st.Opened, st.Endpoints[0].Requests, st.Endpoints[0].LongestGap, st.Requests, period) + } + + // The last open, endpoint 3's, started at 3 intervals and took + // openLatency; after that each slow endpoint's next request came + // at once, since its period had already passed. + time.Sleep(3*period + interval/2 - time.Since(start)) + st = stats() + if st.Opened != c || st.OpeningDone != (c-1)*interval+openLatency { + t.Errorf("Opened = %d, OpeningDone = %v; want %d and %v", st.Opened, st.OpeningDone, c, (c-1)*interval+openLatency) + } + if st.Endpoints[0].LongestGap != period || st.Endpoints[0].Requests != 4 { + t.Errorf("endpoint 0: longest gap %v, %d requests; want %v and 4: the slow opens of the others did not touch it", st.Endpoints[0].LongestGap, st.Endpoints[0].Requests, period) + } + for i := 1; i < c; i++ { + if got := st.Endpoints[i].LongestGap; got != openLatency { + t.Errorf("endpoint %d: longest gap %v, want its own open latency (%v)", i, got, openLatency) + } + } + stopLoop(t, h) + op, _ := histogram(t, h, contract.Opening) + if opMax, _ := op.Max.Value(); st.LongestGap != openLatency || op.Count != c || opMax != openLatency.Seconds() { + t.Errorf("LongestGap = %v, opening samples %d with max %v; want %v, %d and %v", st.LongestGap, op.Count, opMax, openLatency, c, openLatency.Seconds()) + } + }) +} + +// TestSlowRequestResumesOnTheNextTick pins what a request longer than the +// period does to its own endpoint: the next request starts as soon as the +// slow one returns, so the gap is the request's latency, not latency plus a +// period. +func TestSlowRequestResumesOnTheNextTick(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + const period, latency = 50 * time.Millisecond, 120 * time.Millisecond + var mu sync.Mutex + var starts []time.Duration + start := time.Now() + pn := newPipeNet(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + starts = append(starts, time.Since(start)) + mu.Unlock() + select { + case <-time.After(latency): + okHandler(w, r) + case <-r.Context().Done(): + } + })) + defer pn.srv.Close() + h, stats := startPipeLoop(t, pn, contract.StartRequest{Endpoints: 1, IntervalMs: int64(period / time.Millisecond)}) + + time.Sleep(4*latency + period/2) + st := stats() + mu.Lock() + got := slices.Clone(starts) + mu.Unlock() + want := []time.Duration{0, latency, 2 * latency, 3 * latency, 4 * latency} + if !slices.Equal(got, want) { + t.Errorf("request starts = %v, want %v: back to back once each outlasts the period", got, want) + } + if st.LongestGap != latency { + t.Errorf("LongestGap = %v, want the request latency (%v)", st.LongestGap, latency) + } + stopLoop(t, h) + }) +} + +// TestStopWithRequestsInFlightReturnsAtOnce pins that a stop neither waits +// for the in-flight requests' timeouts nor leaks their goroutines. +func TestStopWithRequestsInFlightReturnsAtOnce(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + pn := newPipeNet(http.HandlerFunc(func(_ http.ResponseWriter, r *http.Request) { <-r.Context().Done() })) + defer pn.srv.Close() + const c = 8 + h, _ := startPipeLoop(t, pn, contract.StartRequest{Endpoints: c, RequestTimeoutMs: 5000}) + time.Sleep(10 * time.Millisecond) + synctest.Wait() + + before := time.Now() + st := stopLoop(t, h) + if waited := time.Since(before); waited != 0 { + t.Errorf("/stop took %v with %d requests in flight, want 0: the stop must not wait for a timeout", waited, c) + } + if _, recorded := histogram(t, h, contract.Opening); st.Requests != 0 || st.Opened != 0 || recorded { + t.Errorf("Requests = %d, Opened = %d, opening recorded %v after stopping mid-open, want 0, 0 and nothing", st.Requests, st.Opened, recorded) + } + if rec := get(h, contract.StatsRoute); rec.Code != http.StatusNotFound { + t.Errorf("GET /stats after stop = %d, want 404", rec.Code) + } + }) +} + +// TestDNSAndTLSStopMovingOnceOpened pins the emptiness the driver checks in +// a settled window: once every tunnel is open, the lookup and handshake +// counts do not move while the latency count does. +func TestDNSAndTLSStopMovingOnceOpened(t *testing.T) { + t.Parallel() + byName := newTarget(t, okHandler) + u, err := url.Parse(byName.URL) + if err != nil { + t.Fatal(err) + } + tlsTarget, roots := newTLSTarget(t, okHandler) + for _, tc := range []struct { + name string + h *fixture + req contract.StartRequest + dist contract.Distribution + }{ + {"dns over http by name", serve("localhost:" + u.Port()), contract.StartRequest{Endpoints: 1}, contract.DNS}, + {"tls over https", serveTLS(roots, tlsTarget.host()), contract.StartRequest{Endpoints: 1, Scheme: contract.SchemeHTTPS}, contract.TLS}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + if rec := start(t, tc.h, tc.req); rec.Code != http.StatusNoContent { + t.Fatalf("POST /start = %d %s", rec.Code, rec.Body) + } + waitFor(t, tc.h, atLeast(2)) + opened, first := samples(t, tc.h, tc.dist), samples(t, tc.h, contract.Latency) + if opened == 0 { + t.Errorf("the open recorded no %s sample", tc.dist) + } + // Two more requests after a count taken after the first read. + waitFor(t, tc.h, atLeast(readStats(t, tc.h).Requests+2)) + stopLoop(t, tc.h) + if later, second := samples(t, tc.h, tc.dist), samples(t, tc.h, contract.Latency); later != opened || second <= first { + t.Errorf("%s went %d -> %d and latency %d -> %d over the settled requests; want %s flat and latency up", tc.dist, opened, later, first, second, tc.dist) + } + }) + } +} + +func TestStopReturnsFinalStats(t *testing.T) { + t.Parallel() + tg := newTarget(t, okHandler) + h := serve(tg.host()) + for _, tc := range []struct { + name string + rec *httptest.ResponseRecorder + }{ + {"GET /stats", get(h, contract.StatsRoute)}, + {"POST /stop", post(t, h, contract.StopRoute, nil)}, + } { + if tc.rec.Code != http.StatusNotFound || strings.TrimSpace(tc.rec.Body.String()) != contract.NoLoopMessage { + t.Errorf("%s with no loop = %d %q, want 404 %q", tc.name, tc.rec.Code, tc.rec.Body, contract.NoLoopMessage) + } + } + if rec := start(t, h, contract.StartRequest{Endpoints: 1}); rec.Code != http.StatusNoContent { + t.Fatalf("POST /start = %d %s", rec.Code, rec.Body) + } + waitFor(t, h, atLeast(1)) + rec := post(t, h, contract.StopRoute, nil) + final := decodeStats(t, "POST /stop", rec) + if ct := rec.Header().Get("Content-Type"); ct != "application/json" || final.Scheme != contract.SchemeHTTP || final.Endpoints != nil || final.Requests == 0 { + t.Errorf("POST /stop answered %q with scheme %q, endpoints %v and %d requests; want JSON, http, no endpoints unless asked, and the requests made", ct, final.Scheme, final.Endpoints, final.Requests) + } + if rec := get(h, contract.StatsRoute); rec.Code != http.StatusNotFound { + t.Errorf("GET /stats after stop = %d, want 404", rec.Code) + } + + // The final report is kept: a /stop the driver did not hear is answered + // again, byte for byte, with or without the endpoints, until a new loop + // replaces it. + if again := post(t, h, contract.StopRoute, nil); again.Code != http.StatusOK || !bytes.Equal(again.Body.Bytes(), rec.Body.Bytes()) { + t.Errorf("second POST /stop = %d %s; want 200 and the first answer %s", again.Code, again.Body, rec.Body) + } + if withEP := decodeStats(t, "third POST /stop", post(t, h, stopPath, nil)); len(withEP.Endpoints) != 1 || withEP.Requests != final.Requests { + t.Errorf("POST /stop?endpoints=1 after stop: %d endpoints, %d requests; want 1 and %d", len(withEP.Endpoints), withEP.Requests, final.Requests) + } + if rec := start(t, h, contract.StartRequest{Endpoints: 1, IntervalMs: 60_000, OffsetMs: 60_000}); rec.Code != http.StatusNoContent { + t.Fatalf("second POST /start = %d %s", rec.Code, rec.Body) + } + if next := stopLoop(t, h); next.Requests != 0 { + t.Errorf("the stop after a new /start reports %d requests; want 0, the new loop's own report", next.Requests) + } +} + +// TestRequestsAtOpeningDone pins the count the blended window prices: the +// requests this actor had finished when its last endpoint opened, repeats on +// the endpoints already open included, and fixed from then on. +func TestRequestsAtOpeningDone(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + const c, interval, openLatency = 2, 100 * time.Millisecond, 250 * time.Millisecond + var mu sync.Mutex + opened := map[string]bool{} + pn := newPipeNet(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + first := !opened[r.Host] + opened[r.Host] = true + mu.Unlock() + if first && r.Host == "endpoint-1" { + select { + case <-time.After(openLatency): + case <-r.Context().Done(): + return + } + } + okHandler(w, r) + })) + defer pn.srv.Close() + start := time.Now() + h, stats := startPipeLoop(t, pn, contract.StartRequest{Endpoints: c, IntervalMs: int64(interval / time.Millisecond)}) + + // Endpoint 0 opens at 0 and repeats at 200; endpoint 1 opens at 100 + // and finishes at 350: the third request. Its first repeat follows at + // once, since its period passed during the open. + time.Sleep(interval + openLatency - time.Since(start)) + st := stats() + if st.Opened != c || st.OpeningDone != interval+openLatency || st.RequestsAtOpeningDone != 3 || st.Requests != 4 { + t.Errorf("at %v: opened %d at %v with %d requests, %d requests now; want %d at %v with 3, 4", time.Since(start), st.Opened, st.OpeningDone, st.RequestsAtOpeningDone, st.Requests, c, interval+openLatency) + } + time.Sleep(time.Second - time.Since(start)) + if st = stats(); st.RequestsAtOpeningDone != 3 || st.Requests <= 4 { + t.Errorf("at %v: requestsAtOpeningDone %d with %d requests; want still 3 with more than 4", time.Since(start), st.RequestsAtOpeningDone, st.Requests) + } + if final := stopLoop(t, h); final.RequestsAtOpeningDone != 3 { + t.Errorf("final requestsAtOpeningDone = %d, want 3", final.RequestsAtOpeningDone) + } + }) +} + +// TestFreshRecorderPerLoop pins that a loop starts from nothing: after +// start-stop-start the new loop's counters and instruments are empty. +func TestFreshRecorderPerLoop(t *testing.T) { + t.Parallel() + h := serveTargets(t, 2) + first := runLoop(t, h, contract.StartRequest{Endpoints: 2}, atLeast(10)) + if samples(t, h, contract.Latency) == 0 { + t.Fatalf("first loop recorded no latency over %d requests", first.Requests) + } + // Nothing in the new loop fires for a minute, so its instruments hold + // only what the previous loop might have left behind. + if rec := start(t, h, contract.StartRequest{Endpoints: 2, IntervalMs: 60_000, OffsetMs: 60_000}); rec.Code != http.StatusNoContent { + t.Fatalf("POST /start = %d %s", rec.Code, rec.Body) + } + second := readStats(t, h) + if _, recorded := histogram(t, h, contract.Latency); second.Requests != 0 || second.Opened != 0 || recorded { + t.Errorf("second loop: %d requests, %d opened, latency recorded %v; want nothing", second.Requests, second.Opened, recorded) + } + stopLoop(t, h) +} + +// TestInstrumentsTrackCountersUnderRace reads the instruments while requests +// land: every success records exactly one opening or latency sample under +// the same lock as the counters, so a histogram count read between two +// /stats polls lies between their Successes, every /stats poll holds +// successes + errors = requests, and the final instruments equal the final +// counters. +func TestInstrumentsTrackCountersUnderRace(t *testing.T) { + t.Parallel() + h := serveTargets(t, 8) + if rec := start(t, h, contract.StartRequest{Endpoints: 8}); rec.Code != http.StatusNoContent { + t.Fatalf("POST /start = %d %s", rec.Code, rec.Body) + } + var wg sync.WaitGroup + var badReads atomic.Int64 + for range 4 { + wg.Go(func() { + for range 50 { + st := readStats(t, h) + var failed int64 + for _, n := range st.Errors { + failed += n + } + if st.Successes+failed != st.Requests { + badReads.Add(1) + } + } + }) + } + var outOfOrder int + for range 200 { + before := readStats(t, h) + got := samples(t, h, contract.Opening) + samples(t, h, contract.Latency) + after := readStats(t, h) + if int64(got) < before.Successes || int64(got) > after.Successes { + outOfOrder++ + } + } + wg.Wait() + final := stopLoop(t, h) + opening, latency := samples(t, h, contract.Opening), samples(t, h, contract.Latency) + if badReads.Load() > 0 || outOfOrder > 0 { + t.Errorf("%d reads broke successes + errors = requests, %d instrument reads fell outside the polls around them", badReads.Load(), outOfOrder) + } + if int64(opening+latency) != final.Successes || opening != 8 || final.Successes == 0 { + t.Errorf("final instruments hold %d opening and %d latency samples for %d successes over 8 endpoints", opening, latency, final.Successes) + } +} + +// TestStatsDuringStopIsNever500 races many reads against a stop: each one +// answers 200 while the loop runs or 404 once it is stopped, never a 500. +func TestStatsDuringStopIsNever500(t *testing.T) { + t.Parallel() + h := serveTargets(t, 8) + if rec := start(t, h, contract.StartRequest{Endpoints: 8}); rec.Code != http.StatusNoContent { + t.Fatalf("POST /start = %d %s", rec.Code, rec.Body) + } + var wg sync.WaitGroup + var ok, gone, other atomic.Int64 + stopped := make(chan struct{}) + for range 16 { + wg.Go(func() { + for { + rec := get(h, contract.StatsRoute) + switch { + case rec.Code == http.StatusOK: + ok.Add(1) + case rec.Code == http.StatusNotFound && strings.TrimSpace(rec.Body.String()) == contract.NoLoopMessage: + gone.Add(1) + default: + other.Add(1) + t.Errorf("answered %d %s", rec.Code, rec.Body) + } + select { + case <-stopped: + return + default: + } + } + }) + } + time.Sleep(20 * time.Millisecond) + stopLoop(t, h) + close(stopped) + wg.Wait() + if ok.Load() == 0 || gone.Load() == 0 || other.Load() != 0 { + t.Errorf("%d answers were 200, %d 404, %d something else", ok.Load(), gone.Load(), other.Load()) + } +} + +// TestEveryRouteCarriesProtocolHeader checks the header a driver reads to +// refuse an actor that speaks another contract, on every answer the actor +// gives, errors and unknown routes included. +func TestEveryRouteCarriesProtocolHeader(t *testing.T) { + t.Parallel() + if contract.ProtocolVersion != "1" { + t.Errorf("ProtocolVersion = %q, want 1; a driver requires it", contract.ProtocolVersion) + } + h := newServer().handler() + for _, tc := range []struct { + name string + rec *httptest.ResponseRecorder + want int + }{ + {"readyz", get(h, contract.ReadyzRoute), http.StatusOK}, + {"start rejected", post(t, h, contract.StartRoute, map[string]int{"endpoints": 0}), http.StatusBadRequest}, + {"stats without a loop", get(h, contract.StatsRoute), http.StatusNotFound}, + {"stop without a loop", post(t, h, contract.StopRoute, nil), http.StatusNotFound}, + {"unknown route", get(h, "/metrics"), http.StatusNotFound}, + } { + if tc.rec.Code != tc.want { + t.Errorf("%s: status %d, want %d", tc.name, tc.rec.Code, tc.want) + } + if got := tc.rec.Header().Get(contract.ProtocolHeader); got != contract.ProtocolVersion { + t.Errorf("%s: %s = %q, want %q", tc.name, contract.ProtocolHeader, got, contract.ProtocolVersion) + } + } +} + +// TestPayloadBudget pins the answer sizes at the largest C, since the driver +// polls every actor: /stats stays under 1 KiB, and the per-endpoint counts +// come only when asked for, under 80 KiB. +func TestPayloadBudget(t *testing.T) { + t.Parallel() + const c = contract.MaxEndpoints + run, err := newLoopRun(context.Background(), identified(contract.StartRequest{Endpoints: c, Scheme: contract.SchemeHTTPS}), nil) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = run.rec.Shutdown(context.Background()) }) + run.stats.Requests, run.stats.Successes, run.stats.NewConns, run.stats.Opened = 12345*c, 12339*c, c, c + run.stats.OpeningDone, run.stats.LongestGap = 102400*time.Millisecond, 12345678*time.Microsecond + run.stats.Errors = map[string]int64{"timeout": 3 * c, "HTTP 503": 3 * c} + run.rounds = 12345 + for i := range c { + run.stats.Endpoints[i] = contract.Endpoint{Requests: 12345, Errors: 6, LongestGap: 12345678 * time.Microsecond} + } + s := newServer() + s.run = run + h := s.handler() + + stats := get(h, contract.StatsRoute) + statsEndpoints := get(h, statsPath) + t.Logf("at C=%d: /stats %d B, /stats?endpoints=1 %d B", c, stats.Body.Len(), statsEndpoints.Body.Len()) + for _, tc := range []struct { + name string + rec *httptest.ResponseRecorder + budget int + endpoints bool + }{ + {"/stats", stats, 1 << 10, false}, + {"/stats?endpoints=1", statsEndpoints, 80 << 10, true}, + } { + if tc.rec.Code != http.StatusOK { + t.Fatalf("%s = %d %s", tc.name, tc.rec.Code, tc.rec.Body) + } + if tc.rec.Body.Len() >= tc.budget { + t.Errorf("%s = %d B, budget %d B", tc.name, tc.rec.Body.Len(), tc.budget) + } + var st contract.Stats + if err := json.Unmarshal(tc.rec.Body.Bytes(), &st); err != nil { + t.Fatal(err) + } + if (len(st.Endpoints) == c) != tc.endpoints || st.Rounds != 12345 { + t.Errorf("%s carries %d endpoints and rounds %d; want endpoints %v and rounds 12345", tc.name, len(st.Endpoints), st.Rounds, tc.endpoints) + } + } +} diff --git a/cmd/benchmarking/egress/internal/cmd/actor/classify.go b/cmd/benchmarking/egress/internal/cmd/actor/classify.go new file mode 100644 index 0000000000..09e752a18f --- /dev/null +++ b/cmd/benchmarking/egress/internal/cmd/actor/classify.go @@ -0,0 +1,95 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package actor + +import ( + "context" + "crypto/tls" + "crypto/x509" + "errors" + "fmt" + "io" + "net" + "net/url" + "strings" + "syscall" +) + +// maxOtherLen bounds the message kept for an error classify does not +// recognize, so one odd error cannot flood the report. +const maxOtherLen = 80 + +// classify names the outcome of a request in a few low-cardinality classes. +// It returns "" for a success: no error and HTTP 200. +func classify(err error, status int) string { + if err == nil { + if status == 200 { + return "" + } + return fmt.Sprintf("HTTP %d", status) + } + var netErr net.Error + var dnsErr *net.DNSError + var opErr *net.OpError + switch { + case errors.As(err, &dnsErr): + if dnsErr.IsNotFound { + return "dns: not found" + } + if dnsErr.IsTimeout { + return "dns: timeout" + } + return "dns: error" + case errors.As(err, new(x509.UnknownAuthorityError)): + return "tls: unknown authority" + case errors.As(err, new(x509.HostnameError)): + return "tls: hostname mismatch" + case errors.As(err, new(x509.CertificateInvalidError)): + return "tls: invalid certificate" + case errors.As(err, new(tls.RecordHeaderError)), + // net/http replaces the record error with this when the server + // answered in plain HTTP, without wrapping it. + strings.Contains(err.Error(), "server gave HTTP response to HTTPS client"): + return "tls: not a TLS server" + case errors.As(err, &opErr) && opErr.Op == "remote error": + // crypto/tls reports an alert from the peer this way; its message, + // such as "tls: handshake failure", names the alert. + return "tls: peer alert: " + strings.TrimPrefix(opErr.Err.Error(), "tls: ") + case errors.Is(err, context.DeadlineExceeded), errors.As(err, &netErr) && netErr.Timeout(): + return "timeout" + case errors.Is(err, syscall.ECONNREFUSED): + return "connection refused" + case errors.Is(err, syscall.ECONNRESET): + return "connection reset" + case errors.Is(err, io.EOF), errors.Is(err, io.ErrUnexpectedEOF): + return "EOF" + } + // *url.Error repeats the method and URL, and *net.OpError the addresses + // and the local port; both only add cardinality. + var urlErr *url.Error + if errors.As(err, &urlErr) { + err = urlErr.Err + } + if errors.As(err, &opErr) { + return "net: " + opErr.Op + ": " + opErr.Err.Error() + } + msg := err.Error() + if len(msg) > maxOtherLen { + // Cut on a character boundary, or one message could yield two + // classes depending on where its bytes fall. + msg = strings.ToValidUTF8(msg[:maxOtherLen], "") + } + return "other: " + msg +} diff --git a/cmd/benchmarking/egress/internal/cmd/actor/classify_test.go b/cmd/benchmarking/egress/internal/cmd/actor/classify_test.go new file mode 100644 index 0000000000..0728af73e3 --- /dev/null +++ b/cmd/benchmarking/egress/internal/cmd/actor/classify_test.go @@ -0,0 +1,79 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package actor + +import ( + "context" + "crypto/tls" + "crypto/x509" + "errors" + "io" + "net" + "net/url" + "strings" + "syscall" + "testing" +) + +type timeoutErr struct{} + +func (timeoutErr) Error() string { return "i/o timeout" } +func (timeoutErr) Timeout() bool { return true } +func (timeoutErr) Temporary() bool { return true } + +func TestClassify(t *testing.T) { + t.Parallel() + wrap := func(err error) error { return &url.Error{Op: "Get", URL: "http://x/", Err: err} } + for _, tc := range []struct { + name string + err error + status int + want string + }{ + {"success", nil, 200, ""}, + {"http status", nil, 503, "HTTP 503"}, + {"deadline", wrap(context.DeadlineExceeded), 0, "timeout"}, + {"net timeout", wrap(&net.OpError{Op: "dial", Err: timeoutErr{}}), 0, "timeout"}, + {"refused", wrap(&net.OpError{Op: "dial", Err: syscall.ECONNREFUSED}), 0, "connection refused"}, + {"reset", wrap(&net.OpError{Op: "read", Err: syscall.ECONNRESET}), 0, "connection reset"}, + {"eof", wrap(io.EOF), 0, "EOF"}, + {"unexpected eof", wrap(io.ErrUnexpectedEOF), 0, "EOF"}, + {"dns not found", wrap(&net.DNSError{Err: "no such host", Name: "x", IsNotFound: true}), 0, "dns: not found"}, + {"dns other", wrap(&net.DNSError{Err: "server misbehaving", Name: "x"}), 0, "dns: error"}, + {"dns timeout", wrap(&net.DNSError{Err: "i/o timeout", Name: "x", IsTimeout: true}), 0, "dns: timeout"}, + {"long other is cut", wrap(errors.New(strings.Repeat("x", maxOtherLen+20))), 0, "other: " + strings.Repeat("x", maxOtherLen)}, + {"cut keeps whole characters", wrap(errors.New(strings.Repeat("a", maxOtherLen-1) + "é!")), 0, "other: " + strings.Repeat("a", maxOtherLen-1)}, + {"other strips url", wrap(errors.New("boom")), 0, "other: boom"}, + {"unknown authority", wrap(&tls.CertificateVerificationError{Err: x509.UnknownAuthorityError{}}), 0, "tls: unknown authority"}, + {"hostname mismatch", wrap(&tls.CertificateVerificationError{Err: x509.HostnameError{Host: "x"}}), 0, "tls: hostname mismatch"}, + {"expired", wrap(&tls.CertificateVerificationError{Err: x509.CertificateInvalidError{Reason: x509.Expired}}), 0, "tls: invalid certificate"}, + {"not tls", wrap(tls.RecordHeaderError{Msg: "first record does not look like a TLS handshake"}), 0, "tls: not a TLS server"}, + {"plain http answer", wrap(errors.New("http: server gave HTTP response to HTTPS client")), 0, "tls: not a TLS server"}, + {"peer alert", wrap(&net.OpError{Op: "remote error", Err: errors.New("tls: handshake failure")}), 0, "tls: peer alert: handshake failure"}, + {"net error without its addresses", &url.Error{Op: "Get", URL: "http://10.0.0.1:8080/", Err: &net.OpError{ + Op: "write", Net: "tcp", + Source: &net.TCPAddr{IP: net.IPv4(10, 0, 0, 2), Port: 51234}, + Addr: &net.TCPAddr{IP: net.IPv4(10, 0, 0, 1), Port: 8080}, + Err: syscall.EPIPE, + }}, 0, "net: write: broken pipe"}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + if got := classify(tc.err, tc.status); got != tc.want { + t.Errorf("classify() = %q, want %q", got, tc.want) + } + }) + } +} diff --git a/cmd/benchmarking/egress/internal/cmd/actor/export_test.go b/cmd/benchmarking/egress/internal/cmd/actor/export_test.go new file mode 100644 index 0000000000..e2a243fc35 --- /dev/null +++ b/cmd/benchmarking/egress/internal/cmd/actor/export_test.go @@ -0,0 +1,276 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package actor + +import ( + "context" + "crypto/ecdsa" + "crypto/elliptic" + "crypto/rand" + "crypto/tls" + "crypto/x509" + "crypto/x509/pkix" + "maps" + "math/big" + "net" + "net/http" + "net/http/httptest" + "net/url" + "slices" + "sync" + "testing" + "time" + + collectormetrics "go.opentelemetry.io/proto/otlp/collector/metrics/v1" + commonpb "go.opentelemetry.io/proto/otlp/common/v1" + metricspb "go.opentelemetry.io/proto/otlp/metrics/v1" + "google.golang.org/grpc" + + "go.ate.dev/substrate/cmd/benchmarking/egress/internal/contract" +) + +// fakeCollector is an OTLP/gRPC metrics receiver on a loopback listener +// that keeps every export it was sent. +type fakeCollector struct { + collectormetrics.UnimplementedMetricsServiceServer + addr string + mu sync.Mutex + exports []*collectormetrics.ExportMetricsServiceRequest +} + +func (f *fakeCollector) Export(_ context.Context, req *collectormetrics.ExportMetricsServiceRequest) (*collectormetrics.ExportMetricsServiceResponse, error) { + f.mu.Lock() + defer f.mu.Unlock() + f.exports = append(f.exports, req) + return &collectormetrics.ExportMetricsServiceResponse{}, nil +} + +// received returns the exports so far. +func (f *fakeCollector) received() []*collectormetrics.ExportMetricsServiceRequest { + f.mu.Lock() + defer f.mu.Unlock() + return slices.Clone(f.exports) +} + +func startFakeCollector(t *testing.T) *fakeCollector { + t.Helper() + lis, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + f := &fakeCollector{addr: lis.Addr().String()} + srv := grpc.NewServer() + collectormetrics.RegisterMetricsServiceServer(srv, f) + go srv.Serve(lis) + t.Cleanup(srv.Stop) + return f +} + +// exportEnv points the SDK at f with a short interval, as the template's +// environment points the actor at the collector. The environment is process +// wide, so a test that calls it cannot be parallel. +func exportEnv(t *testing.T, f *fakeCollector) { + t.Helper() + t.Setenv(endpointEnv, "http://"+f.addr) + t.Setenv("OTEL_METRIC_EXPORT_INTERVAL", "100") + t.Setenv("OTEL_METRIC_EXPORT_TIMEOUT", "2000") + t.Setenv("OTEL_RESOURCE_ATTRIBUTES", "service.name="+serviceName) +} + +// attr returns the string attribute key of kvs, "" when absent. +func attr(kvs []*commonpb.KeyValue, key string) string { + for _, kv := range kvs { + if kv.Key == key { + return kv.Value.GetStringValue() + } + } + return "" +} + +// metricsOf flattens one export into its metrics by name, checking the +// resource and the scope on the way. +func metricsOf(t *testing.T, req *collectormetrics.ExportMetricsServiceRequest) map[string]*metricspb.Metric { + t.Helper() + out := map[string]*metricspb.Metric{} + for _, rm := range req.ResourceMetrics { + if name := attr(rm.Resource.Attributes, "service.name"); name != serviceName { + t.Errorf("resource service.name = %q, want %q", name, serviceName) + } + if attr(rm.Resource.Attributes, "service.instance.id") == "" { + t.Errorf("resource has no service.instance.id: %v", rm.Resource.Attributes) + } + for _, sm := range rm.ScopeMetrics { + if sm.Scope.Name != meterName { + t.Errorf("scope %q, want %q", sm.Scope.Name, meterName) + } + for _, m := range sm.Metrics { + out[m.Name] = m + } + } + } + return out +} + +// instanceOf is the service.instance.id of one export. +func instanceOf(req *collectormetrics.ExportMetricsServiceRequest) string { + if len(req.ResourceMetrics) == 0 { + return "" + } + return attr(req.ResourceMetrics[0].Resource.Attributes, "service.instance.id") +} + +// sumOf adds the integer points of a counter. +func sumOf(m *metricspb.Metric) int64 { + var n int64 + for _, p := range m.GetSum().GetDataPoints() { + n += p.GetAsInt() + } + return n +} + +// localhostTLSTarget is an HTTPS target with a certificate for localhost, so a +// loop that names it does a lookup and a handshake on every open. +func localhostTLSTarget(t *testing.T, h http.HandlerFunc) (host string, cert *x509.Certificate) { + t.Helper() + key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + if err != nil { + t.Fatal(err) + } + tmpl := &x509.Certificate{ + SerialNumber: big.NewInt(1), + Subject: pkix.Name{CommonName: "localhost"}, + DNSNames: []string{"localhost"}, + NotBefore: time.Now().Add(-time.Hour), + NotAfter: time.Now().Add(time.Hour), + KeyUsage: x509.KeyUsageDigitalSignature, + ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth}, + } + der, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, &key.PublicKey, key) + if err != nil { + t.Fatal(err) + } + cert, err = x509.ParseCertificate(der) + if err != nil { + t.Fatal(err) + } + srv := httptest.NewUnstartedServer(h) + srv.TLS = &tls.Config{Certificates: []tls.Certificate{{Certificate: [][]byte{der}, PrivateKey: key, Leaf: cert}}} + srv.Config.ErrorLog = quietLog + srv.StartTLS() + t.Cleanup(srv.Close) + u, err := url.Parse(srv.URL) + if err != nil { + t.Fatal(err) + } + return "localhost:" + u.Port(), cert +} + +// TestExportReachesTheCollector runs a loop against a fake OTLP receiver, as +// the actor runs against the benchmark's collector, and pins what arrives: +// the resource, the eleven names, the three attributes on every point, +// cumulative temporality, and a final export after /stop that agrees with +// the counters /stop answered. +func TestExportReachesTheCollector(t *testing.T) { + f := startFakeCollector(t) + exportEnv(t, f) + okHost, okCert := localhostTLSTarget(t, okHandler) + failHost, failCert := localhostTLSTarget(t, func(w http.ResponseWriter, _ *http.Request) { http.Error(w, "denied", http.StatusServiceUnavailable) }) + roots := x509.NewCertPool() + roots.AddCert(okCert) + roots.AddCert(failCert) + h := serveTLS(roots, okHost, failHost) + + req := contract.StartRequest{Endpoints: 2, Scheme: contract.SchemeHTTPS, IntervalMs: 2} + final := runLoop(t, h, req, func(s *contract.Stats) bool { return s.Rounds >= 3 && s.Errors["HTTP 503"] > 0 }) + exports := f.received() + if len(exports) == 0 { + t.Fatal("the collector received nothing") + } + last := metricsOf(t, exports[len(exports)-1]) + for _, name := range metricNames { + if _, ok := last[name]; !ok { + t.Errorf("the final export lacks %s; it has %v", name, slices.Sorted(maps.Keys(last))) + } + } + for name, m := range last { + var points []*commonpb.KeyValue + var temporality metricspb.AggregationTemporality + switch d := m.Data.(type) { + case *metricspb.Metric_Sum: + temporality = d.Sum.AggregationTemporality + for _, p := range d.Sum.DataPoints { + points = append(points, p.Attributes...) + } + case *metricspb.Metric_Histogram: + temporality = d.Histogram.AggregationTemporality + for _, p := range d.Histogram.DataPoints { + points = append(points, p.Attributes...) + if len(p.ExplicitBounds) != len(contract.Bounds) || len(p.BucketCounts) != len(contract.Bounds)+1 { + t.Errorf("%s point has %d bounds and %d buckets, want %d and %d", name, len(p.ExplicitBounds), len(p.BucketCounts), len(contract.Bounds), len(contract.Bounds)+1) + } + } + case *metricspb.Metric_Gauge: + temporality = metricspb.AggregationTemporality_AGGREGATION_TEMPORALITY_CUMULATIVE + for _, p := range d.Gauge.DataPoints { + points = append(points, p.Attributes...) + } + default: + t.Errorf("%s arrived as %T", name, m.Data) + } + if temporality != metricspb.AggregationTemporality_AGGREGATION_TEMPORALITY_CUMULATIVE { + t.Errorf("%s arrived with temporality %v, want cumulative", name, temporality) + } + if a, c, r := attr(points, "actor"), attr(points, "cell"), attr(points, "run"); a != identity.Actor || c != identity.Cell || r != identity.Run { + t.Errorf("%s points carry actor %q cell %q run %q, want %q %q %q", name, a, c, r, identity.Actor, identity.Cell, identity.Run) + } + } + if got := sumOf(last[requestsName]); got != final.Requests { + t.Errorf("final export: requests %d, /stop answered %d", got, final.Requests) + } + if got := sumOf(last[requestErrorsName]); got != final.Requests-final.Successes { + t.Errorf("final export: request_errors %d, /stop answered %d failures", got, final.Requests-final.Successes) + } + if got := sumOf(last[endpointsOpenedName]); got != final.Opened { + t.Errorf("final export: endpoints_opened %d, /stop answered %d", got, final.Opened) + } + + // A second loop in the same process is a new instance from zero. + first := instanceOf(exports[len(exports)-1]) + before := len(exports) + runLoop(t, h, req, atLeast(1)) + exports = f.received() + if len(exports) <= before { + t.Fatal("the second loop exported nothing") + } + if second := instanceOf(exports[len(exports)-1]); second == first || second == "" { + t.Errorf("second loop exported as instance %q, first as %q; want two ids", second, first) + } +} + +// TestNoEndpointMeansNoExport pins the actor without the template's +// environment: /start works, the loop runs, and nothing is pushed anywhere. +func TestNoEndpointMeansNoExport(t *testing.T) { + f := startFakeCollector(t) + exportEnv(t, f) + t.Setenv(endpointEnv, "") + h := serveTargets(t, 2) + final := runLoop(t, h, contract.StartRequest{Endpoints: 2}, atLeast(5)) + if final.Requests < 5 || final.Successes != final.Requests { + t.Errorf("loop without an endpoint: %d requests, %d successes", final.Requests, final.Successes) + } + if got := f.received(); len(got) != 0 { + t.Errorf("the collector received %d exports with the endpoint env unset", len(got)) + } +} diff --git a/cmd/benchmarking/egress/internal/cmd/actor/recorder.go b/cmd/benchmarking/egress/internal/cmd/actor/recorder.go new file mode 100644 index 0000000000..0448a57c41 --- /dev/null +++ b/cmd/benchmarking/egress/internal/cmd/actor/recorder.go @@ -0,0 +1,192 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// Package actor is the workload of the egress scale test: the subcommand +// that holds one tunnel per endpoint open through the egress gateway and +// counts what happened. +package actor + +import ( + "context" + "errors" + "fmt" + "log/slog" + "os" + + "github.com/google/uuid" + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc" + "go.opentelemetry.io/otel/metric" + sdkmetric "go.opentelemetry.io/otel/sdk/metric" + "go.opentelemetry.io/otel/sdk/metric/metricdata" + "go.opentelemetry.io/otel/sdk/resource" + semconv "go.opentelemetry.io/otel/semconv/v1.40.0" + + "go.ate.dev/substrate/cmd/benchmarking/egress/internal/contract" +) + +// The names of the loop's instruments, the contract with the queries that +// read them once the collector has served them to Prometheus: a counter +// gains _total there, a unit of seconds _seconds. +const ( + meterName = "benchmark-egress" + requestsName = "requests" + requestErrorsName = "request_errors" + connsOpenedName = "connections_opened" + endpointsOpenedName = "endpoints_opened" + roundsName = "rounds" + openingDoneName = "opening_done" + longestGapName = "longest_gap" +) + +// serviceName is the resource's service.name unless the environment sets +// one; the collector's exporter writes it as the job label. +const serviceName = "benchmark-egress-actor" + +// endpointEnv is the OTLP endpoint the SDK reads; set, the loop exports. +const endpointEnv = "OTEL_EXPORTER_OTLP_ENDPOINT" + +// Recorder holds one loop's instruments on a private MeterProvider: the +// counters that mirror its Stats and the four latency histograms, every point +// under the loop's actor, cell and run. With OTEL_EXPORTER_OTLP_ENDPOINT set +// the provider pushes them over OTLP at the interval the environment names; +// without it the provider has no reader and every method is a cheap no-op. +type Recorder struct { + provider *sdkmetric.MeterProvider + hists map[contract.Distribution]metric.Float64Histogram + + requests, requestErrors, connsOpened, endpointsOpened metric.Int64Counter + rounds metric.Int64Gauge + openingDone, longestGap metric.Float64Gauge + + // attrs is on every point; success and failure add the result. + attrs, success, failure metric.MeasurementOption +} + +// NewRecorder creates the instruments for the loop req starts, with the +// OTLP reader when the environment names an endpoint, plus readers, which +// tests use to read the instruments without an exporter. The resource takes +// service.name from the environment and a fresh service.instance.id, so a +// restarted actor never continues the old series. +func NewRecorder(ctx context.Context, req contract.StartRequest, readers ...sdkmetric.Reader) (*Recorder, error) { + res, err := resource.New(ctx, + resource.WithAttributes(semconv.ServiceName(serviceName)), + resource.WithFromEnv(), + resource.WithAttributes(semconv.ServiceInstanceID(uuid.NewString())), + ) + if errors.Is(err, resource.ErrPartialResource) { + slog.WarnContext(ctx, "partial telemetry resource", "err", err) + } else if err != nil { + return nil, fmt.Errorf("building the resource: %w", err) + } + opts := []sdkmetric.Option{sdkmetric.WithResource(res)} + if os.Getenv(endpointEnv) != "" { + exp, err := otlpmetricgrpc.New(ctx, otlpmetricgrpc.WithTemporalitySelector(cumulative)) + if err != nil { + return nil, fmt.Errorf("creating the OTLP exporter: %w", err) + } + opts = append(opts, sdkmetric.WithReader(sdkmetric.NewPeriodicReader(exp))) + } + for _, rd := range readers { + opts = append(opts, sdkmetric.WithReader(rd)) + } + r := &Recorder{provider: sdkmetric.NewMeterProvider(opts...), hists: make(map[contract.Distribution]metric.Float64Histogram, len(contract.DistributionNames))} + loop := attribute.NewSet(attribute.String("actor", req.Actor), attribute.String("cell", req.Cell), attribute.String("run", req.Run)) + withResult := func(result string) metric.MeasurementOption { + return metric.WithAttributeSet(attribute.NewSet(append(loop.ToSlice(), attribute.String("result", result))...)) + } + r.attrs, r.success, r.failure = metric.WithAttributeSet(loop), withResult("success"), withResult("error") + if err := r.instruments(r.provider.Meter(meterName)); err != nil { + _ = r.provider.Shutdown(context.Background()) + return nil, err + } + return r, nil +} + +// cumulative is the SDK's default temporality, set explicitly so that +// OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE in the environment cannot +// switch the exporter to delta, which would break every difference between two +// instants a run computes. +func cumulative(sdkmetric.InstrumentKind) metricdata.Temporality { + return metricdata.CumulativeTemporality +} + +// instruments creates the eleven instruments on meter. +func (r *Recorder) instruments(meter metric.Meter) error { + var err error + counter := func(name, desc string) metric.Int64Counter { + c, cerr := meter.Int64Counter(name, metric.WithDescription(desc)) + err = errors.Join(err, cerr) + return c + } + r.requests = counter(requestsName, "Requests the loop finished, by result.") + r.requestErrors = counter(requestErrorsName, "Failed requests, by class.") + r.connsOpened = counter(connsOpenedName, "Requests sent on a freshly dialed connection.") + r.endpointsOpened = counter(endpointsOpenedName, "Endpoints whose first request has completed.") + var gerr error + r.rounds, gerr = meter.Int64Gauge(roundsName, metric.WithDescription("Requests completed by the least-visited endpoint.")) + err = errors.Join(err, gerr) + r.openingDone, gerr = meter.Float64Gauge(openingDoneName, metric.WithUnit("s"), metric.WithDescription("Seconds from /start to the last endpoint's open; zero until then.")) + err = errors.Join(err, gerr) + r.longestGap, gerr = meter.Float64Gauge(longestGapName, metric.WithUnit("s"), metric.WithDescription("Longest gap between two consecutive requests on one endpoint.")) + err = errors.Join(err, gerr) + for _, name := range contract.DistributionNames { + h, herr := meter.Float64Histogram(string(name), metric.WithUnit("s"), metric.WithExplicitBucketBoundaries(contract.Bounds...)) + err = errors.Join(err, herr) + r.hists[name] = h + } + if err != nil { + return fmt.Errorf("creating the instruments: %w", err) + } + return nil +} + +// Record adds one sample, in seconds, to the named distribution. +func (r *Recorder) Record(ctx context.Context, name contract.Distribution, seconds float64) { + r.hists[name].Record(ctx, seconds, r.attrs) +} + +// Request counts one finished request by result. +func (r *Recorder) Request(ctx context.Context, ok bool) { + if ok { + r.requests.Add(ctx, 1, r.success) + } else { + r.requests.Add(ctx, 1, r.failure) + } +} + +// RequestError counts one failed request under its class. +func (r *Recorder) RequestError(ctx context.Context, class string) { + r.requestErrors.Add(ctx, 1, r.attrs, metric.WithAttributes(attribute.String("class", class))) +} + +// ConnectionOpened counts one request sent on a freshly dialed connection. +func (r *Recorder) ConnectionOpened(ctx context.Context) { r.connsOpened.Add(ctx, 1, r.attrs) } + +// EndpointOpened counts one endpoint whose first request completed. +func (r *Recorder) EndpointOpened(ctx context.Context) { r.endpointsOpened.Add(ctx, 1, r.attrs) } + +// Gauges records the three derived values of the Stats. +func (r *Recorder) Gauges(ctx context.Context, rounds int64, openingDone, longestGap float64) { + r.rounds.Record(ctx, rounds, r.attrs) + r.openingDone.Record(ctx, openingDone, r.attrs) + r.longestGap.Record(ctx, longestGap, r.attrs) +} + +// ForceFlush exports what every reader holds, now. +func (r *Recorder) ForceFlush(ctx context.Context) error { return r.provider.ForceFlush(ctx) } + +// Shutdown exports once more and releases the provider. Every method is a +// no-op afterwards. +func (r *Recorder) Shutdown(ctx context.Context) error { return r.provider.Shutdown(ctx) } diff --git a/cmd/benchmarking/egress/internal/cmd/actor/recorder_test.go b/cmd/benchmarking/egress/internal/cmd/actor/recorder_test.go new file mode 100644 index 0000000000..fd065ba997 --- /dev/null +++ b/cmd/benchmarking/egress/internal/cmd/actor/recorder_test.go @@ -0,0 +1,228 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package actor + +import ( + "context" + "reflect" + "slices" + "testing" + + "go.opentelemetry.io/otel/attribute" + sdkmetric "go.opentelemetry.io/otel/sdk/metric" + "go.opentelemetry.io/otel/sdk/metric/metricdata" + + "go.ate.dev/substrate/cmd/benchmarking/egress/internal/contract" +) + +// identity is the actor, cell and run every test loop starts under. +var identity = contract.StartRequest{Actor: "benchmark-workloads/egress-0", Cell: "cell-1", Run: "run-1"} + +// identified is req with the identity fields filled where a test left them +// empty, so a test states only what it is about. +func identified(req contract.StartRequest) contract.StartRequest { + if req.Actor == "" { + req.Actor = identity.Actor + } + if req.Cell == "" { + req.Cell = identity.Cell + } + if req.Run == "" { + req.Run = identity.Run + } + return req +} + +// loopAttrs are the attributes every point of a loop started with identity +// carries. +var loopAttrs = attribute.NewSet(attribute.String("actor", identity.Actor), attribute.String("cell", identity.Cell), attribute.String("run", identity.Run)) + +// metricNames are the eleven instruments of the contract, as the SDK names +// them before the collector adds its suffixes. +var metricNames = []string{requestsName, requestErrorsName, connsOpenedName, endpointsOpenedName, roundsName, openingDoneName, longestGapName, + string(contract.Opening), string(contract.Latency), string(contract.DNS), string(contract.TLS)} + +// newTestRecorder is a Recorder for identity with a manual reader on it. +func newTestRecorder(t *testing.T) (*Recorder, *sdkmetric.ManualReader) { + t.Helper() + reader := sdkmetric.NewManualReader() + r, err := NewRecorder(context.Background(), identity, reader) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = r.Shutdown(context.Background()) }) + return r, reader +} + +func collect(t *testing.T, reader *sdkmetric.ManualReader) metricdata.ResourceMetrics { + t.Helper() + var rm metricdata.ResourceMetrics + if err := reader.Collect(context.Background(), &rm); err != nil { + t.Fatal(err) + } + return rm +} + +// metricsByName indexes the one scope's metrics, failing on any other scope. +func metricsByName(t *testing.T, rm metricdata.ResourceMetrics) map[string]metricdata.Metrics { + t.Helper() + out := map[string]metricdata.Metrics{} + for _, sm := range rm.ScopeMetrics { + if sm.Scope.Name != meterName { + t.Errorf("scope %q, want only %q", sm.Scope.Name, meterName) + } + for _, m := range sm.Metrics { + out[m.Name] = m + } + } + return out +} + +// recordOneOfEach puts one measurement into every instrument. +func recordOneOfEach(ctx context.Context, r *Recorder) { + r.Request(ctx, true) + r.Request(ctx, false) + r.RequestError(ctx, "timeout") + r.ConnectionOpened(ctx) + r.EndpointOpened(ctx) + r.Gauges(ctx, 2, 1.5, 0.25) + for i, name := range contract.DistributionNames { + r.Record(ctx, name, float64(1+i)*1e-3) + } +} + +// pointAttrs returns the attribute set of every data point of m. +func pointAttrs(t *testing.T, m metricdata.Metrics) []attribute.Set { + t.Helper() + var sets []attribute.Set + switch d := m.Data.(type) { + case metricdata.Sum[int64]: + for _, p := range d.DataPoints { + sets = append(sets, p.Attributes) + } + case metricdata.Gauge[int64]: + for _, p := range d.DataPoints { + sets = append(sets, p.Attributes) + } + case metricdata.Gauge[float64]: + for _, p := range d.DataPoints { + sets = append(sets, p.Attributes) + } + case metricdata.Histogram[float64]: + for _, p := range d.DataPoints { + sets = append(sets, p.Attributes) + } + default: + t.Errorf("%s has data of type %T", m.Name, m.Data) + } + return sets +} + +// TestInstrumentsCarryTheLoopAttributes pins the wire contract of the +// recorder: the eleven names, the units that become _seconds, the shared +// bounds, and actor, cell and run on every point, with result and class +// beside them where the contract says so. +func TestInstrumentsCarryTheLoopAttributes(t *testing.T) { + t.Parallel() + r, reader := newTestRecorder(t) + recordOneOfEach(context.Background(), r) + byName := metricsByName(t, collect(t, reader)) + + var names []string + for name := range byName { + names = append(names, name) + } + slices.Sort(names) + if want := slices.Sorted(slices.Values(metricNames)); !slices.Equal(names, want) { + t.Errorf("instruments = %v, want %v", names, want) + } + for name, m := range byName { + wantUnit := "" + if slices.Contains([]string{openingDoneName, longestGapName}, name) || slices.Contains(contract.DistributionNames, contract.Distribution(name)) { + wantUnit = "s" + } + if m.Unit != wantUnit { + t.Errorf("%s has unit %q, want %q", name, m.Unit, wantUnit) + } + sets := pointAttrs(t, m) + if len(sets) == 0 { + t.Errorf("%s has no data point", name) + } + for _, set := range sets { + for _, kv := range loopAttrs.ToSlice() { + if v, ok := set.Value(kv.Key); !ok || v != kv.Value { + t.Errorf("%s point %v lacks %s=%s", name, set.Encoded(attribute.DefaultEncoder()), kv.Key, kv.Value.AsString()) + } + } + extra := set.Len() - loopAttrs.Len() + switch name { + case requestsName: + if v, ok := set.Value("result"); extra != 1 || !ok || (v.AsString() != "success" && v.AsString() != "error") { + t.Errorf("requests point has %v, want the loop attributes plus result", set.Encoded(attribute.DefaultEncoder())) + } + case requestErrorsName: + if v, ok := set.Value("class"); extra != 1 || !ok || v.AsString() != "timeout" { + t.Errorf("request_errors point has %v, want the loop attributes plus class", set.Encoded(attribute.DefaultEncoder())) + } + default: + if extra != 0 { + t.Errorf("%s point has %v, want the loop attributes alone", name, set.Encoded(attribute.DefaultEncoder())) + } + } + } + } + if sum, ok := byName[requestsName].Data.(metricdata.Sum[int64]); !ok || len(sum.DataPoints) != 2 || !sum.IsMonotonic || sum.Temporality != metricdata.CumulativeTemporality { + t.Errorf("requests = %+v, want a cumulative monotonic sum with a success and an error point", byName[requestsName].Data) + } + if g, ok := byName[roundsName].Data.(metricdata.Gauge[int64]); !ok || g.DataPoints[0].Value != 2 { + t.Errorf("rounds = %+v, want a gauge at 2", byName[roundsName].Data) + } + if g, ok := byName[openingDoneName].Data.(metricdata.Gauge[float64]); !ok || g.DataPoints[0].Value != 1.5 { + t.Errorf("opening_done = %+v, want a gauge at 1.5", byName[openingDoneName].Data) + } + for i, name := range contract.DistributionNames { + h, ok := byName[string(name)].Data.(metricdata.Histogram[float64]) + if !ok || len(h.DataPoints) != 1 { + t.Fatalf("%s = %+v, want a histogram with one point", name, byName[string(name)].Data) + } + p := h.DataPoints[0] + if !reflect.DeepEqual(p.Bounds, contract.Bounds) || p.Count != 1 || p.Sum != float64(1+i)*1e-3 || len(p.BucketCounts) != len(contract.Bounds)+1 { + t.Errorf("%s point: %d bounds, count %d, sum %v; want contract.Bounds, 1, %v", name, len(p.Bounds), p.Count, p.Sum, float64(1+i)*1e-3) + } + } +} + +// TestResourceDefaultsTheServiceName pins the recorder's own service.name, +// the job the series are stored under when the environment sets none. +func TestResourceDefaultsTheServiceName(t *testing.T) { + t.Setenv("OTEL_RESOURCE_ATTRIBUTES", "") + _, reader := newTestRecorder(t) + if name, _ := collect(t, reader).Resource.Set().Value("service.name"); name.AsString() != serviceName { + t.Errorf("service.name without the env = %q, want %q", name.AsString(), serviceName) + } +} + +func BenchmarkRecord(b *testing.B) { + r, err := NewRecorder(context.Background(), identity) + if err != nil { + b.Fatal(err) + } + defer r.Shutdown(context.Background()) + ctx := context.Background() + for i := 0; b.Loop(); i++ { + r.Request(ctx, true) + r.Record(ctx, contract.Latency, float64(i%1000)*1e-5) + } +} diff --git a/cmd/benchmarking/egress/internal/cmd/target/target.go b/cmd/benchmarking/egress/internal/cmd/target/target.go new file mode 100644 index 0000000000..1932184105 --- /dev/null +++ b/cmd/benchmarking/egress/internal/cmd/target/target.go @@ -0,0 +1,208 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// Package target is the endpoint every tunnel of the egress scale test leads +// to: the subcommand that answers each request with a small fixed body. +package target + +import ( + "bytes" + "context" + "crypto/tls" + "errors" + "fmt" + "log/slog" + "net" + "net/http" + "strconv" + "strings" + "sync" + "time" + + "github.com/spf13/cobra" +) + +// targetOptions are the flags of the target subcommand. +type targetOptions struct { + listen, listenPorts string + tlsListen, tlsListenPorts string + tlsCertFile, tlsKeyFile string + responseBytes int +} + +// Command is the endpoint the benchmark-egress actors call: every request +// gets a 200 with a small fixed body, over plain HTTP and, when given a +// certificate, HTTPS. One Deployment of it sits behind all of the test's +// Services. --listen-ports and --tls-listen-ports give each Service its own +// backend port: Services on distinct ClusterIPs that share one backend port +// can hand the target two connections with the same 4-tuple. +func Command() *cobra.Command { + var o targetOptions + cmd := &cobra.Command{ + Use: "target", + Short: "Answer every request the actors' tunnels carry with a small fixed body", + Args: cobra.NoArgs, + RunE: func(cmd *cobra.Command, _ []string) error { + return runTarget(cmd.Context(), o) + }, + } + cmd.Flags().StringVar(&o.listen, "listen", ":8080", "Address to serve plain HTTP on.") + cmd.Flags().StringVar(&o.listenPorts, "listen-ports", "", "Also serve plain HTTP on each port of this range, A-B.") + cmd.Flags().StringVar(&o.tlsListen, "tls-listen", "", "Address to serve HTTPS on; empty serves no HTTPS.") + cmd.Flags().StringVar(&o.tlsListenPorts, "tls-listen-ports", "", "Also serve HTTPS on each port of this range, A-B.") + cmd.Flags().StringVar(&o.tlsCertFile, "tls-cert-file", "", "PEM certificate to serve HTTPS with.") + cmd.Flags().StringVar(&o.tlsKeyFile, "tls-key-file", "", "PEM private key of --tls-cert-file.") + cmd.Flags().IntVar(&o.responseBytes, "response-bytes", 16, "Size of the body every request gets back.") + return cmd +} + +func newHandler(responseBytes int) http.Handler { + body := bytes.Repeat([]byte("x"), responseBytes) + mux := http.NewServeMux() + mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, _ *http.Request) { + w.Write([]byte("ok\n")) + }) + mux.HandleFunc("GET /", func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "text/plain") + w.Write(body) + }) + return mux +} + +// newTLSConfig serves the certificate in certFile with the key in keyFile. +func newTLSConfig(certFile, keyFile string) (*tls.Config, error) { + cert, err := tls.LoadX509KeyPair(certFile, keyFile) + if err != nil { + return nil, fmt.Errorf("loading TLS certificate: %w", err) + } + return &tls.Config{Certificates: []tls.Certificate{cert}, MinVersion: tls.VersionTLS12}, nil +} + +// portRange parses "A-B" or "A" into the ports A through B; "" is none. +func portRange(s string) ([]int, error) { + if s == "" { + return nil, nil + } + lo, hi, found := strings.Cut(s, "-") + if !found { + hi = lo + } + a, errA := strconv.Atoi(lo) + b, errB := strconv.Atoi(hi) + if errA != nil || errB != nil || a < 1 || b > 65535 || a > b { + return nil, fmt.Errorf("%q is not a port or a port range A-B", s) + } + ports := make([]int, 0, b-a+1) + for p := a; p <= b; p++ { + ports = append(ports, p) + } + return ports, nil +} + +// openListeners opens addr, when set, and every port in ports on all interfaces. +// On an error it closes what it opened. +func openListeners(addr, ports string) ([]net.Listener, error) { + addrs, err := listenAddrs(addr, ports) + if err != nil { + return nil, err + } + return listenAll(addrs) +} + +// listenAddrs is addr, when set, followed by every port of ports on the +// wildcard address. +func listenAddrs(addr, ports string) ([]string, error) { + ps, err := portRange(ports) + if err != nil { + return nil, err + } + var addrs []string + if addr != "" { + addrs = append(addrs, addr) + } + for _, p := range ps { + addrs = append(addrs, ":"+strconv.Itoa(p)) + } + return addrs, nil +} + +// listenAll listens on every address, or on none. +func listenAll(addrs []string) ([]net.Listener, error) { + lns := make([]net.Listener, 0, len(addrs)) + for _, a := range addrs { + ln, err := net.Listen("tcp", a) + if err != nil { + for _, l := range lns { + l.Close() + } + return nil, fmt.Errorf("listening on %s: %w", a, err) + } + lns = append(lns, ln) + } + return lns, nil +} + +// runTarget serves until ctx ends or a listener fails. +func runTarget(ctx context.Context, o targetOptions) error { + handler := newHandler(o.responseBytes) + plain := &http.Server{Handler: handler, ReadHeaderTimeout: 10 * time.Second} + plainLns, err := openListeners(o.listen, o.listenPorts) + if err != nil { + return fmt.Errorf("opening HTTP listeners: %w", err) + } + servers := []*http.Server{plain} + var tlsSrv *http.Server + var tlsLns []net.Listener + if o.tlsListen != "" || o.tlsListenPorts != "" { + cfg, err := newTLSConfig(o.tlsCertFile, o.tlsKeyFile) + if err != nil { + return err + } + tlsSrv = &http.Server{Handler: handler, TLSConfig: cfg, ReadHeaderTimeout: 10 * time.Second} + if tlsLns, err = openListeners(o.tlsListen, o.tlsListenPorts); err != nil { + return fmt.Errorf("opening HTTPS listeners: %w", err) + } + servers = append(servers, tlsSrv) + } + slog.Info("benchmark-egress target serving", "http_ports", len(plainLns), "https_ports", len(tlsLns)) + + // The first listener to fail stops them all, as the signal does. + failed := make(chan error, len(plainLns)+len(tlsLns)) + var wg sync.WaitGroup + serve := func(ln net.Listener, run func(net.Listener) error) { + wg.Go(func() { + if err := run(ln); !errors.Is(err, http.ErrServerClosed) { + failed <- fmt.Errorf("serving on %s: %w", ln.Addr(), err) + } + }) + } + for _, ln := range plainLns { + serve(ln, plain.Serve) + } + for _, ln := range tlsLns { + serve(ln, func(ln net.Listener) error { return tlsSrv.ServeTLS(ln, "", "") }) + } + + select { + case <-ctx.Done(): + case err = <-failed: + } + shutdownCtx, cancelShutdown := context.WithTimeout(context.Background(), 5*time.Second) + defer cancelShutdown() + for _, srv := range servers { + _ = srv.Shutdown(shutdownCtx) + } + wg.Wait() + return err +} diff --git a/cmd/benchmarking/egress/internal/cmd/target/target_test.go b/cmd/benchmarking/egress/internal/cmd/target/target_test.go new file mode 100644 index 0000000000..a1e44911a7 --- /dev/null +++ b/cmd/benchmarking/egress/internal/cmd/target/target_test.go @@ -0,0 +1,159 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package target + +import ( + "context" + "crypto/tls" + "crypto/x509" + "errors" + "io" + "net" + "net/http" + "net/http/httptest" + "path/filepath" + "slices" + "testing" + "time" + + "go.ate.dev/substrate/cmd/benchmarking/egress/internal/contract" + "go.ate.dev/substrate/cmd/benchmarking/egress/internal/targetcert" +) + +func TestHandler(t *testing.T) { + t.Parallel() + h := newHandler(32) + for _, tc := range []struct { + path string + wantBody int + }{ + {"/", 32}, + {"/anything", 32}, + {"/healthz", len("ok\n")}, + } { + t.Run(tc.path, func(t *testing.T) { + t.Parallel() + rec := httptest.NewRecorder() + h.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, tc.path, nil)) + if rec.Code != http.StatusOK || rec.Body.Len() != tc.wantBody { + t.Errorf("GET %s = %d with %d bytes, want 200 with %d", tc.path, rec.Code, rec.Body.Len(), tc.wantBody) + } + }) + } +} + +func TestHandlerOverTLS(t *testing.T) { + t.Parallel() + bundle, err := targetcert.Generate(time.Now()) + if err != nil { + t.Fatal(err) + } + dir := t.TempDir() + if err := bundle.Write(dir); err != nil { + t.Fatal(err) + } + cfg, err := newTLSConfig(filepath.Join(dir, "tls.crt"), filepath.Join(dir, "tls.key")) + if err != nil { + t.Fatal(err) + } + srv := httptest.NewUnstartedServer(newHandler(32)) + srv.TLS = cfg + srv.StartTLS() + t.Cleanup(srv.Close) + + roots := x509.NewCertPool() + if !roots.AppendCertsFromPEM(bundle.CA) { + t.Fatal("bundle CA does not parse") + } + // Dial the test server whatever the URL names, so the name can be an + // endpoint's while the TLS check runs for real. + client := &http.Client{Transport: &http.Transport{ + TLSClientConfig: &tls.Config{RootCAs: roots}, + DialContext: func(ctx context.Context, network, _ string) (net.Conn, error) { + return (&net.Dialer{}).DialContext(ctx, network, srv.Listener.Addr().String()) + }, + }} + + resp, err := client.Get(contract.EndpointURL(42, contract.SchemeHTTPS)) + if err != nil { + t.Fatalf("GET over TLS as an endpoint name: %v", err) + } + body, _ := io.ReadAll(resp.Body) + resp.Body.Close() + if resp.StatusCode != http.StatusOK || len(body) != 32 { + t.Errorf("GET over TLS = %d with %d bytes, want 200 with 32", resp.StatusCode, len(body)) + } + + _, err = client.Get("https://example.com/") + var hostErr x509.HostnameError + if !errors.As(err, &hostErr) { + t.Errorf("GET over TLS as another name = %v, want a hostname mismatch", err) + } +} + +func TestPortRange(t *testing.T) { + t.Parallel() + for _, tc := range []struct { + in string + want []int + wantErr bool + }{ + {"", nil, false}, + {"8080", []int{8080}, false}, + {"10000-10002", []int{10000, 10001, 10002}, false}, + {"10002-10000", nil, true}, + {"0-3", nil, true}, + {"65535-65536", nil, true}, + {"a-b", nil, true}, + {"10000-", nil, true}, + } { + t.Run(tc.in, func(t *testing.T) { + t.Parallel() + got, err := portRange(tc.in) + if (err != nil) != tc.wantErr || !slices.Equal(got, tc.want) { + t.Errorf("portRange(%q) = %v, %v; want %v, error %v", tc.in, got, err, tc.want, tc.wantErr) + } + }) + } +} + +func TestListenAddrs(t *testing.T) { + t.Parallel() + got, err := listenAddrs("127.0.0.1:0", "10000-10002") + if err != nil { + t.Fatal(err) + } + if want := []string{"127.0.0.1:0", ":10000", ":10001", ":10002"}; !slices.Equal(got, want) { + t.Errorf("listenAddrs() = %v, want %v", got, want) + } + if _, err := listenAddrs("", "a-b"); err == nil { + t.Error("listenAddrs accepted a bad port range") + } +} + +func TestListenAllClosesWhatItOpenedOnFailure(t *testing.T) { + t.Parallel() + held, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { held.Close() }) + if lns, err := listenAll([]string{"127.0.0.1:0", held.Addr().String()}); err == nil { + for _, ln := range lns { + ln.Close() + } + t.Fatal("listenAll bound a port that was taken") + } +} diff --git a/cmd/benchmarking/egress/internal/contract/api.go b/cmd/benchmarking/egress/internal/contract/api.go new file mode 100644 index 0000000000..e304559b87 --- /dev/null +++ b/cmd/benchmarking/egress/internal/contract/api.go @@ -0,0 +1,118 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package contract + +import "fmt" + +// Control routes the actor serves on port 80, the only inbound port of a +// sandbox. The measurements leave the actor another way: as OTel metrics +// pushed over OTLP, so no route serves a distribution. +const ( + ReadyzRoute = "/readyz" + StartRoute = "/start" + // StatsRoute, GET, serves the running loop's Stats: the counters, cheap + // enough to poll, and nothing is consumed by a read. ?endpoints=1 adds + // the per-endpoint counts. + StatsRoute = "/stats" + // StopRoute, POST, ends the loop, flushes the last export and serves the + // final Stats. The actor keeps that final Stats once the loop is gone, so + // a repeated /stop answers it again with 200 until the next /start, while + // /stats answers 404. ?endpoints=1 as on /stats. + StopRoute = "/stop" +) + +// NoLoopMessage is the body of the 404 that /stats and /stop answer while no +// loop runs. +const NoLoopMessage = "no loop is running" + +// ProtocolHeader is on every answer the actor gives, ProtocolVersion its +// value, so a driver can tell what it is talking to before it starts +// anything: an actor whose image speaks a later contract is refused rather +// than misread. +const ( + ProtocolHeader = "Benchmark-Egress-Protocol" + ProtocolVersion = "1" +) + +// DefaultRequestTimeoutMs applies when StartRequest.RequestTimeoutMs is zero. +const DefaultRequestTimeoutMs = 5000 + +// The schemes the loop can request endpoints over. +const ( + SchemeHTTP = "http" + SchemeHTTPS = "https" +) + +// StartRequest is the JSON body of POST /start. +type StartRequest struct { + // Endpoints is C: the loop requests EndpointURL(0) through + // EndpointURL(Endpoints-1), each on its own goroutine, until it is stopped. + Endpoints int `json:"endpoints"` + // RequestTimeoutMs bounds one request; zero means DefaultRequestTimeoutMs. + RequestTimeoutMs int64 `json:"requestTimeoutMs"` + // IntervalMs paces the actor: endpoint i's first request starts OffsetMs + // plus i intervals after the loop, and each endpoint is then requested once + // per period of C intervals, so the actor sends one request per interval + // throughout. Zero sends every endpoint's requests back to back. + IntervalMs int64 `json:"intervalMs"` + // OffsetMs delays the whole schedule: endpoint i's first request starts + // OffsetMs plus i intervals after the loop. A driver gives each actor its + // own value so a fleet started at once does not send in lockstep. + OffsetMs int64 `json:"offsetMs,omitempty"` + // Scheme is SchemeHTTP or SchemeHTTPS; empty means SchemeHTTP. + Scheme string `json:"scheme,omitempty"` + // Actor, Cell and Run name this loop: the actor as /, the + // cell it runs in, and the run the cell belongs to. All three are + // attributes on every metric the loop exports and all three are required. + Actor string `json:"actor"` + Cell string `json:"cell"` + Run string `json:"run"` +} + +// URLScheme is the scheme the loop requests endpoints over. +func (r *StartRequest) URLScheme() string { + if r.Scheme == "" { + return SchemeHTTP + } + return r.Scheme +} + +// Validate reports whether r can start a loop. +func (r *StartRequest) Validate() error { + if r.Endpoints < 1 || r.Endpoints > MaxEndpoints { + return fmt.Errorf("endpoints must be between 1 and %d, got %d", MaxEndpoints, r.Endpoints) + } + if r.RequestTimeoutMs < 0 { + return fmt.Errorf("requestTimeoutMs cannot be negative: %d", r.RequestTimeoutMs) + } + if r.IntervalMs < 0 { + return fmt.Errorf("intervalMs cannot be negative: %d", r.IntervalMs) + } + if r.OffsetMs < 0 { + return fmt.Errorf("offsetMs cannot be negative: %d", r.OffsetMs) + } + if period := int64(r.Endpoints) * r.IntervalMs; r.OffsetMs > period { + return fmt.Errorf("offsetMs cannot exceed one period, endpoints x intervalMs = %d: %d", period, r.OffsetMs) + } + if r.Scheme != "" && r.Scheme != SchemeHTTP && r.Scheme != SchemeHTTPS { + return fmt.Errorf("scheme must be %s or %s, got %q", SchemeHTTP, SchemeHTTPS, r.Scheme) + } + for _, f := range []struct{ name, value string }{{"actor", r.Actor}, {"cell", r.Cell}, {"run", r.Run}} { + if f.value == "" { + return fmt.Errorf("%s is required", f.name) + } + } + return nil +} diff --git a/cmd/benchmarking/egress/internal/contract/api_test.go b/cmd/benchmarking/egress/internal/contract/api_test.go new file mode 100644 index 0000000000..d367d1fad0 --- /dev/null +++ b/cmd/benchmarking/egress/internal/contract/api_test.go @@ -0,0 +1,69 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package contract + +import "testing" + +// named is a StartRequest with the three identity fields filled, so a case +// states only what it is about. +func named(r StartRequest) StartRequest { + r.Actor, r.Cell, r.Run = "benchmark-workloads/egress-0", "cell-1", "run-1" + return r +} + +func TestStartRequestValidate(t *testing.T) { + t.Parallel() + for _, tc := range []struct { + name string + req StartRequest + wantErr bool + }{ + {"ok", named(StartRequest{Endpoints: 10}), false}, + {"http", named(StartRequest{Endpoints: 10, Scheme: SchemeHTTP}), false}, + {"https", named(StartRequest{Endpoints: 10, Scheme: SchemeHTTPS}), false}, + {"bad scheme", named(StartRequest{Endpoints: 10, Scheme: "ftp"}), true}, + {"most endpoints", named(StartRequest{Endpoints: MaxEndpoints}), false}, + {"no endpoints", named(StartRequest{}), true}, + {"too many endpoints", named(StartRequest{Endpoints: MaxEndpoints + 1}), true}, + {"negative timeout", named(StartRequest{Endpoints: 1, RequestTimeoutMs: -1}), true}, + {"negative interval", named(StartRequest{Endpoints: 1, IntervalMs: -1}), true}, + {"offset", named(StartRequest{Endpoints: 3, IntervalMs: 100, OffsetMs: 50}), false}, + {"offset of one period", named(StartRequest{Endpoints: 3, IntervalMs: 100, OffsetMs: 300}), false}, + {"offset over one period", named(StartRequest{Endpoints: 3, IntervalMs: 100, OffsetMs: 301}), true}, + {"negative offset", named(StartRequest{Endpoints: 1, IntervalMs: 100, OffsetMs: -1}), true}, + {"offset without interval", named(StartRequest{Endpoints: 1, OffsetMs: 1}), true}, + {"no identity", StartRequest{Endpoints: 1}, true}, + {"no actor", StartRequest{Endpoints: 1, Cell: "cell-1", Run: "run-1"}, true}, + {"no cell", StartRequest{Endpoints: 1, Actor: "benchmark-workloads/egress-0", Run: "run-1"}, true}, + {"no run", StartRequest{Endpoints: 1, Actor: "benchmark-workloads/egress-0", Cell: "cell-1"}, true}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + if err := tc.req.Validate(); (err != nil) != tc.wantErr { + t.Errorf("Validate() = %v, wantErr %v", err, tc.wantErr) + } + }) + } +} + +func TestURLScheme(t *testing.T) { + t.Parallel() + if got := (&StartRequest{}).URLScheme(); got != SchemeHTTP { + t.Errorf("URLScheme() with no scheme = %q, want %q", got, SchemeHTTP) + } + if got := (&StartRequest{Scheme: SchemeHTTPS}).URLScheme(); got != SchemeHTTPS { + t.Errorf("URLScheme() with https = %q, want %q", got, SchemeHTTPS) + } +} diff --git a/cmd/benchmarking/egress/internal/contract/doc.go b/cmd/benchmarking/egress/internal/contract/doc.go new file mode 100644 index 0000000000..7210d9d3cf --- /dev/null +++ b/cmd/benchmarking/egress/internal/contract/doc.go @@ -0,0 +1,27 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// Package contract is what the programs of the egress scale test agree on: +// the endpoint names the actor's loop calls, the control routes and the +// /start body a driver sends it, and the Stats it answers with, down to the +// bounds of its distributions. +// +// The package imports no SDK, so a driver decodes a report with the standard +// library alone. +// +// The prose form of the contract is the actor's README, +// cmd/benchmarking/egress/internal/cmd/actor/README.md; the overview, +// cmd/benchmarking/egress/README.md, has what the benchmark measures and the +// topology the actor runs in. +package contract diff --git a/cmd/benchmarking/egress/internal/contract/endpoints.go b/cmd/benchmarking/egress/internal/contract/endpoints.go new file mode 100644 index 0000000000..e642c08eb0 --- /dev/null +++ b/cmd/benchmarking/egress/internal/contract/endpoints.go @@ -0,0 +1,49 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package contract + +import "fmt" + +// TargetNamespace holds the endpoint Services the benchmark deploys. +const TargetNamespace = "egress-tests-targets" + +// MaxEndpoints is the most endpoints a run can use. +const MaxEndpoints = 1024 + +// ServiceName is the name of endpoint i's Service. +func ServiceName(i int) string { + return fmt.Sprintf("egress-target-%d", i) +} + +// EndpointDomain is the DNS domain every endpoint name sits directly under. +const EndpointDomain = TargetNamespace + ".svc.cluster.local" + +// EndpointHostPattern matches every endpoint name and nothing below them: a +// wildcard stands for exactly one leftmost label. The target's certificate is +// issued for it. +const EndpointHostPattern = "*." + EndpointDomain + +// EndpointHost is the DNS name of endpoint i. It is the one place the +// endpoint names are defined: the actor's loop calls these names and the +// driver's egress policy allows them. +func EndpointHost(i int) string { + return ServiceName(i) + "." + EndpointDomain +} + +// EndpointURL is the URL the actor's loop requests for endpoint i over scheme, +// http or https, each on its default port. +func EndpointURL(i int, scheme string) string { + return scheme + "://" + EndpointHost(i) + "/" +} diff --git a/cmd/benchmarking/egress/internal/contract/endpoints_test.go b/cmd/benchmarking/egress/internal/contract/endpoints_test.go new file mode 100644 index 0000000000..d0d1693fe4 --- /dev/null +++ b/cmd/benchmarking/egress/internal/contract/endpoints_test.go @@ -0,0 +1,36 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package contract + +import "testing" + +func TestEndpointNames(t *testing.T) { + t.Parallel() + if got, want := ServiceName(7), "egress-target-7"; got != want { + t.Errorf("ServiceName(7) = %q, want %q", got, want) + } + if got, want := EndpointHost(7), "egress-target-7.egress-tests-targets.svc.cluster.local"; got != want { + t.Errorf("EndpointHost(7) = %q, want %q", got, want) + } + if got, want := EndpointURL(7, SchemeHTTP), "http://egress-target-7.egress-tests-targets.svc.cluster.local/"; got != want { + t.Errorf("EndpointURL(7, http) = %q, want %q", got, want) + } + if got, want := EndpointURL(7, SchemeHTTPS), "https://egress-target-7.egress-tests-targets.svc.cluster.local/"; got != want { + t.Errorf("EndpointURL(7, https) = %q, want %q", got, want) + } + if got, want := EndpointHostPattern, "*.egress-tests-targets.svc.cluster.local"; got != want { + t.Errorf("EndpointHostPattern = %q, want %q", got, want) + } +} diff --git a/cmd/benchmarking/egress/internal/contract/hist.go b/cmd/benchmarking/egress/internal/contract/hist.go new file mode 100644 index 0000000000..113f767ef7 --- /dev/null +++ b/cmd/benchmarking/egress/internal/contract/hist.go @@ -0,0 +1,66 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package contract + +import ( + "fmt" + "strconv" +) + +const ( + bucketsPerDecade = 20 + // maxBucketSeconds stops the bounds: the first bound at or past it is the + // last. Slower samples count in the overflow bucket. + maxBucketSeconds = 60 +) + +// mantissas are 10^(j/20) for j = 0..19, to four significant digits. +var mantissas = [bucketsPerDecade]string{ + "1", "1.122", "1.259", "1.413", "1.585", "1.778", "1.995", "2.239", "2.512", "2.818", + "3.162", "3.548", "3.981", "4.467", "5.012", "5.623", "6.31", "7.079", "7.943", "8.913", +} + +// Bounds are the upper bounds, in seconds, of every distribution: 20 per +// decade from 1 µs to 63.1 s. Each is parsed from a decimal string, so every +// build on every architecture holds the same 157 values, and a sum by bucket +// across the fleet is meaningful. Neighboring bounds differ by about 12%, the +// resolution of a quantile read from buckets. +var Bounds = func() []float64 { + var bounds []float64 + for k := 0; ; k++ { + b, err := strconv.ParseFloat(fmt.Sprintf("%se%d", mantissas[k%bucketsPerDecade], k/bucketsPerDecade-6), 64) + if err != nil { + panic(err) + } + bounds = append(bounds, b) + if b >= maxBucketSeconds { + return bounds + } + } +}() + +// Distribution names one of the four latency histograms the actor exports, +// the instrument name on the wire. +type Distribution string + +const ( + Opening Distribution = "opening" // end-to-end time of an endpoint's first request, when it succeeded + Latency Distribution = "latency" // end-to-end time of later successful requests + DNS Distribution = "dns" // lookup time, for requests that did a lookup + TLS Distribution = "tls" // handshake time, for handshakes that completed +) + +// DistributionNames lists the four distributions. +var DistributionNames = []Distribution{Opening, Latency, DNS, TLS} diff --git a/cmd/benchmarking/egress/internal/contract/hist_test.go b/cmd/benchmarking/egress/internal/contract/hist_test.go new file mode 100644 index 0000000000..19a76e5036 --- /dev/null +++ b/cmd/benchmarking/egress/internal/contract/hist_test.go @@ -0,0 +1,38 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package contract + +import ( + "strconv" + "testing" +) + +func TestBounds(t *testing.T) { + t.Parallel() + if n := len(Bounds); n != 157 { + t.Fatalf("len(Bounds) = %d, want 157", n) + } + if Bounds[0] != 1e-6 || Bounds[60] != 0.001 || Bounds[156] != 63.1 { + t.Errorf("Bounds[0,60,156] = %v %v %v, want 1e-06 0.001 63.1", Bounds[0], Bounds[60], Bounds[156]) + } + for i, b := range Bounds { + if i > 0 && b <= Bounds[i-1] { + t.Errorf("Bounds[%d] = %v does not exceed Bounds[%d] = %v", i, b, i-1, Bounds[i-1]) + } + if again, _ := strconv.ParseFloat(strconv.FormatFloat(b, 'g', 4, 64), 64); again != b { + t.Errorf("Bounds[%d] = %v has more than four significant digits", i, b) + } + } +} diff --git a/cmd/benchmarking/egress/internal/contract/stats.go b/cmd/benchmarking/egress/internal/contract/stats.go new file mode 100644 index 0000000000..ed9b0ed13e --- /dev/null +++ b/cmd/benchmarking/egress/internal/contract/stats.go @@ -0,0 +1,69 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package contract + +import "time" + +// Stats is one actor's loop counters, cumulative since /start, as GET /stats +// and POST /stop serve them. The same counts and the four latency +// distributions leave the actor as OTel metrics, not in this payload. +type Stats struct { + // Scheme is the scheme the loop requested endpoints over. + Scheme string `json:"scheme,omitempty"` + Elapsed time.Duration `json:"elapsed"` + // Requests counts every request the loop finished, failed ones included. + Requests int64 `json:"requests"` + Successes int64 `json:"successes"` + // NewConns counts requests that went out on a connection opened for them + // rather than on a kept-alive one. A tunnel the gateway reset shows up + // here rather than in Errors: net/http redials, and retries a GET that + // failed before any of the response arrived. + NewConns int64 `json:"newConns"` + // Opened counts endpoints whose first request has completed, so a reader + // can tell the opening from the steady state. + Opened int64 `json:"opened"` + // OpeningDone is Elapsed at the moment the last endpoint's first request + // completed; zero until every endpoint has opened. + OpeningDone time.Duration `json:"openingDone"` + // RequestsAtOpeningDone is Requests at that same moment, so a reader can + // price this actor's opening alone; zero until every endpoint has opened. + RequestsAtOpeningDone int64 `json:"requestsAtOpeningDone"` + // Rounds is the number of requests the least-visited endpoint has + // completed: whether this actor is still making progress. + Rounds int64 `json:"rounds"` + // LongestGap is the longest time between the starts of two consecutive + // requests on one endpoint: how quiet the loop has left a tunnel. Each + // endpoint is requested once per period, C times the interval, so with + // prompt endpoints every gap is one period; a request that outlasts the + // period delays only that endpoint's next request, so a longer gap means a + // request on that tunnel took about that long. Keep the period under the + // gateway's stream idle timeout. + LongestGap time.Duration `json:"longestGap"` + // Errors counts failed requests by class, as the actor names them: one + // low-cardinality class per failure mode. + Errors map[string]int64 `json:"errors,omitempty"` + // Endpoints holds per-endpoint counts, indexed by endpoint number. It is + // served only when asked for with ?endpoints=1. + Endpoints []Endpoint `json:"endpoints,omitempty"` +} + +// Endpoint holds the counts for one endpoint. +type Endpoint struct { + Requests int64 `json:"requests"` + Errors int64 `json:"errors"` + // LongestGap is Stats.LongestGap for this endpoint alone, so a quiet + // tunnel can be named. + LongestGap time.Duration `json:"longestGap"` +} diff --git a/cmd/benchmarking/egress/internal/contract/stats_test.go b/cmd/benchmarking/egress/internal/contract/stats_test.go new file mode 100644 index 0000000000..0ab7bfb3f2 --- /dev/null +++ b/cmd/benchmarking/egress/internal/contract/stats_test.go @@ -0,0 +1,88 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package contract + +import ( + "encoding/json" + "reflect" + "strings" + "testing" + "time" +) + +// TestStatsJSONRoundTrip pins the JSON a /stats or /stop answers with, as a +// driver decodes it: every counter, durations in nanoseconds, the errors by +// class and the per-endpoint rows, and nothing optional when there is nothing +// to report. +func TestStatsJSONRoundTrip(t *testing.T) { + t.Parallel() + for _, tc := range []struct { + name string + in Stats + present []string + absent []string + }{ + { + name: "every field", + in: Stats{ + Scheme: SchemeHTTPS, + Elapsed: 61200 * time.Millisecond, + Requests: 60, + Successes: 55, + NewConns: 4, + Opened: 4, + OpeningDone: 350 * time.Millisecond, + RequestsAtOpeningDone: 4, + Rounds: 15, + LongestGap: 400 * time.Millisecond, + Errors: map[string]int64{"HTTP 503": 3, "timeout": 2}, + Endpoints: []Endpoint{{Requests: 15, Errors: 1, LongestGap: 400 * time.Millisecond}, {Requests: 15}, {Requests: 15, Errors: 4}, {Requests: 15}}, + }, + present: []string{`"scheme":"https"`, `"elapsed":61200000000`, `"requests":60`, `"successes":55`, `"newConns":4`, `"opened":4`, + `"openingDone":350000000`, `"requestsAtOpeningDone":4`, `"rounds":15`, `"longestGap":400000000`, `"errors":{"HTTP 503":3,"timeout":2}`, `"endpoints":[{"requests":15,"errors":1,"longestGap":400000000}`}, + }, + { + name: "counters only", + in: Stats{Scheme: SchemeHTTP, Requests: 3}, + present: []string{`"scheme":"http"`, `"requests":3`}, + absent: []string{"endpoints", "errors"}, + }, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + b, err := json.Marshal(tc.in) + if err != nil { + t.Fatal(err) + } + for _, want := range tc.present { + if !strings.Contains(string(b), want) { + t.Errorf("Stats JSON lacks %s: %s", want, b) + } + } + for _, absent := range tc.absent { + if strings.Contains(string(b), absent) { + t.Errorf("Stats JSON carries %q: %s", absent, b) + } + } + var back Stats + if err := json.Unmarshal(b, &back); err != nil { + t.Fatal(err) + } + if !reflect.DeepEqual(back, tc.in) { + t.Errorf("round trip = %+v, want %+v", back, tc.in) + } + }) + } +} diff --git a/cmd/benchmarking/egress/internal/targetcert/targetcert.go b/cmd/benchmarking/egress/internal/targetcert/targetcert.go new file mode 100644 index 0000000000..51812fd6db --- /dev/null +++ b/cmd/benchmarking/egress/internal/targetcert/targetcert.go @@ -0,0 +1,159 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// Package targetcert issues the name-constrained CA and the certificate the +// target serves over HTTPS, for the driver to deploy and the actor to verify +// against. +package targetcert + +import ( + "crypto/ecdsa" + "crypto/elliptic" + "crypto/rand" + "crypto/x509" + "crypto/x509/pkix" + "encoding/pem" + "fmt" + "math/big" + "os" + "path/filepath" + "time" + + "go.ate.dev/substrate/cmd/benchmarking/egress/internal/contract" +) + +// validity is long enough that a test cluster never sees the certificate +// expire; there is no rotation, since nothing can sign a replacement. +const validity = 5 * 365 * 24 * time.Hour + +// clockSkew backdates NotBefore so a node whose clock runs slightly behind +// still accepts a certificate issued moments ago. +const clockSkew = time.Hour + +// Bundle is a CA certificate and the target certificate and key it signed, +// all PEM-encoded. +type Bundle struct { + CA, Cert, Key []byte +} + +// Generate issues a new CA and the certificate the target serves +// over HTTPS, valid from now. The egress gateway re-originates TLS to the +// target and has to trust the CA, and the gateway is shared, so the CA vouches +// for nothing but the endpoint names: it is name-constrained to their domain, +// and its private key is thrown away once the target's certificate is signed. +func Generate(now time.Time) (*Bundle, error) { + b, _, _, err := generateWithCA(now) + return b, err +} + +// generateWithCA is Generate, also returning the CA certificate and key so tests +// can show what the CA refuses to vouch for. +func generateWithCA(now time.Time) (*Bundle, *x509.Certificate, *ecdsa.PrivateKey, error) { + caKey, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + if err != nil { + return nil, nil, nil, fmt.Errorf("generating CA key: %w", err) + } + caSerial, err := serialNumber() + if err != nil { + return nil, nil, nil, err + } + caTemplate := &x509.Certificate{ + SerialNumber: caSerial, + Subject: pkix.Name{CommonName: "egress-tests target CA"}, + NotBefore: now.Add(-clockSkew), + NotAfter: now.Add(validity), + KeyUsage: x509.KeyUsageCertSign | x509.KeyUsageCRLSign, + BasicConstraintsValid: true, + IsCA: true, + MaxPathLenZero: true, + // Critical, so a verifier that cannot enforce the constraint rejects + // the CA instead of trusting it for every name. + PermittedDNSDomainsCritical: true, + PermittedDNSDomains: []string{contract.EndpointDomain}, + } + caDER, err := x509.CreateCertificate(rand.Reader, caTemplate, caTemplate, &caKey.PublicKey, caKey) + if err != nil { + return nil, nil, nil, fmt.Errorf("creating CA certificate: %w", err) + } + ca, err := x509.ParseCertificate(caDER) + if err != nil { + return nil, nil, nil, fmt.Errorf("parsing CA certificate: %w", err) + } + + leafKey, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + if err != nil { + return nil, nil, nil, fmt.Errorf("generating target key: %w", err) + } + leafSerial, err := serialNumber() + if err != nil { + return nil, nil, nil, err + } + leafTemplate := &x509.Certificate{ + SerialNumber: leafSerial, + Subject: pkix.Name{CommonName: "egress-tests target"}, + // One wildcard covers every endpoint name, which differ only in the + // leftmost label. + DNSNames: []string{contract.EndpointHostPattern}, + NotBefore: now.Add(-clockSkew), + NotAfter: now.Add(validity), + KeyUsage: x509.KeyUsageDigitalSignature, + ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth}, + } + leafDER, err := x509.CreateCertificate(rand.Reader, leafTemplate, ca, &leafKey.PublicKey, caKey) + if err != nil { + return nil, nil, nil, fmt.Errorf("creating target certificate: %w", err) + } + keyDER, err := x509.MarshalPKCS8PrivateKey(leafKey) + if err != nil { + return nil, nil, nil, fmt.Errorf("encoding target key: %w", err) + } + + b := &Bundle{ + CA: pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: caDER}), + Cert: pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: leafDER}), + Key: pem.EncodeToMemory(&pem.Block{Type: "PRIVATE KEY", Bytes: keyDER}), + } + return b, ca, caKey, nil +} + +// Write stores the bundle in dir as ca.crt, tls.crt and tls.key; only the key +// is private. The files are what the deploy script turns into the target's +// TLS Secret, named as its keys. +func (b *Bundle) Write(dir string) error { + if err := os.MkdirAll(dir, 0o700); err != nil { + return err + } + for _, f := range []struct { + name string + data []byte + mode os.FileMode + }{ + {"ca.crt", b.CA, 0o644}, + {"tls.crt", b.Cert, 0o644}, + {"tls.key", b.Key, 0o600}, + } { + if err := os.WriteFile(filepath.Join(dir, f.name), f.data, f.mode); err != nil { + return err + } + } + return nil +} + +func serialNumber() (*big.Int, error) { + n, err := rand.Int(rand.Reader, new(big.Int).Lsh(big.NewInt(1), 128)) + if err != nil { + return nil, fmt.Errorf("generating serial number: %w", err) + } + return n, nil +} diff --git a/cmd/benchmarking/egress/internal/targetcert/targetcert_test.go b/cmd/benchmarking/egress/internal/targetcert/targetcert_test.go new file mode 100644 index 0000000000..c48ad7b132 --- /dev/null +++ b/cmd/benchmarking/egress/internal/targetcert/targetcert_test.go @@ -0,0 +1,186 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package targetcert + +import ( + "crypto/ecdsa" + "crypto/elliptic" + "crypto/rand" + "crypto/tls" + "crypto/x509" + "crypto/x509/pkix" + "encoding/pem" + "errors" + "math/big" + "os" + "path/filepath" + "testing" + "time" + + "go.ate.dev/substrate/cmd/benchmarking/egress/internal/contract" +) + +func parseCert(t *testing.T, data []byte) *x509.Certificate { + t.Helper() + block, _ := pem.Decode(data) + if block == nil || block.Type != "CERTIFICATE" { + t.Fatalf("not a PEM certificate: %q", data) + } + cert, err := x509.ParseCertificate(block.Bytes) + if err != nil { + t.Fatal(err) + } + return cert +} + +func verify(t *testing.T, b *Bundle, name string, at time.Time) error { + t.Helper() + roots := x509.NewCertPool() + roots.AddCert(parseCert(t, b.CA)) + _, err := parseCert(t, b.Cert).Verify(x509.VerifyOptions{ + DNSName: name, + Roots: roots, + CurrentTime: at, + KeyUsages: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth}, + }) + return err +} + +func TestCertificateCoversEveryEndpoint(t *testing.T) { + t.Parallel() + now := time.Now() + b, err := Generate(now) + if err != nil { + t.Fatal(err) + } + for i := range contract.MaxEndpoints { + if err := verify(t, b, contract.EndpointHost(i), now); err != nil { + t.Fatalf("certificate does not verify for %s: %v", contract.EndpointHost(i), err) + } + } +} + +func TestCertificateRejectsOtherNames(t *testing.T) { + t.Parallel() + now := time.Now() + b, err := Generate(now) + if err != nil { + t.Fatal(err) + } + for _, name := range []string{ + "a.egress-target-0." + contract.EndpointDomain, // a wildcard covers one label only + "egress-target-0.other-namespace.svc.cluster.local", + "example.com", + } { + t.Run(name, func(t *testing.T) { + t.Parallel() + var hostErr x509.HostnameError + if err := verify(t, b, name, now); !errors.As(err, &hostErr) { + t.Errorf("verifying for %s = %v, want a HostnameError", name, err) + } + }) + } +} + +func TestCertificateValidityWindow(t *testing.T) { + t.Parallel() + now := time.Now() + b, err := Generate(now) + if err != nil { + t.Fatal(err) + } + name := contract.EndpointHost(0) + if err := verify(t, b, name, now.Add(-30*time.Minute)); err != nil { + t.Errorf("certificate rejected half an hour before issue, inside the clock-skew allowance: %v", err) + } + for _, tc := range []struct { + name string + at time.Time + }{ + {"before the skew allowance", now.Add(-2 * time.Hour)}, + {"after expiry", now.Add(validity + time.Hour)}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + var invalid x509.CertificateInvalidError + if err := verify(t, b, name, tc.at); !errors.As(err, &invalid) || invalid.Reason != x509.Expired { + t.Errorf("verifying at %v = %v, want it outside the validity window", tc.at, err) + } + }) + } +} + +// The gateway trusts this CA for every upstream it dials, so the CA must be +// unable to vouch for any name outside the endpoints' domain, even in a +// certificate it signed itself. +func TestCARefusesNamesOutsideTheEndpointDomain(t *testing.T) { + t.Parallel() + now := time.Now() + _, ca, caKey, err := generateWithCA(now) + if err != nil { + t.Fatal(err) + } + key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + if err != nil { + t.Fatal(err) + } + der, err := x509.CreateCertificate(rand.Reader, &x509.Certificate{ + SerialNumber: big.NewInt(1), + Subject: pkix.Name{CommonName: "evil"}, + DNSNames: []string{"evil.example.com"}, + NotBefore: now.Add(-time.Hour), + NotAfter: now.Add(time.Hour), + ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth}, + }, ca, &key.PublicKey, caKey) + if err != nil { + t.Fatal(err) + } + evil, err := x509.ParseCertificate(der) + if err != nil { + t.Fatal(err) + } + roots := x509.NewCertPool() + roots.AddCert(ca) + _, err = evil.Verify(x509.VerifyOptions{DNSName: "evil.example.com", Roots: roots, CurrentTime: now}) + var invalid x509.CertificateInvalidError + if !errors.As(err, &invalid) || invalid.Reason != x509.CANotAuthorizedForThisName { + t.Errorf("a certificate for evil.example.com signed by the CA verified with %v, want CANotAuthorizedForThisName", err) + } +} + +func TestWrite(t *testing.T) { + t.Parallel() + b, err := Generate(time.Now()) + if err != nil { + t.Fatal(err) + } + dir := filepath.Join(t.TempDir(), "certs") + if err := b.Write(dir); err != nil { + t.Fatal(err) + } + if _, err := tls.LoadX509KeyPair(filepath.Join(dir, "tls.crt"), filepath.Join(dir, "tls.key")); err != nil { + t.Errorf("written certificate and key do not load as a pair: %v", err) + } + info, err := os.Stat(filepath.Join(dir, "tls.key")) + if err != nil { + t.Fatal(err) + } + if mode := info.Mode().Perm(); mode != 0o600 { + t.Errorf("key file mode = %o, want 600", mode) + } + if ca, err := os.ReadFile(filepath.Join(dir, "ca.crt")); err != nil || string(ca) != string(b.CA) { + t.Errorf("CA file = %q, %v; want the bundle's CA", ca, err) + } +} diff --git a/cmd/benchmarking/egress/main.go b/cmd/benchmarking/egress/main.go new file mode 100644 index 0000000000..fa3f068bb6 --- /dev/null +++ b/cmd/benchmarking/egress/main.go @@ -0,0 +1,56 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// benchmark-egress is the workload of the egress scale test, which holds many +// CONNECT tunnels open through the egress gateway at a fixed pace and measures +// what they cost. Each of its programs is a subcommand: the actor that opens +// the tunnels and counts what happened, and the target they call. +package main + +import ( + "context" + "fmt" + "os" + "os/signal" + "syscall" + + "github.com/spf13/cobra" + + "go.ate.dev/substrate/cmd/benchmarking/egress/internal/cmd/actor" + "go.ate.dev/substrate/cmd/benchmarking/egress/internal/cmd/target" +) + +func main() { + root := &cobra.Command{ + Use: "benchmark-egress", + Short: "The actor and target of the egress scale test", + Long: `benchmark-egress is the workload of the egress scale test. The actor holds +one tunnel per endpoint open through the egress gateway and counts what +happened; the target is the endpoint every tunnel leads to.`, + // No usage text on an error; the error alone names what went wrong. + SilenceUsage: true, + // main prints the error itself, once. + SilenceErrors: true, + // Nobody types these commands often enough to want shell completion. + CompletionOptions: cobra.CompletionOptions{DisableDefaultCmd: true}, + } + root.AddCommand(actor.Command(), target.Command()) + + ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM) + defer stop() + if err := root.ExecuteContext(ctx); err != nil { + fmt.Fprintln(os.Stderr, "benchmark-egress:", err) + os.Exit(1) + } +}