From 42d1cd760fbb066748d6f30050b3d3b3f734cdd2 Mon Sep 17 00:00:00 2001 From: Yuan Gao Date: Tue, 20 Oct 2026 10:30:44 -0700 Subject: [PATCH 1/6] benchmark-egress: the egress scale test's workload binary The egress scale test holds many CONNECT tunnels open through the egress gateway at a fixed pace and measures what they cost. Its programs are subcommands of one binary, so there is one image to build and ship. This is the Cobra root; the subcommands follow. The README opens with the whole picture: what the benchmark measures and the topology every later piece fits into. Each subcommand's commit adds its own section. --- cmd/benchmarking/egress/README.md | 100 ++++++++++++++++++++++++++++++ cmd/benchmarking/egress/main.go | 51 +++++++++++++++ 2 files changed, 151 insertions(+) create mode 100644 cmd/benchmarking/egress/README.md create mode 100644 cmd/benchmarking/egress/main.go diff --git a/cmd/benchmarking/egress/README.md b/cmd/benchmarking/egress/README.md new file mode 100644 index 0000000000..11e4cff9e4 --- /dev/null +++ b/cmd/benchmarking/egress/README.md @@ -0,0 +1,100 @@ +# benchmark-egress + +`benchmark-egress` is the workload of the egress scale test. It holds many +CONNECT tunnels open through the egress gateway at a fixed pace and reports what +each tunnel saw, so that a run can price what one tunnel costs the gateway. + +## 1. What the benchmark measures + +The benchmark sizes one `atenet-egress` replica for a target fleet: A actors, +each reaching C destinations. It reports what one tunnel and one request cost +the gateway, so that the replica count for a fleet follows from fleet size, +node shape, and these costs. + +Three terms: + +- **Tunnel.** One kept-alive CONNECT from one actor to one destination + Service, opened by the actor's first request and reused by every later one. + It is the unit the gateway's cost scales with: a fleet of A actors over C + Services holds A × C tunnels. +- **Blended.** The window from the first actor's start until the last actor + has opened its last tunnel. New tunnels are being set up while tunnels + already open carry repeat requests, so the gateway's CPU mixes setup with + ordinary requests. +- **Settled.** The period, after the blended window, in which every tunnel + has been reused and every request reuses one. Nothing is set up; the + gateway's cost is per request. + +The settled period gives the cost of one request directly. The blended window +gives the cost of setting up one tunnel once its requests are priced at the +settled rate. + +A cell is one fleet size and one protocol. It reports: + +- **Settled req/s per core.** Throughput per gateway core once every tunnel + exists. Its inverse is the CPU cost of one request. +- **Gateway CPU per tunnel opened.** CPU milliseconds to set up one tunnel, the + cost of its requests taken out. +- **Envoy heap per tunnel.** KiB of heap one held tunnel adds. +- **Settled p50, p99, p99.9 and success rate.** Latency of repeat requests as + the actor sees it, and the share that got HTTP 200. Flat across fleet sizes + means the gateway was not saturated while the costs were read. + +The costs price held tunnels, not tunnels that reconnect. Section 5 lists the +checks a cell must pass before its numbers count. + +Per-core and per-tunnel figures do not depend on the node size or the replica +count. Memory sizes as tunnel count times heap per tunnel, against one node's +memory. CPU sizes as request rate times CPU per request, plus the rate of +tunnel opens times CPU per tunnel opened, against one node's cores. The +replica count for a fleet is the larger of the two. + +## 2. Topology + +```mermaid +flowchart LR + D[driver] --> A[actor] + A --> G[egress gateway] + G --> S0[Service 0] + G --> S1[Service 1] + G --> S2[Service 2] + S0 --> T[target Deployment] + S1 --> T + S2 --> T + G --> M[collector] + M --> P[Prometheus] + D --> P +``` + +- **`benchmark-egress actor`** runs inside each Actor. The driver controls it + through the atenet router on port 80: `/readyz`, `/start`, `/stats`, + `/stop`. Once started, it holds one CONNECT tunnel per endpoint through the + gateway and sends a `GET` on each at a fixed pace, and it exports what it + measures as OpenTelemetry metrics over OTLP, through the same gateway, to the + collector named by its environment. The actor's egress policy must name every + endpoint hostname and the collector. Section 3 has the workflow, section 4 + the metrics. +- **Egress gateway.** One `atenet-egress` replica in its sdsmint deployment, + the system under test. It carries one CONNECT tunnel per actor and + destination, plus one per actor to the collector, terminates the actor's TLS + on HTTPS runs, and checks the egress policy on every request. On HTTPS runs + the actor trusts the gateway's MITM CA through `SSL_CERT_FILE`, and the + gateway re-originates TLS to the target and must trust the target CA. +- **Endpoints.** C ClusterIP Services, `egress-target-0` to + `egress-target-`, in namespace `egress-tests-targets`. Each is a + distinct hostname, so each is its own tunnel. +- **`benchmark-egress target`** runs as one Deployment behind all C Services. + It answers every `GET /` with a 200 and a small fixed body, over plain HTTP + and, given a certificate, HTTPS. Each Service maps to its own backend port, + set with `--listen-ports` and `--tls-listen-ports`. The driver issues the + HTTPS certificate for `*.egress-tests-targets.svc.cluster.local` from a + throwaway CA that is name-constrained to + `egress-tests-targets.svc.cluster.local`. +- **Collector and Prometheus.** An OpenTelemetry collector that receives the + actors' OTLP on port 4317 and serves it as a Prometheus exposition, and a + Prometheus that scrapes it and the gateway node's cAdvisor. Both belong to + the benchmark's deployment, not to the actor. The actor only needs the + collector's address. +- **Driver.** Sets up the run, starts the A actors, posts `/start`, polls + `/stats` for the gates, records the instants that bound each window, and + reads every result from Prometheus with the queries of section 5. diff --git a/cmd/benchmarking/egress/main.go b/cmd/benchmarking/egress/main.go new file mode 100644 index 0000000000..34bbe057da --- /dev/null +++ b/cmd/benchmarking/egress/main.go @@ -0,0 +1,51 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// benchmark-egress is the workload of the egress scale test, which holds many +// CONNECT tunnels open through the egress gateway at a fixed pace and measures +// what they cost. Each of its programs is a subcommand: the actor that opens +// the tunnels and counts what happened, and the target they call. +package main + +import ( + "context" + "fmt" + "os" + "os/signal" + "syscall" + + "github.com/spf13/cobra" +) + +func main() { + root := &cobra.Command{ + Use: "benchmark-egress", + Short: "The actor and target of the egress scale test", + Long: `benchmark-egress is the workload of the egress scale test. The actor holds +one tunnel per endpoint open through the egress gateway and counts what +happened; the target is the endpoint every tunnel leads to.`, + // No usage text on an error; the error alone names what went wrong. + SilenceUsage: true, + // main prints the error itself, once. + SilenceErrors: true, + // Nobody types these commands often enough to want shell completion. + CompletionOptions: cobra.CompletionOptions{DisableDefaultCmd: true}, + } + ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM) + defer stop() + if err := root.ExecuteContext(ctx); err != nil { + fmt.Fprintln(os.Stderr, "benchmark-egress:", err) + os.Exit(1) + } +} From f2fc317d5e61e5a81f41969350b620f334838d3e Mon Sep 17 00:00:00 2001 From: Yuan Gao Date: Thu, 22 Oct 2026 10:15:36 -0700 Subject: [PATCH 2/6] benchmark-egress: the actor's contract The contract package is what the actor, the target and the driver agree on: the routes and the /start body, the endpoint names the loop calls and the egress policy allows, and the Stats the actor answers with. The Stats is JSON: plain counters alone, cumulative since /start and cheap enough to poll. The four latency distributions, opening, latency, dns and tls, are not in the payload; the package names them and holds their bounds, 20 per decade from 1 us to 63.1 s out of a mantissa table, so every image buckets the same way and a sum by bucket across the fleet is meaningful. The package imports no SDK, so a driver decodes a report with the standard library alone. The README gains the routes, the /start fields and the Stats fields. --- cmd/benchmarking/egress/README.md | 71 +++++++++++ .../egress/internal/contract/api.go | 111 ++++++++++++++++++ .../egress/internal/contract/api_test.go | 58 +++++++++ .../egress/internal/contract/endpoints.go | 49 ++++++++ .../internal/contract/endpoints_test.go | 36 ++++++ .../egress/internal/contract/hist.go | 66 +++++++++++ .../egress/internal/contract/hist_test.go | 38 ++++++ .../egress/internal/contract/stats.go | 69 +++++++++++ .../egress/internal/contract/stats_test.go | 88 ++++++++++++++ 9 files changed, 586 insertions(+) create mode 100644 cmd/benchmarking/egress/internal/contract/api.go create mode 100644 cmd/benchmarking/egress/internal/contract/api_test.go create mode 100644 cmd/benchmarking/egress/internal/contract/endpoints.go create mode 100644 cmd/benchmarking/egress/internal/contract/endpoints_test.go create mode 100644 cmd/benchmarking/egress/internal/contract/hist.go create mode 100644 cmd/benchmarking/egress/internal/contract/hist_test.go create mode 100644 cmd/benchmarking/egress/internal/contract/stats.go create mode 100644 cmd/benchmarking/egress/internal/contract/stats_test.go diff --git a/cmd/benchmarking/egress/README.md b/cmd/benchmarking/egress/README.md index 11e4cff9e4..c6c7f6872f 100644 --- a/cmd/benchmarking/egress/README.md +++ b/cmd/benchmarking/egress/README.md @@ -98,3 +98,74 @@ flowchart LR - **Driver.** Sets up the run, starts the A actors, posts `/start`, polls `/stats` for the gates, records the instants that bound each window, and reads every result from Prometheus with the queries of section 5. + +## 3. What the actor does + +The actor is idle or running. It boots idle and sends nothing until `/start`: +every actor restores from a golden snapshot, and a loop started at boot would +have run in the golden actor, which has no egress policy. A restarted actor is +idle again with nothing kept. + +| Route | Idle | Running | +|---|---|---| +| `GET /readyz` | 200 | 200 | +| `POST /start` | 204 | 409 | +| `GET /stats` | 404 | 200, section 4 | +| `POST /stop` | 200 with the last loop's final counters, 404 if none | 200 with the final counters once every goroutine has exited | + +Every answer carries `Benchmark-Egress-Protocol: 1`; the driver requires that +value, and the header is what lets a later contract be refused. + +`POST /start` takes one JSON object, at most 1 MiB; unknown fields are ignored. + +| Field | Meaning | 400 when | +|---|---|---| +| `endpoints` | C, the number of endpoints. | Missing, under 1, or over 1024. | +| `intervalMs` | Pacing. Endpoint *i* opens *i* intervals after `/start`; each endpoint is then requested once per period of C × `intervalMs`. Zero sends every endpoint's requests back to back. | Negative. | +| `offsetMs` | Delay before endpoint 0 opens; endpoint *i* opens `offsetMs` + *i* × `intervalMs` after `/start`. Lets a driver give each actor its own phase, so a fleet started at once does not send in lockstep; see below for how a driver picks it. | Negative, or more than C × `intervalMs`. | +| `scheme` | `http` or `https`; empty means `http`. | Any other value. | +| `requestTimeoutMs` | Bound on one request, handshake included; zero means 5000. | Negative. | + +```json +{"endpoints": 1000, "intervalMs": 100, "scheme": "https"} +``` + +204 means the goroutines exist, not that any request has been sent. A 400 +carries a one-line reason. `/start` does not probe the endpoints: a missing +egress policy or Service shows up as errors in `/stats`, not as a failed +`/start`. The instruments are created at `/start` and released at `/stop`, so +every counter counts from `/start` and a later `/start` begins from zero. + +## 4. What the actor reports + +### 4.1 `GET /stats` and `POST /stop` + +Durations are Go `time.Duration` values, nanoseconds. The per-endpoint list +comes only with `?endpoints=1`, on either route. + +| Field | What it holds | Used for | +|---|---|---| +| `scheme` | `http` or `https`. | The scheme in use. | +| `elapsed` | Time since `/start`. | Rates between two polls. | +| `requests`, `successes` | Requests finished; HTTP 200 with no error is a success. | Progress; a `requests` that went down means a restart. | +| `newConns` | Requests sent on a freshly dialed connection. | One per endpoint is the tunnel. Any more is a reconnect after the gateway cut a tunnel or a request on it timed out. | +| `opened` | Endpoints whose first request has completed. | Progress of this actor's opens; the gate on C. | +| `openingDone` | Elapsed when the last endpoint opened; zero until then. | Added to this actor's `/start` time, the latest across actors ends the blended window. | +| `requestsAtOpeningDone` | `requests` at that same moment; zero until then. | This actor's request count for its own opening, section 5. | +| `rounds` | Requests completed by the least-visited endpoint. | Two means every tunnel has been reused; the settled gate. | +| `longestGap` | Longest time between two consecutive requests on one endpoint. | Whether any tunnel went quiet long enough to be reset. | +| `errors` | Failures by class, such as `timeout`, `connection reset`, `HTTP 403`; one entry per class seen. | One class per failure mode. | +| `endpoints` | Per endpoint, by number: `requests`, `errors`, `longestGap`. `?endpoints=1` only. | Naming a failing or quiet tunnel. | + +At C = 1024 a `/stats` is about 250 bytes and the endpoint list about 60 KB +more. + +```json +{ + "scheme": "https", "elapsed": 612000000000, + "requests": 61440, "successes": 61437, "newConns": 1024, "opened": 1024, + "openingDone": 102400000000, "requestsAtOpeningDone": 1536, + "rounds": 60, "longestGap": 102600000000, + "errors": {"timeout": 3} +} +``` diff --git a/cmd/benchmarking/egress/internal/contract/api.go b/cmd/benchmarking/egress/internal/contract/api.go new file mode 100644 index 0000000000..59611fc9e7 --- /dev/null +++ b/cmd/benchmarking/egress/internal/contract/api.go @@ -0,0 +1,111 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// Package contract is what the programs of the egress scale test agree on: +// the endpoint names the actor's loop calls, the control routes and the +// /start body a driver sends it, and the Stats it answers with, down to the +// bounds of its distributions. +package contract + +import "fmt" + +// Control routes the actor serves on port 80, the only inbound port of a +// sandbox. The measurements leave the actor another way: as OTel metrics +// pushed over OTLP, so no route serves a distribution. +const ( + ReadyzRoute = "/readyz" + StartRoute = "/start" + // StatsRoute, GET, serves the running loop's Stats: the counters, cheap + // enough to poll, and nothing is consumed by a read. ?endpoints=1 adds + // the per-endpoint counts. + StatsRoute = "/stats" + // StopRoute, POST, ends the loop, flushes the last export and serves the + // final Stats. The actor keeps that final Stats once the loop is gone, so + // a repeated /stop answers it again with 200 until the next /start, while + // /stats answers 404. ?endpoints=1 as on /stats. + StopRoute = "/stop" +) + +// NoLoopMessage is the body of the 404 that /stats and /stop answer while no +// loop runs. +const NoLoopMessage = "no loop is running" + +// ProtocolHeader is on every answer the actor gives, ProtocolVersion its +// value, so a driver can tell what it is talking to before it starts +// anything: an actor whose image speaks a later contract is refused rather +// than misread. +const ( + ProtocolHeader = "Benchmark-Egress-Protocol" + ProtocolVersion = "1" +) + +// DefaultRequestTimeoutMs applies when StartRequest.RequestTimeoutMs is zero. +const DefaultRequestTimeoutMs = 5000 + +// The schemes the loop can request endpoints over. +const ( + SchemeHTTP = "http" + SchemeHTTPS = "https" +) + +// StartRequest is the JSON body of POST /start. +type StartRequest struct { + // Endpoints is C: the loop requests EndpointURL(0) through + // EndpointURL(Endpoints-1), each on its own goroutine, until it is stopped. + Endpoints int `json:"endpoints"` + // RequestTimeoutMs bounds one request; zero means DefaultRequestTimeoutMs. + RequestTimeoutMs int64 `json:"requestTimeoutMs"` + // IntervalMs paces the actor: endpoint i's first request starts OffsetMs + // plus i intervals after the loop, and each endpoint is then requested once + // per period of C intervals, so the actor sends one request per interval + // throughout. Zero sends every endpoint's requests back to back. + IntervalMs int64 `json:"intervalMs"` + // OffsetMs delays the whole schedule: endpoint i's first request starts + // OffsetMs plus i intervals after the loop. A driver gives each actor its + // own value so a fleet started at once does not send in lockstep. + OffsetMs int64 `json:"offsetMs,omitempty"` + // Scheme is SchemeHTTP or SchemeHTTPS; empty means SchemeHTTP. + Scheme string `json:"scheme,omitempty"` +} + +// URLScheme is the scheme the loop requests endpoints over. +func (r *StartRequest) URLScheme() string { + if r.Scheme == "" { + return SchemeHTTP + } + return r.Scheme +} + +// Validate reports whether r can start a loop. +func (r *StartRequest) Validate() error { + if r.Endpoints < 1 || r.Endpoints > MaxEndpoints { + return fmt.Errorf("endpoints must be between 1 and %d, got %d", MaxEndpoints, r.Endpoints) + } + if r.RequestTimeoutMs < 0 { + return fmt.Errorf("requestTimeoutMs cannot be negative: %d", r.RequestTimeoutMs) + } + if r.IntervalMs < 0 { + return fmt.Errorf("intervalMs cannot be negative: %d", r.IntervalMs) + } + if r.OffsetMs < 0 { + return fmt.Errorf("offsetMs cannot be negative: %d", r.OffsetMs) + } + if period := int64(r.Endpoints) * r.IntervalMs; r.OffsetMs > period { + return fmt.Errorf("offsetMs cannot exceed one period, endpoints x intervalMs = %d: %d", period, r.OffsetMs) + } + if r.Scheme != "" && r.Scheme != SchemeHTTP && r.Scheme != SchemeHTTPS { + return fmt.Errorf("scheme must be %s or %s, got %q", SchemeHTTP, SchemeHTTPS, r.Scheme) + } + return nil +} diff --git a/cmd/benchmarking/egress/internal/contract/api_test.go b/cmd/benchmarking/egress/internal/contract/api_test.go new file mode 100644 index 0000000000..e833923002 --- /dev/null +++ b/cmd/benchmarking/egress/internal/contract/api_test.go @@ -0,0 +1,58 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package contract + +import "testing" + +func TestStartRequestValidate(t *testing.T) { + t.Parallel() + for _, tc := range []struct { + name string + req StartRequest + wantErr bool + }{ + {"ok", StartRequest{Endpoints: 10}, false}, + {"http", StartRequest{Endpoints: 10, Scheme: SchemeHTTP}, false}, + {"https", StartRequest{Endpoints: 10, Scheme: SchemeHTTPS}, false}, + {"bad scheme", StartRequest{Endpoints: 10, Scheme: "ftp"}, true}, + {"most endpoints", StartRequest{Endpoints: MaxEndpoints}, false}, + {"no endpoints", StartRequest{}, true}, + {"too many endpoints", StartRequest{Endpoints: MaxEndpoints + 1}, true}, + {"negative timeout", StartRequest{Endpoints: 1, RequestTimeoutMs: -1}, true}, + {"negative interval", StartRequest{Endpoints: 1, IntervalMs: -1}, true}, + {"offset", StartRequest{Endpoints: 3, IntervalMs: 100, OffsetMs: 50}, false}, + {"offset of one period", StartRequest{Endpoints: 3, IntervalMs: 100, OffsetMs: 300}, false}, + {"offset over one period", StartRequest{Endpoints: 3, IntervalMs: 100, OffsetMs: 301}, true}, + {"negative offset", StartRequest{Endpoints: 1, IntervalMs: 100, OffsetMs: -1}, true}, + {"offset without interval", StartRequest{Endpoints: 1, OffsetMs: 1}, true}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + if err := tc.req.Validate(); (err != nil) != tc.wantErr { + t.Errorf("Validate() = %v, wantErr %v", err, tc.wantErr) + } + }) + } +} + +func TestURLScheme(t *testing.T) { + t.Parallel() + if got := (&StartRequest{}).URLScheme(); got != SchemeHTTP { + t.Errorf("URLScheme() with no scheme = %q, want %q", got, SchemeHTTP) + } + if got := (&StartRequest{Scheme: SchemeHTTPS}).URLScheme(); got != SchemeHTTPS { + t.Errorf("URLScheme() with https = %q, want %q", got, SchemeHTTPS) + } +} diff --git a/cmd/benchmarking/egress/internal/contract/endpoints.go b/cmd/benchmarking/egress/internal/contract/endpoints.go new file mode 100644 index 0000000000..e642c08eb0 --- /dev/null +++ b/cmd/benchmarking/egress/internal/contract/endpoints.go @@ -0,0 +1,49 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package contract + +import "fmt" + +// TargetNamespace holds the endpoint Services the benchmark deploys. +const TargetNamespace = "egress-tests-targets" + +// MaxEndpoints is the most endpoints a run can use. +const MaxEndpoints = 1024 + +// ServiceName is the name of endpoint i's Service. +func ServiceName(i int) string { + return fmt.Sprintf("egress-target-%d", i) +} + +// EndpointDomain is the DNS domain every endpoint name sits directly under. +const EndpointDomain = TargetNamespace + ".svc.cluster.local" + +// EndpointHostPattern matches every endpoint name and nothing below them: a +// wildcard stands for exactly one leftmost label. The target's certificate is +// issued for it. +const EndpointHostPattern = "*." + EndpointDomain + +// EndpointHost is the DNS name of endpoint i. It is the one place the +// endpoint names are defined: the actor's loop calls these names and the +// driver's egress policy allows them. +func EndpointHost(i int) string { + return ServiceName(i) + "." + EndpointDomain +} + +// EndpointURL is the URL the actor's loop requests for endpoint i over scheme, +// http or https, each on its default port. +func EndpointURL(i int, scheme string) string { + return scheme + "://" + EndpointHost(i) + "/" +} diff --git a/cmd/benchmarking/egress/internal/contract/endpoints_test.go b/cmd/benchmarking/egress/internal/contract/endpoints_test.go new file mode 100644 index 0000000000..d0d1693fe4 --- /dev/null +++ b/cmd/benchmarking/egress/internal/contract/endpoints_test.go @@ -0,0 +1,36 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package contract + +import "testing" + +func TestEndpointNames(t *testing.T) { + t.Parallel() + if got, want := ServiceName(7), "egress-target-7"; got != want { + t.Errorf("ServiceName(7) = %q, want %q", got, want) + } + if got, want := EndpointHost(7), "egress-target-7.egress-tests-targets.svc.cluster.local"; got != want { + t.Errorf("EndpointHost(7) = %q, want %q", got, want) + } + if got, want := EndpointURL(7, SchemeHTTP), "http://egress-target-7.egress-tests-targets.svc.cluster.local/"; got != want { + t.Errorf("EndpointURL(7, http) = %q, want %q", got, want) + } + if got, want := EndpointURL(7, SchemeHTTPS), "https://egress-target-7.egress-tests-targets.svc.cluster.local/"; got != want { + t.Errorf("EndpointURL(7, https) = %q, want %q", got, want) + } + if got, want := EndpointHostPattern, "*.egress-tests-targets.svc.cluster.local"; got != want { + t.Errorf("EndpointHostPattern = %q, want %q", got, want) + } +} diff --git a/cmd/benchmarking/egress/internal/contract/hist.go b/cmd/benchmarking/egress/internal/contract/hist.go new file mode 100644 index 0000000000..113f767ef7 --- /dev/null +++ b/cmd/benchmarking/egress/internal/contract/hist.go @@ -0,0 +1,66 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package contract + +import ( + "fmt" + "strconv" +) + +const ( + bucketsPerDecade = 20 + // maxBucketSeconds stops the bounds: the first bound at or past it is the + // last. Slower samples count in the overflow bucket. + maxBucketSeconds = 60 +) + +// mantissas are 10^(j/20) for j = 0..19, to four significant digits. +var mantissas = [bucketsPerDecade]string{ + "1", "1.122", "1.259", "1.413", "1.585", "1.778", "1.995", "2.239", "2.512", "2.818", + "3.162", "3.548", "3.981", "4.467", "5.012", "5.623", "6.31", "7.079", "7.943", "8.913", +} + +// Bounds are the upper bounds, in seconds, of every distribution: 20 per +// decade from 1 µs to 63.1 s. Each is parsed from a decimal string, so every +// build on every architecture holds the same 157 values, and a sum by bucket +// across the fleet is meaningful. Neighboring bounds differ by about 12%, the +// resolution of a quantile read from buckets. +var Bounds = func() []float64 { + var bounds []float64 + for k := 0; ; k++ { + b, err := strconv.ParseFloat(fmt.Sprintf("%se%d", mantissas[k%bucketsPerDecade], k/bucketsPerDecade-6), 64) + if err != nil { + panic(err) + } + bounds = append(bounds, b) + if b >= maxBucketSeconds { + return bounds + } + } +}() + +// Distribution names one of the four latency histograms the actor exports, +// the instrument name on the wire. +type Distribution string + +const ( + Opening Distribution = "opening" // end-to-end time of an endpoint's first request, when it succeeded + Latency Distribution = "latency" // end-to-end time of later successful requests + DNS Distribution = "dns" // lookup time, for requests that did a lookup + TLS Distribution = "tls" // handshake time, for handshakes that completed +) + +// DistributionNames lists the four distributions. +var DistributionNames = []Distribution{Opening, Latency, DNS, TLS} diff --git a/cmd/benchmarking/egress/internal/contract/hist_test.go b/cmd/benchmarking/egress/internal/contract/hist_test.go new file mode 100644 index 0000000000..19a76e5036 --- /dev/null +++ b/cmd/benchmarking/egress/internal/contract/hist_test.go @@ -0,0 +1,38 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package contract + +import ( + "strconv" + "testing" +) + +func TestBounds(t *testing.T) { + t.Parallel() + if n := len(Bounds); n != 157 { + t.Fatalf("len(Bounds) = %d, want 157", n) + } + if Bounds[0] != 1e-6 || Bounds[60] != 0.001 || Bounds[156] != 63.1 { + t.Errorf("Bounds[0,60,156] = %v %v %v, want 1e-06 0.001 63.1", Bounds[0], Bounds[60], Bounds[156]) + } + for i, b := range Bounds { + if i > 0 && b <= Bounds[i-1] { + t.Errorf("Bounds[%d] = %v does not exceed Bounds[%d] = %v", i, b, i-1, Bounds[i-1]) + } + if again, _ := strconv.ParseFloat(strconv.FormatFloat(b, 'g', 4, 64), 64); again != b { + t.Errorf("Bounds[%d] = %v has more than four significant digits", i, b) + } + } +} diff --git a/cmd/benchmarking/egress/internal/contract/stats.go b/cmd/benchmarking/egress/internal/contract/stats.go new file mode 100644 index 0000000000..ed9b0ed13e --- /dev/null +++ b/cmd/benchmarking/egress/internal/contract/stats.go @@ -0,0 +1,69 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package contract + +import "time" + +// Stats is one actor's loop counters, cumulative since /start, as GET /stats +// and POST /stop serve them. The same counts and the four latency +// distributions leave the actor as OTel metrics, not in this payload. +type Stats struct { + // Scheme is the scheme the loop requested endpoints over. + Scheme string `json:"scheme,omitempty"` + Elapsed time.Duration `json:"elapsed"` + // Requests counts every request the loop finished, failed ones included. + Requests int64 `json:"requests"` + Successes int64 `json:"successes"` + // NewConns counts requests that went out on a connection opened for them + // rather than on a kept-alive one. A tunnel the gateway reset shows up + // here rather than in Errors: net/http redials, and retries a GET that + // failed before any of the response arrived. + NewConns int64 `json:"newConns"` + // Opened counts endpoints whose first request has completed, so a reader + // can tell the opening from the steady state. + Opened int64 `json:"opened"` + // OpeningDone is Elapsed at the moment the last endpoint's first request + // completed; zero until every endpoint has opened. + OpeningDone time.Duration `json:"openingDone"` + // RequestsAtOpeningDone is Requests at that same moment, so a reader can + // price this actor's opening alone; zero until every endpoint has opened. + RequestsAtOpeningDone int64 `json:"requestsAtOpeningDone"` + // Rounds is the number of requests the least-visited endpoint has + // completed: whether this actor is still making progress. + Rounds int64 `json:"rounds"` + // LongestGap is the longest time between the starts of two consecutive + // requests on one endpoint: how quiet the loop has left a tunnel. Each + // endpoint is requested once per period, C times the interval, so with + // prompt endpoints every gap is one period; a request that outlasts the + // period delays only that endpoint's next request, so a longer gap means a + // request on that tunnel took about that long. Keep the period under the + // gateway's stream idle timeout. + LongestGap time.Duration `json:"longestGap"` + // Errors counts failed requests by class, as the actor names them: one + // low-cardinality class per failure mode. + Errors map[string]int64 `json:"errors,omitempty"` + // Endpoints holds per-endpoint counts, indexed by endpoint number. It is + // served only when asked for with ?endpoints=1. + Endpoints []Endpoint `json:"endpoints,omitempty"` +} + +// Endpoint holds the counts for one endpoint. +type Endpoint struct { + Requests int64 `json:"requests"` + Errors int64 `json:"errors"` + // LongestGap is Stats.LongestGap for this endpoint alone, so a quiet + // tunnel can be named. + LongestGap time.Duration `json:"longestGap"` +} diff --git a/cmd/benchmarking/egress/internal/contract/stats_test.go b/cmd/benchmarking/egress/internal/contract/stats_test.go new file mode 100644 index 0000000000..0ab7bfb3f2 --- /dev/null +++ b/cmd/benchmarking/egress/internal/contract/stats_test.go @@ -0,0 +1,88 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package contract + +import ( + "encoding/json" + "reflect" + "strings" + "testing" + "time" +) + +// TestStatsJSONRoundTrip pins the JSON a /stats or /stop answers with, as a +// driver decodes it: every counter, durations in nanoseconds, the errors by +// class and the per-endpoint rows, and nothing optional when there is nothing +// to report. +func TestStatsJSONRoundTrip(t *testing.T) { + t.Parallel() + for _, tc := range []struct { + name string + in Stats + present []string + absent []string + }{ + { + name: "every field", + in: Stats{ + Scheme: SchemeHTTPS, + Elapsed: 61200 * time.Millisecond, + Requests: 60, + Successes: 55, + NewConns: 4, + Opened: 4, + OpeningDone: 350 * time.Millisecond, + RequestsAtOpeningDone: 4, + Rounds: 15, + LongestGap: 400 * time.Millisecond, + Errors: map[string]int64{"HTTP 503": 3, "timeout": 2}, + Endpoints: []Endpoint{{Requests: 15, Errors: 1, LongestGap: 400 * time.Millisecond}, {Requests: 15}, {Requests: 15, Errors: 4}, {Requests: 15}}, + }, + present: []string{`"scheme":"https"`, `"elapsed":61200000000`, `"requests":60`, `"successes":55`, `"newConns":4`, `"opened":4`, + `"openingDone":350000000`, `"requestsAtOpeningDone":4`, `"rounds":15`, `"longestGap":400000000`, `"errors":{"HTTP 503":3,"timeout":2}`, `"endpoints":[{"requests":15,"errors":1,"longestGap":400000000}`}, + }, + { + name: "counters only", + in: Stats{Scheme: SchemeHTTP, Requests: 3}, + present: []string{`"scheme":"http"`, `"requests":3`}, + absent: []string{"endpoints", "errors"}, + }, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + b, err := json.Marshal(tc.in) + if err != nil { + t.Fatal(err) + } + for _, want := range tc.present { + if !strings.Contains(string(b), want) { + t.Errorf("Stats JSON lacks %s: %s", want, b) + } + } + for _, absent := range tc.absent { + if strings.Contains(string(b), absent) { + t.Errorf("Stats JSON carries %q: %s", absent, b) + } + } + var back Stats + if err := json.Unmarshal(b, &back); err != nil { + t.Fatal(err) + } + if !reflect.DeepEqual(back, tc.in) { + t.Errorf("round trip = %+v, want %+v", back, tc.in) + } + }) + } +} From 3f1bdc88adff9f330e3d004ebf7314e565c23982 Mon Sep 17 00:00:00 2001 From: Yuan Gao Date: Thu, 22 Oct 2026 10:15:36 -0700 Subject: [PATCH 3/6] benchmark-egress: name-constrained target certificate targetcert issues what an HTTPS run trusts: a throwaway CA that is name-constrained to the endpoint domain, and the certificate for the endpoint Services that the target serves. The constraint keeps a certificate the driver hands out from vouching for anything but the test's own Services. The actor's HTTPS tests verify against it. --- .../egress/internal/targetcert/targetcert.go | 159 +++++++++++++++ .../internal/targetcert/targetcert_test.go | 186 ++++++++++++++++++ 2 files changed, 345 insertions(+) create mode 100644 cmd/benchmarking/egress/internal/targetcert/targetcert.go create mode 100644 cmd/benchmarking/egress/internal/targetcert/targetcert_test.go diff --git a/cmd/benchmarking/egress/internal/targetcert/targetcert.go b/cmd/benchmarking/egress/internal/targetcert/targetcert.go new file mode 100644 index 0000000000..51812fd6db --- /dev/null +++ b/cmd/benchmarking/egress/internal/targetcert/targetcert.go @@ -0,0 +1,159 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// Package targetcert issues the name-constrained CA and the certificate the +// target serves over HTTPS, for the driver to deploy and the actor to verify +// against. +package targetcert + +import ( + "crypto/ecdsa" + "crypto/elliptic" + "crypto/rand" + "crypto/x509" + "crypto/x509/pkix" + "encoding/pem" + "fmt" + "math/big" + "os" + "path/filepath" + "time" + + "go.ate.dev/substrate/cmd/benchmarking/egress/internal/contract" +) + +// validity is long enough that a test cluster never sees the certificate +// expire; there is no rotation, since nothing can sign a replacement. +const validity = 5 * 365 * 24 * time.Hour + +// clockSkew backdates NotBefore so a node whose clock runs slightly behind +// still accepts a certificate issued moments ago. +const clockSkew = time.Hour + +// Bundle is a CA certificate and the target certificate and key it signed, +// all PEM-encoded. +type Bundle struct { + CA, Cert, Key []byte +} + +// Generate issues a new CA and the certificate the target serves +// over HTTPS, valid from now. The egress gateway re-originates TLS to the +// target and has to trust the CA, and the gateway is shared, so the CA vouches +// for nothing but the endpoint names: it is name-constrained to their domain, +// and its private key is thrown away once the target's certificate is signed. +func Generate(now time.Time) (*Bundle, error) { + b, _, _, err := generateWithCA(now) + return b, err +} + +// generateWithCA is Generate, also returning the CA certificate and key so tests +// can show what the CA refuses to vouch for. +func generateWithCA(now time.Time) (*Bundle, *x509.Certificate, *ecdsa.PrivateKey, error) { + caKey, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + if err != nil { + return nil, nil, nil, fmt.Errorf("generating CA key: %w", err) + } + caSerial, err := serialNumber() + if err != nil { + return nil, nil, nil, err + } + caTemplate := &x509.Certificate{ + SerialNumber: caSerial, + Subject: pkix.Name{CommonName: "egress-tests target CA"}, + NotBefore: now.Add(-clockSkew), + NotAfter: now.Add(validity), + KeyUsage: x509.KeyUsageCertSign | x509.KeyUsageCRLSign, + BasicConstraintsValid: true, + IsCA: true, + MaxPathLenZero: true, + // Critical, so a verifier that cannot enforce the constraint rejects + // the CA instead of trusting it for every name. + PermittedDNSDomainsCritical: true, + PermittedDNSDomains: []string{contract.EndpointDomain}, + } + caDER, err := x509.CreateCertificate(rand.Reader, caTemplate, caTemplate, &caKey.PublicKey, caKey) + if err != nil { + return nil, nil, nil, fmt.Errorf("creating CA certificate: %w", err) + } + ca, err := x509.ParseCertificate(caDER) + if err != nil { + return nil, nil, nil, fmt.Errorf("parsing CA certificate: %w", err) + } + + leafKey, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + if err != nil { + return nil, nil, nil, fmt.Errorf("generating target key: %w", err) + } + leafSerial, err := serialNumber() + if err != nil { + return nil, nil, nil, err + } + leafTemplate := &x509.Certificate{ + SerialNumber: leafSerial, + Subject: pkix.Name{CommonName: "egress-tests target"}, + // One wildcard covers every endpoint name, which differ only in the + // leftmost label. + DNSNames: []string{contract.EndpointHostPattern}, + NotBefore: now.Add(-clockSkew), + NotAfter: now.Add(validity), + KeyUsage: x509.KeyUsageDigitalSignature, + ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth}, + } + leafDER, err := x509.CreateCertificate(rand.Reader, leafTemplate, ca, &leafKey.PublicKey, caKey) + if err != nil { + return nil, nil, nil, fmt.Errorf("creating target certificate: %w", err) + } + keyDER, err := x509.MarshalPKCS8PrivateKey(leafKey) + if err != nil { + return nil, nil, nil, fmt.Errorf("encoding target key: %w", err) + } + + b := &Bundle{ + CA: pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: caDER}), + Cert: pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: leafDER}), + Key: pem.EncodeToMemory(&pem.Block{Type: "PRIVATE KEY", Bytes: keyDER}), + } + return b, ca, caKey, nil +} + +// Write stores the bundle in dir as ca.crt, tls.crt and tls.key; only the key +// is private. The files are what the deploy script turns into the target's +// TLS Secret, named as its keys. +func (b *Bundle) Write(dir string) error { + if err := os.MkdirAll(dir, 0o700); err != nil { + return err + } + for _, f := range []struct { + name string + data []byte + mode os.FileMode + }{ + {"ca.crt", b.CA, 0o644}, + {"tls.crt", b.Cert, 0o644}, + {"tls.key", b.Key, 0o600}, + } { + if err := os.WriteFile(filepath.Join(dir, f.name), f.data, f.mode); err != nil { + return err + } + } + return nil +} + +func serialNumber() (*big.Int, error) { + n, err := rand.Int(rand.Reader, new(big.Int).Lsh(big.NewInt(1), 128)) + if err != nil { + return nil, fmt.Errorf("generating serial number: %w", err) + } + return n, nil +} diff --git a/cmd/benchmarking/egress/internal/targetcert/targetcert_test.go b/cmd/benchmarking/egress/internal/targetcert/targetcert_test.go new file mode 100644 index 0000000000..c48ad7b132 --- /dev/null +++ b/cmd/benchmarking/egress/internal/targetcert/targetcert_test.go @@ -0,0 +1,186 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package targetcert + +import ( + "crypto/ecdsa" + "crypto/elliptic" + "crypto/rand" + "crypto/tls" + "crypto/x509" + "crypto/x509/pkix" + "encoding/pem" + "errors" + "math/big" + "os" + "path/filepath" + "testing" + "time" + + "go.ate.dev/substrate/cmd/benchmarking/egress/internal/contract" +) + +func parseCert(t *testing.T, data []byte) *x509.Certificate { + t.Helper() + block, _ := pem.Decode(data) + if block == nil || block.Type != "CERTIFICATE" { + t.Fatalf("not a PEM certificate: %q", data) + } + cert, err := x509.ParseCertificate(block.Bytes) + if err != nil { + t.Fatal(err) + } + return cert +} + +func verify(t *testing.T, b *Bundle, name string, at time.Time) error { + t.Helper() + roots := x509.NewCertPool() + roots.AddCert(parseCert(t, b.CA)) + _, err := parseCert(t, b.Cert).Verify(x509.VerifyOptions{ + DNSName: name, + Roots: roots, + CurrentTime: at, + KeyUsages: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth}, + }) + return err +} + +func TestCertificateCoversEveryEndpoint(t *testing.T) { + t.Parallel() + now := time.Now() + b, err := Generate(now) + if err != nil { + t.Fatal(err) + } + for i := range contract.MaxEndpoints { + if err := verify(t, b, contract.EndpointHost(i), now); err != nil { + t.Fatalf("certificate does not verify for %s: %v", contract.EndpointHost(i), err) + } + } +} + +func TestCertificateRejectsOtherNames(t *testing.T) { + t.Parallel() + now := time.Now() + b, err := Generate(now) + if err != nil { + t.Fatal(err) + } + for _, name := range []string{ + "a.egress-target-0." + contract.EndpointDomain, // a wildcard covers one label only + "egress-target-0.other-namespace.svc.cluster.local", + "example.com", + } { + t.Run(name, func(t *testing.T) { + t.Parallel() + var hostErr x509.HostnameError + if err := verify(t, b, name, now); !errors.As(err, &hostErr) { + t.Errorf("verifying for %s = %v, want a HostnameError", name, err) + } + }) + } +} + +func TestCertificateValidityWindow(t *testing.T) { + t.Parallel() + now := time.Now() + b, err := Generate(now) + if err != nil { + t.Fatal(err) + } + name := contract.EndpointHost(0) + if err := verify(t, b, name, now.Add(-30*time.Minute)); err != nil { + t.Errorf("certificate rejected half an hour before issue, inside the clock-skew allowance: %v", err) + } + for _, tc := range []struct { + name string + at time.Time + }{ + {"before the skew allowance", now.Add(-2 * time.Hour)}, + {"after expiry", now.Add(validity + time.Hour)}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + var invalid x509.CertificateInvalidError + if err := verify(t, b, name, tc.at); !errors.As(err, &invalid) || invalid.Reason != x509.Expired { + t.Errorf("verifying at %v = %v, want it outside the validity window", tc.at, err) + } + }) + } +} + +// The gateway trusts this CA for every upstream it dials, so the CA must be +// unable to vouch for any name outside the endpoints' domain, even in a +// certificate it signed itself. +func TestCARefusesNamesOutsideTheEndpointDomain(t *testing.T) { + t.Parallel() + now := time.Now() + _, ca, caKey, err := generateWithCA(now) + if err != nil { + t.Fatal(err) + } + key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + if err != nil { + t.Fatal(err) + } + der, err := x509.CreateCertificate(rand.Reader, &x509.Certificate{ + SerialNumber: big.NewInt(1), + Subject: pkix.Name{CommonName: "evil"}, + DNSNames: []string{"evil.example.com"}, + NotBefore: now.Add(-time.Hour), + NotAfter: now.Add(time.Hour), + ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth}, + }, ca, &key.PublicKey, caKey) + if err != nil { + t.Fatal(err) + } + evil, err := x509.ParseCertificate(der) + if err != nil { + t.Fatal(err) + } + roots := x509.NewCertPool() + roots.AddCert(ca) + _, err = evil.Verify(x509.VerifyOptions{DNSName: "evil.example.com", Roots: roots, CurrentTime: now}) + var invalid x509.CertificateInvalidError + if !errors.As(err, &invalid) || invalid.Reason != x509.CANotAuthorizedForThisName { + t.Errorf("a certificate for evil.example.com signed by the CA verified with %v, want CANotAuthorizedForThisName", err) + } +} + +func TestWrite(t *testing.T) { + t.Parallel() + b, err := Generate(time.Now()) + if err != nil { + t.Fatal(err) + } + dir := filepath.Join(t.TempDir(), "certs") + if err := b.Write(dir); err != nil { + t.Fatal(err) + } + if _, err := tls.LoadX509KeyPair(filepath.Join(dir, "tls.crt"), filepath.Join(dir, "tls.key")); err != nil { + t.Errorf("written certificate and key do not load as a pair: %v", err) + } + info, err := os.Stat(filepath.Join(dir, "tls.key")) + if err != nil { + t.Fatal(err) + } + if mode := info.Mode().Perm(); mode != 0o600 { + t.Errorf("key file mode = %o, want 600", mode) + } + if ca, err := os.ReadFile(filepath.Join(dir, "ca.crt")); err != nil || string(ca) != string(b.CA) { + t.Errorf("CA file = %q, %v; want the bundle's CA", ca, err) + } +} From fc5bd5cb0e1b5ac861bc54a07ea099b352aca978 Mon Sep 17 00:00:00 2001 From: Yuan Gao Date: Thu, 22 Oct 2026 10:15:38 -0700 Subject: [PATCH 4/6] benchmark-egress: actor subcommand The actor is the workload inside the sandbox. On POST /start it runs one goroutine per endpoint: endpoint i opens its tunnel i intervals after the start, then requests the endpoint once per period of C intervals, so the actor sends one request per interval and a request that outlasts the period delays only its own tunnel. Each request updates the counters under one lock and records into a Recorder: the same counts and the four latency histograms as OTel instruments on a private MeterProvider. GET /stats serves the counters; POST /stop ends the loop and serves the final counters. The loop starts on request rather than at boot because every actor restores from a golden snapshot taken before any egress policy exists. Every answer carries Benchmark-Egress-Protocol: 1, so a driver can refuse an actor whose image speaks a later contract instead of misreading its report. With it come classify's low-cardinality error classes. The README gains the loop, its pacing and offsets, and the two windows a run computes its results over. --- cmd/benchmarking/egress/README.md | 114 ++ .../egress/internal/cmd/actor/actor.go | 548 +++++++ .../egress/internal/cmd/actor/actor_test.go | 1322 +++++++++++++++++ .../egress/internal/cmd/actor/classify.go | 95 ++ .../internal/cmd/actor/classify_test.go | 79 + .../egress/internal/cmd/actor/recorder.go | 147 ++ .../internal/cmd/actor/recorder_test.go | 35 + cmd/benchmarking/egress/main.go | 4 + 8 files changed, 2344 insertions(+) create mode 100644 cmd/benchmarking/egress/internal/cmd/actor/actor.go create mode 100644 cmd/benchmarking/egress/internal/cmd/actor/actor_test.go create mode 100644 cmd/benchmarking/egress/internal/cmd/actor/classify.go create mode 100644 cmd/benchmarking/egress/internal/cmd/actor/classify_test.go create mode 100644 cmd/benchmarking/egress/internal/cmd/actor/recorder.go create mode 100644 cmd/benchmarking/egress/internal/cmd/actor/recorder_test.go diff --git a/cmd/benchmarking/egress/README.md b/cmd/benchmarking/egress/README.md index c6c7f6872f..9cbf1e2cd6 100644 --- a/cmd/benchmarking/egress/README.md +++ b/cmd/benchmarking/egress/README.md @@ -136,8 +136,53 @@ egress policy or Service shows up as errors in `/stats`, not as a failed `/start`. The instruments are created at `/start` and released at `/stop`, so every counter counts from `/start` and a later `/start` begins from zero. +The loop then runs one goroutine per endpoint: + +1. Wait offset + *i* × interval, so C handshakes do not reach the gateway at + once. +2. Send the first `GET` to the endpoint. When it completes, success or not, the + endpoint counts as opened. When the last endpoint opens, the actor stamps + `openingDone`. +3. Send one `GET` per period on a ticker started when the open starts. At most + one request is in flight per endpoint. A request that outlasts the period + delays only its own endpoint, whose next request starts as soon as the slow + one returns. +4. Stop on `POST /stop`, which waits for every goroutine and answers with the + final counters of section 4. A request cut off by the stop is not counted. + The actor keeps those counters until the next `/start`, so a `/stop` whose + answer was lost is asked again and answered again. + +The client keeps one connection per endpoint and never closes it for idling, +so a redial, one more in `newConns`, means the gateway cut the tunnel or a +request on it timed out. The gateway resets a tunnel that stays quiet for its stream idle +timeout, 5 minutes by default, so keep the per-endpoint period, C × +`intervalMs` between two requests on the same endpoint, under it. + +Only an actor's phase modulo `intervalMs` matters to the fleet: once started +it sends at `/start` + `offsetMs` + k × `intervalMs`. A driver gives actor k +of A `offsetMs` = ⌊k × `intervalMs` / A⌋, which spreads the phases evenly; a +uniform draw from [0, `intervalMs`) does about as well but needs its seed +recorded. Four actors with `intervalMs` 100: + +| Offsets | Fleet sends in each interval | +|---|---| +| 0, 0, 0, 0 | 4 at 0 ms | +| 0, 25, 50, 75 | 1 at 0, 25, 50, 75 ms | + +The offset lives on the actor because the router's fan-out of `/start` sets +each actor's real phase, and the driver does not control it. + +With these rules an actor sends one request per interval throughout the run, +and holds C tunnels. + ## 4. What the actor reports +`GET /stats` answers one small JSON object, the counters cumulative since +`/start`, for a driver's gates and progress: poll it, it consumes nothing. The +actor records the same counts and the four latency distributions as OTel +instruments on a private MeterProvider; they are the measurement, the counters +are the view a driver polls. + ### 4.1 `GET /stats` and `POST /stop` Durations are Go `time.Duration` values, nanoseconds. The per-endpoint list @@ -169,3 +214,72 @@ more. "errors": {"timeout": 3} } ``` + +## 5. What a run aggregates + +The driver is the next PR. This section is the contract it implements. + +The driver polls every actor's `/stats` for the gates and records four +instants: `t0`, the first `/start`; `t1`, when every actor's `opened` is C, +the end of the blended window; `t2`, when every actor's `rounds` is at least +two and `newConns` is flat between two polls, the start of the settled window; +`t3`, `t2` plus the cell's duration, when it posts `/stop`. Every result is a +difference between two instants. + +The results of a cell come from two windows. The blended window runs from the +first `/start` to the latest `openingDone`, each actor's shifted by its own +`/start` time. An actor's `openingDone` counts from its `/start`, so each +actor's window also covers its `offsetMs`. The settled window starts once +every actor has opened, every actor's `rounds` has reached two, checked per +actor before the sum, and `newConns` is flat between two polls. Two actors and +two endpoints, 100 ms apart, both started at the same time, actor 2 with a +50 ms offset and a slow second open: + +```mermaid +sequenceDiagram + autonumber + participant A1 as Actor 1 + participant A2 as Actor 2 + participant S0 as Service 0 + participant S1 as Service 1 + rect rgb(240, 248, 255) + A1->>+S0: open (t=0ms) + S0-->>-A1: done (t=20ms) + A2->>+S0: open (t=50ms, offset) + S0-->>-A2: done (t=70ms) + A1->>+S1: open (t=100ms) + S1-->>-A1: done (t=120ms) + A2->>+S1: open, SLOW (t=150ms) + Note over A1,S1: Repeat requests begin while the slow open is still pending + A1->>S0: repeat (t=200ms) + A2->>S0: repeat (t=250ms) + A1->>S1: repeat (t=300ms) + S1-->>-A2: latest openingDone (t=320ms) + end + Note over A1,S1: Settled, after the blended window + A2->>S1: repeat (t=350ms) + A1->>S0: repeat (t=400ms) + A2->>S0: repeat (t=450ms) + A1->>S1: repeat (t=500ms) +``` + +Both actors' first repeats fall inside the blended window, because it ends +when the slowest actor finishes its last open. The offset puts actor 2's sends +halfway between actor 1's, so the two together send one request every 50 ms +rather than two every 100 ms; without it every send of actor 2 would coincide +with one of actor 1's. The gateway's CPU in that window paid for four tunnel +setups and for every request in it; the settled rate prices every request, the +first on each tunnel included, and what is left is the setups. `t1` is the +edge the driver reads the gateway's CPU against, and the requests completed +over `t0..t1` are the window's request count; `requestsAtOpeningDone`, summed +from `/stats`, is the count at each actor's own opening end, for the opening +alone. + +| Result | Source | How it is computed | +|---|---|---| +| Settled req/s per gateway core | actors + gateway | Requests completed in the settled window divided by the gateway pod's CPU-seconds in the same window, `container_cpu_usage_seconds_total` of the envoy, ext-proc and sdsmint containers over `t2..t3`. | +| Gateway CPU per tunnel opened | actors + gateway | The blended window's gateway CPU, minus every request in it priced at the settled per-request cost, divided by the tunnels opened. What remains is setup alone, independent of pacing, on the assumption that a request on an open tunnel costs the same in either window. Computed after the settled window, since it needs the settled rate. | +| Envoy heap | gateway | The maximum of `server.memory_allocated` in the cell; per-tunnel cost is the slope across fleet sizes. | +| Settled p50, p99, p99.9 | actors | The settled window's `latency` histogram, to one bucket step. | +| Success rate | actors | `successes` over `requests` in the window. | +| Run validity | actors + gateway | `newConns` equals `opened` and stays flat; `longestGap` stays under the stream idle timeout; `dns` and `tls` do not move over `t2..t3`; `cluster.mitm_internal.upstream_cx_total` equals A × C and stays flat; `http.egress_connect.downstream_rq_idle_timeout`, `downstream_cx_idle_timeout` and `downstream_cx_protocol_error` do not move. | diff --git a/cmd/benchmarking/egress/internal/cmd/actor/actor.go b/cmd/benchmarking/egress/internal/cmd/actor/actor.go new file mode 100644 index 0000000000..23f186f005 --- /dev/null +++ b/cmd/benchmarking/egress/internal/cmd/actor/actor.go @@ -0,0 +1,548 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package actor + +import ( + "cmp" + "context" + "crypto/tls" + "crypto/x509" + "encoding/json" + "errors" + "fmt" + "io" + "log/slog" + "maps" + "net" + "net/http" + "net/http/httptrace" + "slices" + "strings" + "sync" + "time" + + "github.com/spf13/cobra" + sdkmetric "go.opentelemetry.io/otel/sdk/metric" + + "go.ate.dev/substrate/cmd/benchmarking/egress/internal/contract" +) + +// Command is the workload of the egress scale test. Once told to start, +// it opens every endpoint, one per interval, then on one goroutine per +// endpoint sends it an HTTP GET once per period of C intervals, counts what +// happened under one lock, and exports the counts and the latencies as OTel +// metrics over OTLP. +// +// The loop starts on POST /start rather than at boot: every actor restores +// from a golden snapshot taken after the template's warmup, so a loop started +// at boot would run in the golden actor, which has no egress policy, and be +// frozen into every actor's snapshot, failed requests included. +func Command() *cobra.Command { + var listen string + cmd := &cobra.Command{ + Use: "actor", + Short: "Hold one tunnel per endpoint open through the gateway and count what happened", + Args: cobra.NoArgs, + RunE: func(cmd *cobra.Command, _ []string) error { + return runActor(cmd.Context(), listen) + }, + } + cmd.Flags().StringVar(&listen, "listen", ":80", "Address to serve the readyz, start, stats and stop routes on.") + return cmd +} + +// runActor serves the actor's routes on listen until ctx ends. +func runActor(ctx context.Context, listen string) error { + srv := &http.Server{ + Addr: listen, + Handler: newServer().handler(), + ReadHeaderTimeout: 10 * time.Second, + } + go func() { + <-ctx.Done() + shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + _ = srv.Shutdown(shutdownCtx) + }() + + slog.Info("benchmark-egress actor listening", "addr", listen) + if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) { + return fmt.Errorf("serving: %w", err) + } + return nil +} + +// maxBodyBytes bounds what the actor reads, from a /start body or from an +// endpoint's response. +const maxBodyBytes = 1 << 20 + +// server serves the actor routes. At most one loop runs at a time. +type server struct { + // endpointURL maps an endpoint number and scheme to the URL the loop + // requests. Tests point it at local servers. + endpointURL func(i int, scheme string) string + // rootCAs verifies HTTPS endpoints; nil uses the system roots, which + // SSL_CERT_FILE extends with the gateway's MITM CA. Tests set their own. + rootCAs *x509.CertPool + // dial opens the loop's connections; nil uses a plain net.Dialer. Tests + // send the in-cluster endpoint names to local listeners through it. + dial dialFunc + // readers, when set, is called at each /start for readers to add to the + // loop's MeterProvider. Tests read the instruments through it. + readers func() []sdkmetric.Reader + + mu sync.Mutex + run *loopRun // nil when no loop is running + // final is the last stopped loop's Stats, endpoints included, kept until + // the next /start so a repeated /stop answers it again. + final *contract.Stats +} + +func newServer() *server { + return &server{endpointURL: contract.EndpointURL} +} + +func (s *server) handler() http.Handler { + mux := http.NewServeMux() + mux.HandleFunc("GET "+contract.ReadyzRoute, func(w http.ResponseWriter, _ *http.Request) { + w.Write([]byte("ok\n")) + }) + mux.HandleFunc("POST "+contract.StartRoute, s.handleStart) + mux.HandleFunc("GET "+contract.StatsRoute, s.handleStats) + mux.HandleFunc("POST "+contract.StopRoute, s.handleStop) + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set(contract.ProtocolHeader, contract.ProtocolVersion) + mux.ServeHTTP(w, r) + }) +} + +func (s *server) handleStart(w http.ResponseWriter, r *http.Request) { + var req contract.StartRequest + if err := json.NewDecoder(io.LimitReader(r.Body, maxBodyBytes)).Decode(&req); err != nil { + http.Error(w, "decoding start request: "+err.Error(), http.StatusBadRequest) + return + } + if err := req.Validate(); err != nil { + http.Error(w, err.Error(), http.StatusBadRequest) + return + } + + s.mu.Lock() + defer s.mu.Unlock() + if s.run != nil { + http.Error(w, "a loop is already running; POST "+contract.StopRoute+" first", http.StatusConflict) + return + } + urls := make([]string, req.Endpoints) + for i := range urls { + urls[i] = s.endpointURL(i, req.URLScheme()) + } + var readers []sdkmetric.Reader + if s.readers != nil { + readers = s.readers() + } + run, err := startLoop(r.Context(), req, urls, s.rootCAs, s.dial, readers) + if err != nil { + http.Error(w, "starting loop: "+err.Error(), http.StatusInternalServerError) + return + } + s.run, s.final = run, nil + w.WriteHeader(http.StatusNoContent) +} + +// withEndpoints reports whether the request asked for the per-endpoint +// counts. +func withEndpoints(r *http.Request) bool { + return r.URL.Query().Get("endpoints") == "1" +} + +// view is a kept Stats as a request asked for it: the per-endpoint slice +// stays only with ?endpoints=1. The kept value is not touched. +func view(s contract.Stats, endpoints bool) contract.Stats { + if !endpoints { + s.Endpoints = nil + } + return s +} + +// runOr404 returns the loop, or answers 404 and returns nil. +func (s *server) runOr404(w http.ResponseWriter) *loopRun { + s.mu.Lock() + run := s.run + s.mu.Unlock() + if run == nil { + http.Error(w, contract.NoLoopMessage, http.StatusNotFound) + } + return run +} + +func (s *server) handleStats(w http.ResponseWriter, r *http.Request) { + run := s.runOr404(w) + if run == nil { + return + } + st, ok := run.snapshot(withEndpoints(r)) + if !ok { + http.Error(w, contract.NoLoopMessage, http.StatusNotFound) + return + } + writeJSON(w, st) +} + +func (s *server) handleStop(w http.ResponseWriter, r *http.Request) { + // Held across the wait so a /start cannot slip in before the loop exits. + s.mu.Lock() + defer s.mu.Unlock() + if s.run == nil { + if s.final == nil { + http.Error(w, contract.NoLoopMessage, http.StatusNotFound) + return + } + writeJSON(w, view(*s.final, withEndpoints(r))) + return + } + st := s.run.stop() + s.run, s.final = nil, &st + writeJSON(w, view(st, withEndpoints(r))) +} + +func writeJSON(w http.ResponseWriter, v any) { + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(v) +} + +// loopRun is one started loop and what it has measured: the counters under +// its lock, mirrored into its Recorder as they change. +type loopRun struct { + cancel context.CancelFunc + wg sync.WaitGroup // one goroutine per endpoint + started time.Time + transport *http.Transport + rec *Recorder + + mu sync.Mutex + // stopped is set by stop once the goroutines are gone, so a read that + // raced the stop answers 404. + stopped bool + // stats holds the counters; Elapsed is derived at snapshot. + stats contract.Stats + last []time.Time // start of each endpoint's most recent request + // rounds is Stats.Rounds kept current, for the gauge: the least-visited + // endpoint's request count. + rounds int64 +} + +// newLoopRun is a loop over req's endpoints with a fresh Recorder and no +// goroutines yet; startLoop runs it. +func newLoopRun(ctx context.Context, req contract.StartRequest, readers []sdkmetric.Reader) (*loopRun, error) { + rec, err := NewRecorder(ctx, req, readers...) + if err != nil { + return nil, err + } + return &loopRun{ + started: time.Now(), + rec: rec, + stats: contract.Stats{Scheme: req.URLScheme(), Endpoints: make([]contract.Endpoint, req.Endpoints)}, + last: make([]time.Time, req.Endpoints), + }, nil +} + +func startLoop(ctx context.Context, req contract.StartRequest, urls []string, roots *x509.CertPool, dial dialFunc, readers []sdkmetric.Reader) (*loopRun, error) { + timeoutMs := req.RequestTimeoutMs + if timeoutMs == 0 { + timeoutMs = contract.DefaultRequestTimeoutMs + } + r, err := newLoopRun(ctx, req, readers) + if err != nil { + return nil, err + } + ctx, cancel := context.WithCancel(context.Background()) + r.cancel, r.transport = cancel, newTransport(roots, dial) + client := &http.Client{ + Transport: r.transport, + // Count a redirect as the endpoint's answer rather than following it + // somewhere the egress policy may not allow. + CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }, + } + timeout := time.Duration(timeoutMs) * time.Millisecond + interval := time.Duration(req.IntervalMs) * time.Millisecond + offset := time.Duration(req.OffsetMs) * time.Millisecond + period := interval * time.Duration(len(urls)) + for i, u := range urls { + r.wg.Go(func() { r.endpoint(ctx, client, i, u, timeout, offset+interval*time.Duration(i), period) }) + } + return r, nil +} + +// dialFunc opens one connection, as net.Dialer.DialContext does. +type dialFunc = func(ctx context.Context, network, addr string) (net.Conn, error) + +// newTransport keeps one connection alive per endpoint and verifies HTTPS +// endpoints against roots. It ignores proxy environment variables: the +// sandbox's egress capture is the only proxy under test. +func newTransport(roots *x509.CertPool, dial dialFunc) *http.Transport { + if dial == nil { + dial = (&net.Dialer{Timeout: 30 * time.Second, KeepAlive: 30 * time.Second}).DialContext + } + return &http.Transport{ + DialContext: dial, + TLSClientConfig: &tls.Config{RootCAs: roots}, + ForceAttemptHTTP2: false, // HTTP/1.1 over either scheme: one request per connection at a time + MaxIdleConns: 0, // no total cap, so C endpoints keep C connections + MaxIdleConnsPerHost: 1, // one request in flight per endpoint, so one per endpoint is enough + // No IdleConnTimeout: a kept-alive connection the client closed for + // idling would redial like one the gateway reset, and the gateway's + // stream idle timeout is what the run measures. + IdleConnTimeout: 0, + // No TLSHandshakeTimeout: the request's own timeout bounds the + // handshake, so a stalled one is counted as a timeout like any other. + } +} + +// endpoint waits offset, opens endpoint i, then requests it once per period +// until ctx ends. The opens are one interval apart across endpoints, so C +// handshakes do not reach the gateway at once. At most one request is in +// flight per endpoint, and the period is counted from the start of the open: +// a request that outlasts the period delays only this endpoint, whose next +// request starts as soon as the slow one returns. +func (r *loopRun) endpoint(ctx context.Context, client *http.Client, i int, u string, timeout, offset, period time.Duration) { + if !pause(ctx, offset) { + return + } + var tick <-chan time.Time + if period > 0 { + ticker := time.NewTicker(period) + defer ticker.Stop() + tick = ticker.C + } + r.request(ctx, client, i, u, timeout) + if ctx.Err() != nil { + return // stopped mid-open: not an open + } + r.mu.Lock() + r.stats.Opened++ + r.rec.EndpointOpened(context.Background()) + if r.stats.Opened == int64(len(r.stats.Endpoints)) { + r.stats.OpeningDone = time.Since(r.started) + r.stats.RequestsAtOpeningDone = r.stats.Requests + r.rec.Gauges(context.Background(), r.rounds, r.stats.OpeningDone.Seconds(), r.stats.LongestGap.Seconds()) + } + r.mu.Unlock() + for ctx.Err() == nil { + if tick != nil { + select { + case <-ctx.Done(): + return + case <-tick: + } + } + r.request(ctx, client, i, u, timeout) + } +} + +// pause waits d, or until ctx ends, and reports whether to go on. +func pause(ctx context.Context, d time.Duration) bool { + if d <= 0 { + return ctx.Err() == nil + } + select { + case <-ctx.Done(): + return false + case <-time.After(d): + return true + } +} + +// requestTrace collects what httptrace reports about one request. The dial, +// and with it the DNS callbacks, can still be running on another goroutine +// after a timed-out request returns, hence the lock. +type requestTrace struct { + mu sync.Mutex + gotConn bool + newConn bool + dnsStart time.Time + dns time.Duration + didDNS bool + tlsStart time.Time + tls time.Duration + tlsStarted bool + didTLS bool +} + +func (t *requestTrace) clientTrace() *httptrace.ClientTrace { + return &httptrace.ClientTrace{ + GotConn: func(info httptrace.GotConnInfo) { + t.mu.Lock() + defer t.mu.Unlock() + t.gotConn, t.newConn = true, !info.Reused + }, + DNSStart: func(httptrace.DNSStartInfo) { + t.mu.Lock() + defer t.mu.Unlock() + t.dnsStart = time.Now() + }, + DNSDone: func(httptrace.DNSDoneInfo) { + t.mu.Lock() + defer t.mu.Unlock() + t.dns, t.didDNS = time.Since(t.dnsStart), true + }, + TLSHandshakeStart: func() { + t.mu.Lock() + defer t.mu.Unlock() + t.tlsStart, t.tlsStarted = time.Now(), true + }, + TLSHandshakeDone: func(_ tls.ConnectionState, err error) { + t.mu.Lock() + defer t.mu.Unlock() + if err == nil { + t.tls, t.didTLS = time.Since(t.tlsStart), true + } + }, + } +} + +func (r *loopRun) request(ctx context.Context, client *http.Client, i int, u string, timeout time.Duration) { + reqCtx, cancel := context.WithTimeout(ctx, timeout) + defer cancel() + trace := &requestTrace{} + req, err := http.NewRequestWithContext(httptrace.WithClientTrace(reqCtx, trace.clientTrace()), http.MethodGet, u, nil) + if err != nil { + r.record(i, 0, 0, err, trace) + return + } + + start := time.Now() + r.mu.Lock() + if last := r.last[i]; !last.IsZero() { + gap := start.Sub(last) + r.stats.LongestGap = max(r.stats.LongestGap, gap) + r.stats.Endpoints[i].LongestGap = max(r.stats.Endpoints[i].LongestGap, gap) + } + r.last[i] = start + r.mu.Unlock() + status := 0 + resp, err := client.Do(req) + if err == nil { + status = resp.StatusCode + _, err = io.Copy(io.Discard, io.LimitReader(resp.Body, maxBodyBytes)) + resp.Body.Close() + } + latency := time.Since(start) + if ctx.Err() != nil { + // Stopped mid-request: the failure is the stop, not the egress path. + return + } + r.record(i, latency, status, err, trace) +} + +func (r *loopRun) record(i int, latency time.Duration, status int, err error, trace *requestTrace) { + trace.mu.Lock() + newConn := trace.gotConn && trace.newConn + didDNS, dns := trace.didDNS, trace.dns + tlsStarted, didTLS, tlsTime := trace.tlsStarted, trace.didTLS, trace.tls + trace.mu.Unlock() + + r.mu.Lock() + defer r.mu.Unlock() + s, ctx := &r.stats, context.Background() + first := s.Endpoints[i].Requests == 0 + s.Requests++ + s.Endpoints[i].Requests++ + if s.Endpoints[i].Requests-1 == r.rounds { + // Only the least-visited endpoint can raise the minimum. + r.rounds = slices.MinFunc(s.Endpoints, func(a, b contract.Endpoint) int { return cmp.Compare(a.Requests, b.Requests) }).Requests + } + if newConn { + s.NewConns++ + r.rec.ConnectionOpened(ctx) + } + r.rec.Gauges(ctx, r.rounds, s.OpeningDone.Seconds(), s.LongestGap.Seconds()) + if didDNS { + r.rec.Record(ctx, contract.DNS, dns.Seconds()) + } + if didTLS { + r.rec.Record(ctx, contract.TLS, tlsTime.Seconds()) + } + class := classify(err, status) + if class != "" && tlsStarted && !didTLS && !strings.HasPrefix(class, "tls: ") { + // A failure inside the handshake, such as the gateway closing a + // connection whose SNI no rule allows, is told apart from the same + // error in an HTTP exchange. + class = "tls handshake: " + class + } + r.rec.Request(ctx, class == "") + if class == "" { + s.Successes++ + if first { + r.rec.Record(ctx, contract.Opening, latency.Seconds()) + } else { + r.rec.Record(ctx, contract.Latency, latency.Seconds()) + } + return + } + if s.Errors == nil { + s.Errors = make(map[string]int64) + } + s.Errors[class]++ + s.Endpoints[i].Errors++ + r.rec.RequestError(ctx, class) +} + +// snapshotLocked copies the counters so far, with Elapsed derived. The +// caller holds r.mu. +func (r *loopRun) snapshotLocked(endpoints bool) contract.Stats { + s := r.stats + s.Elapsed, s.Rounds = time.Since(r.started), r.rounds + s.Errors = maps.Clone(s.Errors) + if endpoints { + s.Endpoints = slices.Clone(s.Endpoints) + } else { + s.Endpoints = nil + } + return s +} + +// snapshot answers GET /stats: the counters alone. ok is false once the +// loop is stopped. +func (r *loopRun) snapshot(endpoints bool) (contract.Stats, bool) { + r.mu.Lock() + defer r.mu.Unlock() + if r.stopped { + return contract.Stats{}, false + } + return r.snapshotLocked(endpoints), true +} + +// stop cancels the loop, waits for every endpoint's goroutine to exit, and +// returns the final Stats, endpoints included. It then flushes the last +// export and releases the Recorder; a read that raced the stop answers 404. +func (r *loopRun) stop() contract.Stats { + r.cancel() + r.wg.Wait() + r.transport.CloseIdleConnections() + r.mu.Lock() + r.stopped = true + s := r.snapshotLocked(true) + r.mu.Unlock() + // On its own context: a client that hung up must not leave the provider + // running. Bounded at twice the template's export timeout: a collector + // that is gone must not hold the driver's /stop for good. + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + if err := errors.Join(r.rec.ForceFlush(ctx), r.rec.Shutdown(ctx)); err != nil { + slog.Warn("final export failed", "err", err) + } + return s +} diff --git a/cmd/benchmarking/egress/internal/cmd/actor/actor_test.go b/cmd/benchmarking/egress/internal/cmd/actor/actor_test.go new file mode 100644 index 0000000000..ff22e57edd --- /dev/null +++ b/cmd/benchmarking/egress/internal/cmd/actor/actor_test.go @@ -0,0 +1,1322 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package actor + +import ( + "bytes" + "context" + "crypto/tls" + "crypto/x509" + "encoding/json" + "errors" + "fmt" + "io" + "log" + "net" + "net/http" + "net/http/httptest" + "net/url" + "slices" + "strings" + "sync" + "sync/atomic" + "testing" + "testing/synctest" + "time" + + sdkmetric "go.opentelemetry.io/otel/sdk/metric" + "go.opentelemetry.io/otel/sdk/metric/metricdata" + + "go.ate.dev/substrate/cmd/benchmarking/egress/internal/contract" + "go.ate.dev/substrate/cmd/benchmarking/egress/internal/targetcert" +) + +// target is an endpoint that counts the TCP connections opened to it. +type target struct { + *httptest.Server + conns atomic.Int64 + requests atomic.Int64 +} + +func newTarget(t *testing.T, h http.HandlerFunc) *target { + t.Helper() + return newTargetConnState(t, h, nil) +} + +// newTargetConnState is newTarget with onState called on every connection +// state change, after the connection count. +func newTargetConnState(t *testing.T, h http.HandlerFunc, onState func(net.Conn, http.ConnState)) *target { + t.Helper() + tg := &target{} + tg.Server = httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + tg.requests.Add(1) + h(w, r) + })) + tg.Config.ConnState = func(c net.Conn, s http.ConnState) { + if s == http.StateNew { + tg.conns.Add(1) + } + if onState != nil { + onState(c, s) + } + } + tg.Start() + t.Cleanup(tg.Close) + return tg +} + +func okHandler(w http.ResponseWriter, _ *http.Request) { w.Write([]byte("ok")) } + +// quietLog discards a test server's own error log. +var quietLog = log.New(io.Discard, "", 0) + +// host is the target's host:port. +func (tg *target) host() string { return tg.Listener.Addr().String() } + +// fixture is the actor's handler with a reader on every loop it starts, so +// a test reads the instruments the actor exports beside the counters it +// answers. +type fixture struct { + http.Handler + mu sync.Mutex + readers []*finalReader // one per /start, the current loop's last +} + +func newFixture(s *server) *fixture { + f := &fixture{} + s.readers = func() []sdkmetric.Reader { + r := &finalReader{ManualReader: sdkmetric.NewManualReader()} + f.mu.Lock() + f.readers = append(f.readers, r) + f.mu.Unlock() + return []sdkmetric.Reader{r} + } + f.Handler = s.handler() + return f +} + +// metrics is the current loop's instruments: collected now while it runs, +// as of its Shutdown once it is stopped. +func (f *fixture) metrics(t *testing.T) metricdata.ResourceMetrics { + t.Helper() + f.mu.Lock() + r := f.readers[len(f.readers)-1] + f.mu.Unlock() + rm, err := r.read(context.Background()) + if err != nil { + t.Fatal(err) + } + return rm +} + +// finalReader is a manual reader that keeps the collection of its Shutdown, +// which the loop calls once its goroutines have exited, so a test reads the +// instruments as they stood when the final Stats were answered. +type finalReader struct { + *sdkmetric.ManualReader + mu sync.Mutex + final *metricdata.ResourceMetrics +} + +func (r *finalReader) Shutdown(ctx context.Context) error { + var rm metricdata.ResourceMetrics + err := r.ManualReader.Collect(ctx, &rm) + r.mu.Lock() + r.final = &rm + r.mu.Unlock() + return errors.Join(err, r.ManualReader.Shutdown(ctx)) +} + +func (r *finalReader) read(ctx context.Context) (metricdata.ResourceMetrics, error) { + r.mu.Lock() + final := r.final + r.mu.Unlock() + if final != nil { + return *final, nil + } + var rm metricdata.ResourceMetrics + err := r.ManualReader.Collect(ctx, &rm) + return rm, err +} + +// serve returns the actor with endpoint i mapped to hosts[i], a host:port, +// in place of the in-cluster Service names. The URL keeps the scheme the +// loop asks for. +func serve(hosts ...string) *fixture { + return serveTLS(nil, hosts...) +} + +// serveTLS is serve, verifying HTTPS endpoints against roots. +func serveTLS(roots *x509.CertPool, hosts ...string) *fixture { + s := newServer() + s.rootCAs = roots + s.endpointURL = func(i int, scheme string) string { return scheme + "://" + hosts[i] + "/" } + return newFixture(s) +} + +// serveTargets is serve over n fresh targets, one per endpoint. Endpoints +// that share one host would share the transport's single idle connection per +// host and redial on every request. +func serveTargets(t *testing.T, n int) *fixture { + t.Helper() + hosts := make([]string, n) + for i := range hosts { + hosts[i] = newTarget(t, okHandler).host() + } + return serve(hosts...) +} + +// newTLSTarget is newTarget serving HTTPS with httptest's certificate, which +// the returned pool trusts. +func newTLSTarget(t *testing.T, h http.HandlerFunc) (*target, *x509.CertPool) { + t.Helper() + tg := &target{} + tg.Server = httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + tg.requests.Add(1) + h(w, r) + })) + tg.Config.ConnState = func(_ net.Conn, s http.ConnState) { + if s == http.StateNew { + tg.conns.Add(1) + } + } + tg.Config.ErrorLog = quietLog // failed handshakes are what some tests expect + tg.StartTLS() + t.Cleanup(tg.Close) + roots := x509.NewCertPool() + roots.AddCert(tg.Certificate()) + return tg, roots +} + +func post(t *testing.T, h http.Handler, path string, body any) *httptest.ResponseRecorder { + t.Helper() + var buf bytes.Buffer + if body != nil { + if err := json.NewEncoder(&buf).Encode(body); err != nil { + t.Fatal(err) + } + } + rec := httptest.NewRecorder() + h.ServeHTTP(rec, httptest.NewRequest(http.MethodPost, path, &buf)) + return rec +} + +func get(h http.Handler, path string) *httptest.ResponseRecorder { + rec := httptest.NewRecorder() + h.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, path, nil)) + return rec +} + +// decodeStats reads a 200 answer's Stats. +func decodeStats(t *testing.T, what string, rec *httptest.ResponseRecorder) *contract.Stats { + t.Helper() + if rec.Code != http.StatusOK { + t.Fatalf("%s = %d %s", what, rec.Code, rec.Body) + } + var s contract.Stats + if err := json.Unmarshal(rec.Body.Bytes(), &s); err != nil { + t.Fatalf("%s: decoding %s: %v", what, rec.Body, err) + } + return &s +} + +// The poll and the stop of a running loop, each with the per-endpoint counts. +const ( + statsPath = contract.StatsRoute + "?endpoints=1" + stopPath = contract.StopRoute + "?endpoints=1" +) + +// start posts /start with req. +func start(t *testing.T, h http.Handler, req contract.StartRequest) *httptest.ResponseRecorder { + t.Helper() + return post(t, h, contract.StartRoute, req) +} + +// readStats polls the counters. +func readStats(t *testing.T, h http.Handler) *contract.Stats { + t.Helper() + return decodeStats(t, "GET /stats", get(h, statsPath)) +} + +// stopLoop stops the loop and returns its final Stats. +func stopLoop(t *testing.T, h http.Handler) *contract.Stats { + t.Helper() + return decodeStats(t, "POST /stop", post(t, h, stopPath, nil)) +} + +// runLoop starts a loop with req, lets it run until cond holds on the live +// counters (or fails after 10s), then stops it and returns the final Stats. +func runLoop(t *testing.T, h http.Handler, req contract.StartRequest, cond func(*contract.Stats) bool) *contract.Stats { + t.Helper() + if rec := start(t, h, req); rec.Code != http.StatusNoContent { + t.Fatalf("POST /start = %d %s", rec.Code, rec.Body) + } + waitFor(t, h, cond) + return stopLoop(t, h) +} + +// waitFor polls the counters until cond holds, or fails after 10s. +func waitFor(t *testing.T, h http.Handler, cond func(*contract.Stats) bool) { + t.Helper() + deadline := time.Now().Add(10 * time.Second) + for { + s := readStats(t, h) + if cond(s) { + return + } + if time.Now().After(deadline) { + t.Fatalf("condition not reached; last stats %+v", s) + } + time.Sleep(5 * time.Millisecond) + } +} + +func atLeast(n int64) func(*contract.Stats) bool { + return func(s *contract.Stats) bool { return s.Requests >= n } +} + +// histogram is the named distribution's data point in the current loop's +// instruments; ok is false when the loop recorded nothing in it. +func histogram(t *testing.T, f *fixture, name contract.Distribution) (p metricdata.HistogramDataPoint[float64], ok bool) { + t.Helper() + for _, sm := range f.metrics(t).ScopeMetrics { + for _, m := range sm.Metrics { + if m.Name != string(name) { + continue + } + h, isHist := m.Data.(metricdata.Histogram[float64]) + if !isHist || len(h.DataPoints) != 1 { + t.Fatalf("%s is %+v, want a histogram with one point", name, m.Data) + } + return h.DataPoints[0], true + } + } + return p, false +} + +// samples is the count of a distribution in the current loop, 0 when absent. +func samples(t *testing.T, f *fixture, name contract.Distribution) uint64 { + t.Helper() + p, _ := histogram(t, f, name) + return p.Count +} + +func TestKeepAliveReusesOneConnectionPerEndpoint(t *testing.T) { + t.Parallel() + targets := []*target{newTarget(t, okHandler), newTarget(t, okHandler), newTarget(t, okHandler)} + var hosts []string + for _, tg := range targets { + hosts = append(hosts, tg.host()) + } + h := serve(hosts...) + + s := runLoop(t, h, contract.StartRequest{Endpoints: len(hosts)}, func(s *contract.Stats) bool { return s.Rounds >= 10 }) + if s.Scheme != contract.SchemeHTTP || samples(t, h, contract.TLS) != 0 || samples(t, h, contract.DNS) != 0 { + t.Errorf("a start with no scheme against IP hosts ran over %q with %d TLS handshakes and %d DNS lookups, want http with neither", s.Scheme, samples(t, h, contract.TLS), samples(t, h, contract.DNS)) + } + + if s.Successes != s.Requests || len(s.Errors) != 0 { + t.Errorf("successes %d of %d requests, errors %v", s.Successes, s.Requests, s.Errors) + } + for i, tg := range targets { + if got := tg.conns.Load(); got != 1 { + t.Errorf("target %d saw %d connections, want 1 kept alive", i, got) + } + } + if s.NewConns != int64(len(targets)) { + t.Errorf("NewConns = %d, want %d (one per endpoint)", s.NewConns, len(targets)) + } + if opening, latency := samples(t, h, contract.Opening), samples(t, h, contract.Latency); opening != uint64(len(targets)) || int64(opening+latency) != s.Successes { + t.Errorf("opening %d and latency %d samples for %d successes; want one opening per endpoint and the rest latency", opening, latency, s.Successes) + } + for i, e := range s.Endpoints { + if e.Requests < 10 { + t.Errorf("endpoint %d got %d requests, want at least the %d rounds the loop was stopped at", i, e.Requests, 10) + } + } +} + +func TestStopDuringOpeningDoesNotCountTheCancelledOpen(t *testing.T) { + t.Parallel() + first := newTarget(t, okHandler) + blocked := newTarget(t, func(w http.ResponseWriter, r *http.Request) { + <-r.Context().Done() // holds the second open until the client gives up + }) + h := serve(first.host(), blocked.host()) + + s := runLoop(t, h, contract.StartRequest{Endpoints: 2}, func(s *contract.Stats) bool { return s.Opened >= 1 }) + if s.Opened != 1 { + t.Errorf("Opened = %d after stopping during the second open, want 1", s.Opened) + } + if s.Rounds != 0 || s.OpeningDone != 0 || s.Endpoints[1].Requests != 0 { + t.Errorf("Rounds = %d, OpeningDone = %v, endpoint 1 requests = %d during the opening, want 0, 0 and 0", s.Rounds, s.OpeningDone, s.Endpoints[1].Requests) + } +} + +func TestRequestTimeoutIsClassified(t *testing.T) { + t.Parallel() + release := make(chan struct{}) + slow := newTarget(t, func(w http.ResponseWriter, r *http.Request) { + select { + case <-release: + case <-r.Context().Done(): + } + }) + t.Cleanup(func() { close(release) }) + h := serve(slow.host()) + + s := runLoop(t, h, contract.StartRequest{Endpoints: 1, RequestTimeoutMs: 20}, atLeast(2)) + + if s.Errors["timeout"] != s.Requests || s.Successes != 0 { + t.Errorf("errors %v, successes %d for %d requests; want every request a timeout", s.Errors, s.Successes, s.Requests) + } + if n := samples(t, h, contract.Latency); n != 0 { + t.Errorf("failed requests recorded %d latency samples, want 0", n) + } + if s.Endpoints[0].Errors != s.Requests { + t.Errorf("endpoint errors = %d, want %d", s.Endpoints[0].Errors, s.Requests) + } +} + +func TestNon200IsAnError(t *testing.T) { + t.Parallel() + ok := newTarget(t, okHandler) + failing := newTarget(t, func(w http.ResponseWriter, _ *http.Request) { + http.Error(w, "egress denied", http.StatusServiceUnavailable) + }) + h := serve(ok.host(), failing.host()) + + s := runLoop(t, h, contract.StartRequest{Endpoints: 2}, atLeast(10)) + + if s.Errors["HTTP 503"] == 0 { + t.Errorf("errors = %v, want HTTP 503 counted", s.Errors) + } + if s.Endpoints[0].Errors != 0 || s.Endpoints[1].Errors != s.Endpoints[1].Requests { + t.Errorf("endpoints = %+v, want only the second one failing", s.Endpoints) + } +} + +func TestStartStopLifecycle(t *testing.T) { + t.Parallel() + tg := newTarget(t, okHandler) + h := serve(tg.host()) + req := contract.StartRequest{Endpoints: 1} + + if rec := get(h, contract.StatsRoute); rec.Code != http.StatusNotFound { + t.Errorf("GET /stats before start = %d, want 404", rec.Code) + } + if rec := post(t, h, contract.StopRoute, nil); rec.Code != http.StatusNotFound { + t.Errorf("POST /stop before start = %d, want 404", rec.Code) + } + + first := runLoop(t, h, req, atLeast(5)) + time.Sleep(20 * time.Millisecond) + // A request the stop cut short can still reach the target after /stop + // returns; nothing else may. + if extra := tg.requests.Load() - first.Requests; extra > 1 { + t.Errorf("target got %d requests the stopped loop did not count, want at most the one cut short by the stop", extra) + } + + if rec := start(t, h, req); rec.Code != http.StatusNoContent { + t.Fatalf("restart = %d %s", rec.Code, rec.Body) + } + if rec := start(t, h, req); rec.Code != http.StatusConflict { + t.Errorf("second /start while running = %d, want 409", rec.Code) + } + second := stopLoop(t, h) + // The target saw every request of both runs (plus any cut short by a + // stop), so a second run that started from zero counts no more than the + // target saw after the first run. + if sinceRestart := tg.requests.Load() - first.Requests; second.Requests > sinceRestart { + t.Errorf("restarted loop counted %d requests, more than the %d sent since the restart: the old counts were kept", second.Requests, sinceRestart) + } +} + +func TestStartRejectsBadRequests(t *testing.T) { + t.Parallel() + h := newServer().handler() + for _, tc := range []struct{ name, body string }{ + {"not json", `not json`}, + {"empty object", `{}`}, + {"invalid", `{"endpoints":1,"scheme":"ftp"}`}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + rec := httptest.NewRecorder() + h.ServeHTTP(rec, httptest.NewRequest(http.MethodPost, contract.StartRoute, strings.NewReader(tc.body))) + if rec.Code != http.StatusBadRequest { + t.Errorf("POST /start %s = %d, want 400", tc.body, rec.Code) + } + }) + } + if rec := get(h, contract.ReadyzRoute); rec.Code != http.StatusOK { + t.Errorf("GET /readyz = %d, want 200", rec.Code) + } +} + +func TestDefaultURLsAreTheEndpointServices(t *testing.T) { + t.Parallel() + s := newServer() + for _, i := range []int{0, 9, 255} { + for _, scheme := range []string{contract.SchemeHTTP, contract.SchemeHTTPS} { + if got, want := s.endpointURL(i, scheme), contract.EndpointURL(i, scheme); got != want { + t.Errorf("endpoint %d over %s = %q, want %q", i, scheme, got, want) + } + } + } + if s.rootCAs != nil { + t.Error("the actor's roots are set by default; they must stay nil so SSL_CERT_FILE applies") + } + if s.dial != nil { + t.Error("the actor dials through a hook by default; it must stay nil so a plain dialer is used") + } +} + +func TestHTTPSFailuresAreClassified(t *testing.T) { + t.Parallel() + bundle, err := targetcert.Generate(time.Now()) + if err != nil { + t.Fatal(err) + } + endpointCert, err := tls.X509KeyPair(bundle.Cert, bundle.Key) + if err != nil { + t.Fatal(err) + } + endpointRoots := x509.NewCertPool() + endpointRoots.AppendCertsFromPEM(bundle.CA) + // Serves the endpoints' certificate, which names no IP address. + wrongName := &target{Server: httptest.NewUnstartedServer(http.HandlerFunc(okHandler))} + wrongName.TLS = &tls.Config{Certificates: []tls.Certificate{endpointCert}} + wrongName.Config.ErrorLog = quietLog + wrongName.StartTLS() + t.Cleanup(wrongName.Close) + + untrusted, _ := newTLSTarget(t, okHandler) + plain := newTarget(t, okHandler) + + // Accepts, reads the ClientHello so closing sends a FIN rather than a + // reset, and closes: a peer that refuses the handshake, as the gateway + // does for an SNI no rule allows. + closer, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { closer.Close() }) + go func() { + for { + conn, err := closer.Accept() + if err != nil { + return + } + _, _ = conn.Read(make([]byte, 4096)) + conn.Close() + } + }() + + for _, tc := range []struct { + name string + roots *x509.CertPool + host string + want string + }{ + {"untrusted CA", x509.NewCertPool(), untrusted.host(), "tls: unknown authority"}, + {"hostname mismatch", endpointRoots, wrongName.host(), "tls: hostname mismatch"}, + {"plain HTTP server", x509.NewCertPool(), plain.host(), "tls: not a TLS server"}, + {"peer closes during handshake", x509.NewCertPool(), closer.Addr().String(), "tls handshake: "}, + } { + t.Run(tc.name, func(t *testing.T) { + h := serveTLS(tc.roots, tc.host) + s := runLoop(t, h, contract.StartRequest{Endpoints: 1, Scheme: contract.SchemeHTTPS}, atLeast(2)) + var matched int64 + for class, n := range s.Errors { + if strings.HasPrefix(class, tc.want) { + matched += n + } + } + if matched != s.Requests || s.Successes != 0 || samples(t, h, contract.TLS) != 0 { + t.Errorf("errors %v, %d successes, %d handshakes for %d requests; want every request classed %q", s.Errors, s.Successes, samples(t, h, contract.TLS), s.Requests, tc.want) + } + }) + } +} + +// dialTo returns a dial function that connects host:port to addrs[host] +// instead, so a loop can request the in-cluster endpoint names against local +// listeners. TLS still verifies the name the loop asked for. +func dialTo(addrs map[string]string) dialFunc { + return func(ctx context.Context, network, addr string) (net.Conn, error) { + host, _, err := net.SplitHostPort(addr) + if err != nil { + return nil, err + } + real, ok := addrs[host] + if !ok { + return nil, fmt.Errorf("no local listener for %s", addr) + } + return (&net.Dialer{}).DialContext(ctx, network, real) + } +} + +// targetCertServer is an HTTPS target serving the endpoints' certificate. +func targetCertServer(t *testing.T, bundle *targetcert.Bundle) *target { + t.Helper() + cert, err := tls.X509KeyPair(bundle.Cert, bundle.Key) + if err != nil { + t.Fatal(err) + } + tg := &target{Server: httptest.NewUnstartedServer(http.HandlerFunc(okHandler))} + tg.Config.ConnState = func(_ net.Conn, s http.ConnState) { + if s == http.StateNew { + tg.conns.Add(1) + } + } + tg.TLS = &tls.Config{Certificates: []tls.Certificate{cert}} + tg.StartTLS() + t.Cleanup(tg.Close) + return tg +} + +// TestEndpointsByNameOverBothSchemes runs the loop as the actor does in the +// cluster: the default endpoint URLs, eight endpoints, and over HTTPS the +// target's own certificate verified under each endpoint's name. +func TestEndpointsByNameOverBothSchemes(t *testing.T) { + t.Parallel() + const c = 8 + bundle, err := targetcert.Generate(time.Now()) + if err != nil { + t.Fatal(err) + } + roots := x509.NewCertPool() + if !roots.AppendCertsFromPEM(bundle.CA) { + t.Fatal("no CA in the bundle") + } + for _, scheme := range []string{contract.SchemeHTTP, contract.SchemeHTTPS} { + t.Run(scheme, func(t *testing.T) { + t.Parallel() + addrs := make(map[string]string, c) + targets := make([]*target, c) + for i := range c { + if scheme == contract.SchemeHTTPS { + targets[i] = targetCertServer(t, bundle) + } else { + targets[i] = newTarget(t, okHandler) + } + addrs[contract.EndpointHost(i)] = targets[i].host() + } + s := newServer() + s.rootCAs = roots + s.dial = dialTo(addrs) + h := newFixture(s) + + const interval = 2 * time.Millisecond + st := runLoop(t, h, contract.StartRequest{Endpoints: c, Scheme: scheme, IntervalMs: int64(interval / time.Millisecond)}, + func(s *contract.Stats) bool { return s.Rounds >= 3 }) + + if st.Scheme != scheme || st.Opened != c || st.Rounds < 3 { + t.Errorf("Scheme = %q, Opened = %d, Rounds = %d, want %s, %d and at least 3", st.Scheme, st.Opened, st.Rounds, scheme, c) + } + if st.Successes != st.Requests || len(st.Errors) != 0 { + t.Errorf("successes %d of %d requests, errors %v", st.Successes, st.Requests, st.Errors) + } + if st.NewConns != c { + t.Errorf("NewConns = %d, want one per endpoint (%d)", st.NewConns, c) + } + wantTLS := uint64(0) + if scheme == contract.SchemeHTTPS { + wantTLS = c + } + if n := samples(t, h, contract.TLS); n != wantTLS { + t.Errorf("TLS handshakes = %d, want %d", n, wantTLS) + } + for i, tg := range targets { + if got := tg.conns.Load(); got != 1 { + t.Errorf("endpoint %d saw %d connections, want 1 kept alive", i, got) + } + } + // Every endpoint is requested once per period of C intervals; a gap + // is about one period, never anywhere near Elapsed. + if period := c * interval; st.LongestGap < period/2 || st.LongestGap > 5*time.Second { + t.Errorf("LongestGap = %v, want about one period (%v) and well under Elapsed", st.LongestGap, period) + } + }) + } +} + +// TestResetConnectionIsRedialedWithoutAnError pins how a tunnel the gateway +// reset shows up: the loop redials, net/http retries the GET, and NewConns +// grows by one while Errors stays empty. +func TestResetConnectionIsRedialedWithoutAnError(t *testing.T) { + t.Parallel() + var cutOnce atomic.Bool + cut := newTargetConnState(t, okHandler, func(c net.Conn, s http.ConnState) { + // Closes the server side once the first response is out, as the + // gateway does to an idle tunnel. + if s == http.StateIdle && cutOnce.CompareAndSwap(false, true) { + c.Close() + } + }) + steady := []*target{newTarget(t, okHandler), newTarget(t, okHandler)} + h := serve(steady[0].host(), cut.host(), steady[1].host()) + + s := runLoop(t, h, contract.StartRequest{Endpoints: 3, IntervalMs: 2}, func(s *contract.Stats) bool { return s.Rounds >= 2 }) + + if s.Successes != s.Requests || len(s.Errors) != 0 { + t.Errorf("successes %d of %d requests, errors %v; want the reset absorbed", s.Successes, s.Requests, s.Errors) + } + if s.NewConns != 4 { + t.Errorf("NewConns = %d, want 4: one per endpoint plus the redial", s.NewConns) + } + if got := cut.conns.Load(); got != 2 { + t.Errorf("the cut endpoint saw %d connections, want 2", got) + } + for i, tg := range steady { + if got := tg.conns.Load(); got != 1 { + t.Errorf("steady endpoint %d saw %d connections, want 1", i, got) + } + } +} + +// pipeNet serves a handler over in-memory connections, so a loop can run +// inside a synctest bubble, where a real socket would not count as blocked. +type pipeNet struct { + conns chan net.Conn + done chan struct{} + once sync.Once + srv *http.Server +} + +func newPipeNet(h http.Handler) *pipeNet { + p := &pipeNet{conns: make(chan net.Conn), done: make(chan struct{}), srv: &http.Server{Handler: h, ErrorLog: quietLog}} + go p.srv.Serve(p) + return p +} + +func (p *pipeNet) dial(ctx context.Context, _, _ string) (net.Conn, error) { + client, server := net.Pipe() + select { + case p.conns <- server: + return client, nil + case <-ctx.Done(): + return nil, ctx.Err() + } +} + +func (p *pipeNet) Accept() (net.Conn, error) { + select { + case c := <-p.conns: + return c, nil + case <-p.done: + return nil, net.ErrClosed + } +} + +func (p *pipeNet) Close() error { + p.once.Do(func() { close(p.done) }) + return nil +} + +func (p *pipeNet) Addr() net.Addr { return pipeAddr{} } + +type pipeAddr struct{} + +func (pipeAddr) Network() string { return "pipe" } +func (pipeAddr) String() string { return "pipe" } + +// startPipeLoop serves the actor over pn, with endpoint i named endpoint-i, +// and starts a loop with req. It returns the handler and a counters reader +// that waits for the bubble to settle first. +func startPipeLoop(t *testing.T, pn *pipeNet, req contract.StartRequest) (*fixture, func() *contract.Stats) { + t.Helper() + s := newServer() + s.dial = pn.dial + s.endpointURL = func(i int, scheme string) string { return fmt.Sprintf("%s://endpoint-%d/", scheme, i) } + h := newFixture(s) + if rec := start(t, h, req); rec.Code != http.StatusNoContent { + t.Fatalf("POST /start = %d %s", rec.Code, rec.Body) + } + return h, func() *contract.Stats { + synctest.Wait() + return readStats(t, h) + } +} + +// TestPacingAndGapsAreExact runs the loop on a fake clock with instant +// endpoints, so the schedule can be checked to the millisecond: endpoint i +// opens i intervals in, each endpoint is then requested once per period of C +// intervals, and so every gap is one period. At each step the instruments +// hold one opening per opened endpoint and the rest latency. +func TestPacingAndGapsAreExact(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + pn := newPipeNet(http.HandlerFunc(okHandler)) + defer pn.srv.Close() + const c, interval = 4, 100 * time.Millisecond + const period = c * interval + start := time.Now() + h, stats := startPipeLoop(t, pn, contract.StartRequest{Endpoints: c, IntervalMs: int64(interval / time.Millisecond)}) + for _, tc := range []struct { + at time.Duration + requests, opened, rounds int64 + openingDone, longestGap time.Duration + why string + }{ + {interval / 2, 1, 1, 0, 0, 0, "endpoint 0 opens at once, the others wait their turn"}, + {period - interval/2, c, c, 1, (c - 1) * interval, 0, "one open per interval; the last completes at (C-1) intervals"}, + {2*period - interval/2, 2 * c, c, 2, (c - 1) * interval, period, "each endpoint's second request comes one period after its first"}, + {3*period - interval/2, 3 * c, c, 3, (c - 1) * interval, period, "steady state: every gap is one period"}, + } { + time.Sleep(tc.at - time.Since(start)) + st := stats() + if st.Requests != tc.requests || st.Opened != tc.opened || st.Rounds != tc.rounds || st.OpeningDone != tc.openingDone || st.LongestGap != tc.longestGap { + t.Errorf("at %v: requests %d, opened %d, rounds %d, opening done %v, longest gap %v; want %d, %d, %d, %v, %v (%s)", + tc.at, st.Requests, st.Opened, st.Rounds, st.OpeningDone, st.LongestGap, tc.requests, tc.opened, tc.rounds, tc.openingDone, tc.longestGap, tc.why) + } + opening, latency := samples(t, h, contract.Opening), samples(t, h, contract.Latency) + if int64(opening) != tc.opened || int64(opening+latency) != st.Successes { + t.Errorf("at %v: %d opening and %d latency samples for %d successes; want one opening per opened endpoint (%d) and the rest latency", + tc.at, opening, latency, st.Successes, tc.opened) + } + } + synctest.Wait() + for i, e := range readStats(t, h).Endpoints { + if e.LongestGap != period { + t.Errorf("endpoint %d: longest gap %v, want one period (%v)", i, e.LongestGap, period) + } + } + stopLoop(t, h) + }) +} + +// TestOffsetShiftsEveryEndpoint pins that offsetMs delays the whole schedule +// and nothing else: endpoint 0 opens at the offset, the last endpoint at the +// offset plus C-1 intervals, and every gap is still one period. +func TestOffsetShiftsEveryEndpoint(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + pn := newPipeNet(http.HandlerFunc(okHandler)) + defer pn.srv.Close() + const c, interval, offset = 4, 100 * time.Millisecond, 50 * time.Millisecond + const period = c * interval + start := time.Now() + h, stats := startPipeLoop(t, pn, contract.StartRequest{Endpoints: c, IntervalMs: int64(interval / time.Millisecond), OffsetMs: int64(offset / time.Millisecond)}) + for _, tc := range []struct { + at time.Duration + requests, opened, rounds int64 + openingDone, longestGap time.Duration + why string + }{ + {offset / 2, 0, 0, 0, 0, 0, "nothing opens before the offset"}, + {offset + interval/2, 1, 1, 0, 0, 0, "endpoint 0 opens at the offset"}, + {offset + period - interval/2, c, c, 1, offset + (c-1)*interval, 0, "the last open completes at the offset plus C-1 intervals"}, + {offset + 2*period - interval/2, 2 * c, c, 2, offset + (c-1)*interval, period, "each repeat comes one period after the open it follows"}, + } { + time.Sleep(tc.at - time.Since(start)) + st := stats() + if st.Requests != tc.requests || st.Opened != tc.opened || st.Rounds != tc.rounds || st.OpeningDone != tc.openingDone || st.LongestGap != tc.longestGap { + t.Errorf("at %v: requests %d, opened %d, rounds %d, opening done %v, longest gap %v; want %d, %d, %d, %v, %v (%s)", + tc.at, st.Requests, st.Opened, st.Rounds, st.OpeningDone, st.LongestGap, tc.requests, tc.opened, tc.rounds, tc.openingDone, tc.longestGap, tc.why) + } + } + for i, e := range stats().Endpoints { + if e.LongestGap != period { + t.Errorf("endpoint %d: longest gap %v, want one period (%v)", i, e.LongestGap, period) + } + } + stopLoop(t, h) + }) +} + +// TestSlowEndpointDelaysOnlyItself pins that a request hanging until its +// timeout on one endpoint leaves the other endpoints' gaps at one period, and +// that the hung endpoint's own gap is the timeout: its next request starts as +// soon as the slow one returns. +func TestSlowEndpointDelaysOnlyItself(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + pn := newPipeNet(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Host == "endpoint-0" { + <-r.Context().Done() + return + } + okHandler(w, r) + })) + defer pn.srv.Close() + const c, interval, timeout = 3, 50 * time.Millisecond, 300 * time.Millisecond + const period = c * interval + h, stats := startPipeLoop(t, pn, contract.StartRequest{Endpoints: c, IntervalMs: int64(interval / time.Millisecond), RequestTimeoutMs: int64(timeout / time.Millisecond)}) + + time.Sleep(time.Second + interval/2) + st := stats() + for i := 1; i < c; i++ { + if got := st.Endpoints[i].LongestGap; got != period { + t.Errorf("endpoint %d: longest gap %v, want one period (%v) with endpoint 0 hung", i, got, period) + } + if st.Endpoints[i].Errors != 0 { + t.Errorf("endpoint %d: %d errors, want 0", i, st.Endpoints[i].Errors) + } + } + if st.Endpoints[0].LongestGap != timeout || st.LongestGap != timeout { + t.Errorf("endpoint 0 longest gap %v, overall %v; want the timeout (%v) for both", st.Endpoints[0].LongestGap, st.LongestGap, timeout) + } + if st.Errors["timeout"] == 0 || st.Errors["timeout"] != st.Endpoints[0].Errors { + t.Errorf("errors %v, endpoint 0 errors %d; want every timeout on endpoint 0", st.Errors, st.Endpoints[0].Errors) + } + if st.Opened != c || st.OpeningDone != timeout { + t.Errorf("Opened = %d, OpeningDone = %v; want %d and %v: the hung open counts once it times out, and it completes last", st.Opened, st.OpeningDone, c, timeout) + } + stopLoop(t, h) + }) +} + +// TestSlowOpensDoNotDelayOpenedEndpoints pins the schedule when opens are +// slow, as they are while a gateway works through a fleet's handshakes: an +// endpoint already opened is requested one period after its open while the +// others are still opening, and a slow open stretches only its own +// endpoint's first gap, to the open's latency. +func TestSlowOpensDoNotDelayOpenedEndpoints(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + const c, interval, openLatency = 4, 100 * time.Millisecond, 500 * time.Millisecond + const period = c * interval + var mu sync.Mutex + opened := map[string]bool{} + pn := newPipeNet(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + first := !opened[r.Host] + opened[r.Host] = true + mu.Unlock() + if first && r.Host != "endpoint-0" { + select { + case <-time.After(openLatency): + case <-r.Context().Done(): + return + } + } + okHandler(w, r) + })) + defer pn.srv.Close() + start := time.Now() + h, stats := startPipeLoop(t, pn, contract.StartRequest{Endpoints: c, IntervalMs: int64(interval / time.Millisecond)}) + + // endpoint 0 opened at once; its second request is due at one period, + // while endpoints 1 to 3 are still inside their first. + time.Sleep(period + interval/2) + st := stats() + if st.Opened != 1 || st.Endpoints[0].Requests != 2 || st.Endpoints[0].LongestGap != period || st.Requests != 2 { + t.Errorf("at %v: opened %d, endpoint 0 %d requests with gap %v, %d requests in all; want 1, 2, %v, 2: the opened endpoint keeps its period while the others open", + time.Since(start), st.Opened, st.Endpoints[0].Requests, st.Endpoints[0].LongestGap, st.Requests, period) + } + + // The last open, endpoint 3's, started at 3 intervals and took + // openLatency; after that each slow endpoint's next request came + // at once, since its period had already passed. + time.Sleep(3*period + interval/2 - time.Since(start)) + st = stats() + if st.Opened != c || st.OpeningDone != (c-1)*interval+openLatency { + t.Errorf("Opened = %d, OpeningDone = %v; want %d and %v", st.Opened, st.OpeningDone, c, (c-1)*interval+openLatency) + } + if st.Endpoints[0].LongestGap != period || st.Endpoints[0].Requests != 4 { + t.Errorf("endpoint 0: longest gap %v, %d requests; want %v and 4: the slow opens of the others did not touch it", st.Endpoints[0].LongestGap, st.Endpoints[0].Requests, period) + } + for i := 1; i < c; i++ { + if got := st.Endpoints[i].LongestGap; got != openLatency { + t.Errorf("endpoint %d: longest gap %v, want its own open latency (%v)", i, got, openLatency) + } + } + stopLoop(t, h) + op, _ := histogram(t, h, contract.Opening) + if opMax, _ := op.Max.Value(); st.LongestGap != openLatency || op.Count != c || opMax != openLatency.Seconds() { + t.Errorf("LongestGap = %v, opening samples %d with max %v; want %v, %d and %v", st.LongestGap, op.Count, opMax, openLatency, c, openLatency.Seconds()) + } + }) +} + +// TestSlowRequestResumesOnTheNextTick pins what a request longer than the +// period does to its own endpoint: the next request starts as soon as the +// slow one returns, so the gap is the request's latency, not latency plus a +// period. +func TestSlowRequestResumesOnTheNextTick(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + const period, latency = 50 * time.Millisecond, 120 * time.Millisecond + var mu sync.Mutex + var starts []time.Duration + start := time.Now() + pn := newPipeNet(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + starts = append(starts, time.Since(start)) + mu.Unlock() + select { + case <-time.After(latency): + okHandler(w, r) + case <-r.Context().Done(): + } + })) + defer pn.srv.Close() + h, stats := startPipeLoop(t, pn, contract.StartRequest{Endpoints: 1, IntervalMs: int64(period / time.Millisecond)}) + + time.Sleep(4*latency + period/2) + st := stats() + mu.Lock() + got := slices.Clone(starts) + mu.Unlock() + want := []time.Duration{0, latency, 2 * latency, 3 * latency, 4 * latency} + if !slices.Equal(got, want) { + t.Errorf("request starts = %v, want %v: back to back once each outlasts the period", got, want) + } + if st.LongestGap != latency { + t.Errorf("LongestGap = %v, want the request latency (%v)", st.LongestGap, latency) + } + stopLoop(t, h) + }) +} + +// TestStopWithRequestsInFlightReturnsAtOnce pins that a stop neither waits +// for the in-flight requests' timeouts nor leaks their goroutines. +func TestStopWithRequestsInFlightReturnsAtOnce(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + pn := newPipeNet(http.HandlerFunc(func(_ http.ResponseWriter, r *http.Request) { <-r.Context().Done() })) + defer pn.srv.Close() + const c = 8 + h, _ := startPipeLoop(t, pn, contract.StartRequest{Endpoints: c, RequestTimeoutMs: 5000}) + time.Sleep(10 * time.Millisecond) + synctest.Wait() + + before := time.Now() + st := stopLoop(t, h) + if waited := time.Since(before); waited != 0 { + t.Errorf("/stop took %v with %d requests in flight, want 0: the stop must not wait for a timeout", waited, c) + } + if _, recorded := histogram(t, h, contract.Opening); st.Requests != 0 || st.Opened != 0 || recorded { + t.Errorf("Requests = %d, Opened = %d, opening recorded %v after stopping mid-open, want 0, 0 and nothing", st.Requests, st.Opened, recorded) + } + if rec := get(h, contract.StatsRoute); rec.Code != http.StatusNotFound { + t.Errorf("GET /stats after stop = %d, want 404", rec.Code) + } + }) +} + +// TestDNSAndTLSStopMovingOnceOpened pins the emptiness the driver checks in +// a settled window: once every tunnel is open, the lookup and handshake +// counts do not move while the latency count does. +func TestDNSAndTLSStopMovingOnceOpened(t *testing.T) { + t.Parallel() + byName := newTarget(t, okHandler) + u, err := url.Parse(byName.URL) + if err != nil { + t.Fatal(err) + } + tlsTarget, roots := newTLSTarget(t, okHandler) + for _, tc := range []struct { + name string + h *fixture + req contract.StartRequest + dist contract.Distribution + }{ + {"dns over http by name", serve("localhost:" + u.Port()), contract.StartRequest{Endpoints: 1}, contract.DNS}, + {"tls over https", serveTLS(roots, tlsTarget.host()), contract.StartRequest{Endpoints: 1, Scheme: contract.SchemeHTTPS}, contract.TLS}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + if rec := start(t, tc.h, tc.req); rec.Code != http.StatusNoContent { + t.Fatalf("POST /start = %d %s", rec.Code, rec.Body) + } + waitFor(t, tc.h, atLeast(2)) + opened, first := samples(t, tc.h, tc.dist), samples(t, tc.h, contract.Latency) + if opened == 0 { + t.Errorf("the open recorded no %s sample", tc.dist) + } + // Two more requests after a count taken after the first read. + waitFor(t, tc.h, atLeast(readStats(t, tc.h).Requests+2)) + stopLoop(t, tc.h) + if later, second := samples(t, tc.h, tc.dist), samples(t, tc.h, contract.Latency); later != opened || second <= first { + t.Errorf("%s went %d -> %d and latency %d -> %d over the settled requests; want %s flat and latency up", tc.dist, opened, later, first, second, tc.dist) + } + }) + } +} + +func TestStopReturnsFinalStats(t *testing.T) { + t.Parallel() + tg := newTarget(t, okHandler) + h := serve(tg.host()) + for _, tc := range []struct { + name string + rec *httptest.ResponseRecorder + }{ + {"GET /stats", get(h, contract.StatsRoute)}, + {"POST /stop", post(t, h, contract.StopRoute, nil)}, + } { + if tc.rec.Code != http.StatusNotFound || strings.TrimSpace(tc.rec.Body.String()) != contract.NoLoopMessage { + t.Errorf("%s with no loop = %d %q, want 404 %q", tc.name, tc.rec.Code, tc.rec.Body, contract.NoLoopMessage) + } + } + if rec := start(t, h, contract.StartRequest{Endpoints: 1}); rec.Code != http.StatusNoContent { + t.Fatalf("POST /start = %d %s", rec.Code, rec.Body) + } + waitFor(t, h, atLeast(1)) + rec := post(t, h, contract.StopRoute, nil) + final := decodeStats(t, "POST /stop", rec) + if ct := rec.Header().Get("Content-Type"); ct != "application/json" || final.Scheme != contract.SchemeHTTP || final.Endpoints != nil || final.Requests == 0 { + t.Errorf("POST /stop answered %q with scheme %q, endpoints %v and %d requests; want JSON, http, no endpoints unless asked, and the requests made", ct, final.Scheme, final.Endpoints, final.Requests) + } + if rec := get(h, contract.StatsRoute); rec.Code != http.StatusNotFound { + t.Errorf("GET /stats after stop = %d, want 404", rec.Code) + } + + // The final report is kept: a /stop the driver did not hear is answered + // again, byte for byte, with or without the endpoints, until a new loop + // replaces it. + if again := post(t, h, contract.StopRoute, nil); again.Code != http.StatusOK || !bytes.Equal(again.Body.Bytes(), rec.Body.Bytes()) { + t.Errorf("second POST /stop = %d %s; want 200 and the first answer %s", again.Code, again.Body, rec.Body) + } + if withEP := decodeStats(t, "third POST /stop", post(t, h, stopPath, nil)); len(withEP.Endpoints) != 1 || withEP.Requests != final.Requests { + t.Errorf("POST /stop?endpoints=1 after stop: %d endpoints, %d requests; want 1 and %d", len(withEP.Endpoints), withEP.Requests, final.Requests) + } + if rec := start(t, h, contract.StartRequest{Endpoints: 1, IntervalMs: 60_000, OffsetMs: 60_000}); rec.Code != http.StatusNoContent { + t.Fatalf("second POST /start = %d %s", rec.Code, rec.Body) + } + if next := stopLoop(t, h); next.Requests != 0 { + t.Errorf("the stop after a new /start reports %d requests; want 0, the new loop's own report", next.Requests) + } +} + +// TestRequestsAtOpeningDone pins the count the blended window prices: the +// requests this actor had finished when its last endpoint opened, repeats on +// the endpoints already open included, and fixed from then on. +func TestRequestsAtOpeningDone(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + const c, interval, openLatency = 2, 100 * time.Millisecond, 250 * time.Millisecond + var mu sync.Mutex + opened := map[string]bool{} + pn := newPipeNet(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + first := !opened[r.Host] + opened[r.Host] = true + mu.Unlock() + if first && r.Host == "endpoint-1" { + select { + case <-time.After(openLatency): + case <-r.Context().Done(): + return + } + } + okHandler(w, r) + })) + defer pn.srv.Close() + start := time.Now() + h, stats := startPipeLoop(t, pn, contract.StartRequest{Endpoints: c, IntervalMs: int64(interval / time.Millisecond)}) + + // Endpoint 0 opens at 0 and repeats at 200; endpoint 1 opens at 100 + // and finishes at 350: the third request. Its first repeat follows at + // once, since its period passed during the open. + time.Sleep(interval + openLatency - time.Since(start)) + st := stats() + if st.Opened != c || st.OpeningDone != interval+openLatency || st.RequestsAtOpeningDone != 3 || st.Requests != 4 { + t.Errorf("at %v: opened %d at %v with %d requests, %d requests now; want %d at %v with 3, 4", time.Since(start), st.Opened, st.OpeningDone, st.RequestsAtOpeningDone, st.Requests, c, interval+openLatency) + } + time.Sleep(time.Second - time.Since(start)) + if st = stats(); st.RequestsAtOpeningDone != 3 || st.Requests <= 4 { + t.Errorf("at %v: requestsAtOpeningDone %d with %d requests; want still 3 with more than 4", time.Since(start), st.RequestsAtOpeningDone, st.Requests) + } + if final := stopLoop(t, h); final.RequestsAtOpeningDone != 3 { + t.Errorf("final requestsAtOpeningDone = %d, want 3", final.RequestsAtOpeningDone) + } + }) +} + +// TestFreshRecorderPerLoop pins that a loop starts from nothing: after +// start-stop-start the new loop's counters and instruments are empty. +func TestFreshRecorderPerLoop(t *testing.T) { + t.Parallel() + h := serveTargets(t, 2) + first := runLoop(t, h, contract.StartRequest{Endpoints: 2}, atLeast(10)) + if samples(t, h, contract.Latency) == 0 { + t.Fatalf("first loop recorded no latency over %d requests", first.Requests) + } + // Nothing in the new loop fires for a minute, so its instruments hold + // only what the previous loop might have left behind. + if rec := start(t, h, contract.StartRequest{Endpoints: 2, IntervalMs: 60_000, OffsetMs: 60_000}); rec.Code != http.StatusNoContent { + t.Fatalf("POST /start = %d %s", rec.Code, rec.Body) + } + second := readStats(t, h) + if _, recorded := histogram(t, h, contract.Latency); second.Requests != 0 || second.Opened != 0 || recorded { + t.Errorf("second loop: %d requests, %d opened, latency recorded %v; want nothing", second.Requests, second.Opened, recorded) + } + stopLoop(t, h) +} + +// TestInstrumentsTrackCountersUnderRace reads the instruments while requests +// land: every success records exactly one opening or latency sample under +// the same lock as the counters, so a histogram count read between two +// /stats polls lies between their Successes, every /stats poll holds +// successes + errors = requests, and the final instruments equal the final +// counters. +func TestInstrumentsTrackCountersUnderRace(t *testing.T) { + t.Parallel() + h := serveTargets(t, 8) + if rec := start(t, h, contract.StartRequest{Endpoints: 8}); rec.Code != http.StatusNoContent { + t.Fatalf("POST /start = %d %s", rec.Code, rec.Body) + } + var wg sync.WaitGroup + var badReads atomic.Int64 + for range 4 { + wg.Go(func() { + for range 50 { + st := readStats(t, h) + var failed int64 + for _, n := range st.Errors { + failed += n + } + if st.Successes+failed != st.Requests { + badReads.Add(1) + } + } + }) + } + var outOfOrder int + for range 200 { + before := readStats(t, h) + got := samples(t, h, contract.Opening) + samples(t, h, contract.Latency) + after := readStats(t, h) + if int64(got) < before.Successes || int64(got) > after.Successes { + outOfOrder++ + } + } + wg.Wait() + final := stopLoop(t, h) + opening, latency := samples(t, h, contract.Opening), samples(t, h, contract.Latency) + if badReads.Load() > 0 || outOfOrder > 0 { + t.Errorf("%d reads broke successes + errors = requests, %d instrument reads fell outside the polls around them", badReads.Load(), outOfOrder) + } + if int64(opening+latency) != final.Successes || opening != 8 || final.Successes == 0 { + t.Errorf("final instruments hold %d opening and %d latency samples for %d successes over 8 endpoints", opening, latency, final.Successes) + } +} + +// TestStatsDuringStopIsNever500 races many reads against a stop: each one +// answers 200 while the loop runs or 404 once it is stopped, never a 500. +func TestStatsDuringStopIsNever500(t *testing.T) { + t.Parallel() + h := serveTargets(t, 8) + if rec := start(t, h, contract.StartRequest{Endpoints: 8}); rec.Code != http.StatusNoContent { + t.Fatalf("POST /start = %d %s", rec.Code, rec.Body) + } + var wg sync.WaitGroup + var ok, gone, other atomic.Int64 + stopped := make(chan struct{}) + for range 16 { + wg.Go(func() { + for { + rec := get(h, contract.StatsRoute) + switch { + case rec.Code == http.StatusOK: + ok.Add(1) + case rec.Code == http.StatusNotFound && strings.TrimSpace(rec.Body.String()) == contract.NoLoopMessage: + gone.Add(1) + default: + other.Add(1) + t.Errorf("answered %d %s", rec.Code, rec.Body) + } + select { + case <-stopped: + return + default: + } + } + }) + } + time.Sleep(20 * time.Millisecond) + stopLoop(t, h) + close(stopped) + wg.Wait() + if ok.Load() == 0 || gone.Load() == 0 || other.Load() != 0 { + t.Errorf("%d answers were 200, %d 404, %d something else", ok.Load(), gone.Load(), other.Load()) + } +} + +// TestEveryRouteCarriesProtocolHeader checks the header a driver reads to +// refuse an actor that speaks another contract, on every answer the actor +// gives, errors and unknown routes included. +func TestEveryRouteCarriesProtocolHeader(t *testing.T) { + t.Parallel() + if contract.ProtocolVersion != "1" { + t.Errorf("ProtocolVersion = %q, want 1; a driver requires it", contract.ProtocolVersion) + } + h := newServer().handler() + for _, tc := range []struct { + name string + rec *httptest.ResponseRecorder + want int + }{ + {"readyz", get(h, contract.ReadyzRoute), http.StatusOK}, + {"start rejected", post(t, h, contract.StartRoute, map[string]int{"endpoints": 0}), http.StatusBadRequest}, + {"stats without a loop", get(h, contract.StatsRoute), http.StatusNotFound}, + {"stop without a loop", post(t, h, contract.StopRoute, nil), http.StatusNotFound}, + {"unknown route", get(h, "/metrics"), http.StatusNotFound}, + } { + if tc.rec.Code != tc.want { + t.Errorf("%s: status %d, want %d", tc.name, tc.rec.Code, tc.want) + } + if got := tc.rec.Header().Get(contract.ProtocolHeader); got != contract.ProtocolVersion { + t.Errorf("%s: %s = %q, want %q", tc.name, contract.ProtocolHeader, got, contract.ProtocolVersion) + } + } +} + +// TestPayloadBudget pins the answer sizes at the largest C, since the driver +// polls every actor: /stats stays under 1 KiB, and the per-endpoint counts +// come only when asked for, under 80 KiB. +func TestPayloadBudget(t *testing.T) { + t.Parallel() + const c = contract.MaxEndpoints + run, err := newLoopRun(context.Background(), contract.StartRequest{Endpoints: c, Scheme: contract.SchemeHTTPS}, nil) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = run.rec.Shutdown(context.Background()) }) + run.stats.Requests, run.stats.Successes, run.stats.NewConns, run.stats.Opened = 12345*c, 12339*c, c, c + run.stats.OpeningDone, run.stats.LongestGap = 102400*time.Millisecond, 12345678*time.Microsecond + run.stats.Errors = map[string]int64{"timeout": 3 * c, "HTTP 503": 3 * c} + run.rounds = 12345 + for i := range c { + run.stats.Endpoints[i] = contract.Endpoint{Requests: 12345, Errors: 6, LongestGap: 12345678 * time.Microsecond} + } + s := newServer() + s.run = run + h := s.handler() + + stats := get(h, contract.StatsRoute) + statsEndpoints := get(h, statsPath) + t.Logf("at C=%d: /stats %d B, /stats?endpoints=1 %d B", c, stats.Body.Len(), statsEndpoints.Body.Len()) + for _, tc := range []struct { + name string + rec *httptest.ResponseRecorder + budget int + endpoints bool + }{ + {"/stats", stats, 1 << 10, false}, + {"/stats?endpoints=1", statsEndpoints, 80 << 10, true}, + } { + if tc.rec.Code != http.StatusOK { + t.Fatalf("%s = %d %s", tc.name, tc.rec.Code, tc.rec.Body) + } + if tc.rec.Body.Len() >= tc.budget { + t.Errorf("%s = %d B, budget %d B", tc.name, tc.rec.Body.Len(), tc.budget) + } + var st contract.Stats + if err := json.Unmarshal(tc.rec.Body.Bytes(), &st); err != nil { + t.Fatal(err) + } + if (len(st.Endpoints) == c) != tc.endpoints || st.Rounds != 12345 { + t.Errorf("%s carries %d endpoints and rounds %d; want endpoints %v and rounds 12345", tc.name, len(st.Endpoints), st.Rounds, tc.endpoints) + } + } +} diff --git a/cmd/benchmarking/egress/internal/cmd/actor/classify.go b/cmd/benchmarking/egress/internal/cmd/actor/classify.go new file mode 100644 index 0000000000..09e752a18f --- /dev/null +++ b/cmd/benchmarking/egress/internal/cmd/actor/classify.go @@ -0,0 +1,95 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package actor + +import ( + "context" + "crypto/tls" + "crypto/x509" + "errors" + "fmt" + "io" + "net" + "net/url" + "strings" + "syscall" +) + +// maxOtherLen bounds the message kept for an error classify does not +// recognize, so one odd error cannot flood the report. +const maxOtherLen = 80 + +// classify names the outcome of a request in a few low-cardinality classes. +// It returns "" for a success: no error and HTTP 200. +func classify(err error, status int) string { + if err == nil { + if status == 200 { + return "" + } + return fmt.Sprintf("HTTP %d", status) + } + var netErr net.Error + var dnsErr *net.DNSError + var opErr *net.OpError + switch { + case errors.As(err, &dnsErr): + if dnsErr.IsNotFound { + return "dns: not found" + } + if dnsErr.IsTimeout { + return "dns: timeout" + } + return "dns: error" + case errors.As(err, new(x509.UnknownAuthorityError)): + return "tls: unknown authority" + case errors.As(err, new(x509.HostnameError)): + return "tls: hostname mismatch" + case errors.As(err, new(x509.CertificateInvalidError)): + return "tls: invalid certificate" + case errors.As(err, new(tls.RecordHeaderError)), + // net/http replaces the record error with this when the server + // answered in plain HTTP, without wrapping it. + strings.Contains(err.Error(), "server gave HTTP response to HTTPS client"): + return "tls: not a TLS server" + case errors.As(err, &opErr) && opErr.Op == "remote error": + // crypto/tls reports an alert from the peer this way; its message, + // such as "tls: handshake failure", names the alert. + return "tls: peer alert: " + strings.TrimPrefix(opErr.Err.Error(), "tls: ") + case errors.Is(err, context.DeadlineExceeded), errors.As(err, &netErr) && netErr.Timeout(): + return "timeout" + case errors.Is(err, syscall.ECONNREFUSED): + return "connection refused" + case errors.Is(err, syscall.ECONNRESET): + return "connection reset" + case errors.Is(err, io.EOF), errors.Is(err, io.ErrUnexpectedEOF): + return "EOF" + } + // *url.Error repeats the method and URL, and *net.OpError the addresses + // and the local port; both only add cardinality. + var urlErr *url.Error + if errors.As(err, &urlErr) { + err = urlErr.Err + } + if errors.As(err, &opErr) { + return "net: " + opErr.Op + ": " + opErr.Err.Error() + } + msg := err.Error() + if len(msg) > maxOtherLen { + // Cut on a character boundary, or one message could yield two + // classes depending on where its bytes fall. + msg = strings.ToValidUTF8(msg[:maxOtherLen], "") + } + return "other: " + msg +} diff --git a/cmd/benchmarking/egress/internal/cmd/actor/classify_test.go b/cmd/benchmarking/egress/internal/cmd/actor/classify_test.go new file mode 100644 index 0000000000..0728af73e3 --- /dev/null +++ b/cmd/benchmarking/egress/internal/cmd/actor/classify_test.go @@ -0,0 +1,79 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package actor + +import ( + "context" + "crypto/tls" + "crypto/x509" + "errors" + "io" + "net" + "net/url" + "strings" + "syscall" + "testing" +) + +type timeoutErr struct{} + +func (timeoutErr) Error() string { return "i/o timeout" } +func (timeoutErr) Timeout() bool { return true } +func (timeoutErr) Temporary() bool { return true } + +func TestClassify(t *testing.T) { + t.Parallel() + wrap := func(err error) error { return &url.Error{Op: "Get", URL: "http://x/", Err: err} } + for _, tc := range []struct { + name string + err error + status int + want string + }{ + {"success", nil, 200, ""}, + {"http status", nil, 503, "HTTP 503"}, + {"deadline", wrap(context.DeadlineExceeded), 0, "timeout"}, + {"net timeout", wrap(&net.OpError{Op: "dial", Err: timeoutErr{}}), 0, "timeout"}, + {"refused", wrap(&net.OpError{Op: "dial", Err: syscall.ECONNREFUSED}), 0, "connection refused"}, + {"reset", wrap(&net.OpError{Op: "read", Err: syscall.ECONNRESET}), 0, "connection reset"}, + {"eof", wrap(io.EOF), 0, "EOF"}, + {"unexpected eof", wrap(io.ErrUnexpectedEOF), 0, "EOF"}, + {"dns not found", wrap(&net.DNSError{Err: "no such host", Name: "x", IsNotFound: true}), 0, "dns: not found"}, + {"dns other", wrap(&net.DNSError{Err: "server misbehaving", Name: "x"}), 0, "dns: error"}, + {"dns timeout", wrap(&net.DNSError{Err: "i/o timeout", Name: "x", IsTimeout: true}), 0, "dns: timeout"}, + {"long other is cut", wrap(errors.New(strings.Repeat("x", maxOtherLen+20))), 0, "other: " + strings.Repeat("x", maxOtherLen)}, + {"cut keeps whole characters", wrap(errors.New(strings.Repeat("a", maxOtherLen-1) + "é!")), 0, "other: " + strings.Repeat("a", maxOtherLen-1)}, + {"other strips url", wrap(errors.New("boom")), 0, "other: boom"}, + {"unknown authority", wrap(&tls.CertificateVerificationError{Err: x509.UnknownAuthorityError{}}), 0, "tls: unknown authority"}, + {"hostname mismatch", wrap(&tls.CertificateVerificationError{Err: x509.HostnameError{Host: "x"}}), 0, "tls: hostname mismatch"}, + {"expired", wrap(&tls.CertificateVerificationError{Err: x509.CertificateInvalidError{Reason: x509.Expired}}), 0, "tls: invalid certificate"}, + {"not tls", wrap(tls.RecordHeaderError{Msg: "first record does not look like a TLS handshake"}), 0, "tls: not a TLS server"}, + {"plain http answer", wrap(errors.New("http: server gave HTTP response to HTTPS client")), 0, "tls: not a TLS server"}, + {"peer alert", wrap(&net.OpError{Op: "remote error", Err: errors.New("tls: handshake failure")}), 0, "tls: peer alert: handshake failure"}, + {"net error without its addresses", &url.Error{Op: "Get", URL: "http://10.0.0.1:8080/", Err: &net.OpError{ + Op: "write", Net: "tcp", + Source: &net.TCPAddr{IP: net.IPv4(10, 0, 0, 2), Port: 51234}, + Addr: &net.TCPAddr{IP: net.IPv4(10, 0, 0, 1), Port: 8080}, + Err: syscall.EPIPE, + }}, 0, "net: write: broken pipe"}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + if got := classify(tc.err, tc.status); got != tc.want { + t.Errorf("classify() = %q, want %q", got, tc.want) + } + }) + } +} diff --git a/cmd/benchmarking/egress/internal/cmd/actor/recorder.go b/cmd/benchmarking/egress/internal/cmd/actor/recorder.go new file mode 100644 index 0000000000..99de7879bd --- /dev/null +++ b/cmd/benchmarking/egress/internal/cmd/actor/recorder.go @@ -0,0 +1,147 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// Package actor is the workload of the egress scale test: the subcommand +// that holds one tunnel per endpoint open through the egress gateway and +// counts what happened. +package actor + +import ( + "context" + "errors" + "fmt" + + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/metric" + sdkmetric "go.opentelemetry.io/otel/sdk/metric" + + "go.ate.dev/substrate/cmd/benchmarking/egress/internal/contract" +) + +// The names of the loop's instruments, the contract with the queries that +// read them once the collector has served them to Prometheus: a counter +// gains _total there, a unit of seconds _seconds. +const ( + meterName = "benchmark-egress" + requestsName = "requests" + requestErrorsName = "request_errors" + connsOpenedName = "connections_opened" + endpointsOpenedName = "endpoints_opened" + roundsName = "rounds" + openingDoneName = "opening_done" + longestGapName = "longest_gap" +) + +// Recorder holds one loop's instruments on a private MeterProvider: the +// counters that mirror its Stats and the four latency histograms. Without a +// reader every method is a cheap no-op. +type Recorder struct { + provider *sdkmetric.MeterProvider + hists map[contract.Distribution]metric.Float64Histogram + + requests, requestErrors, connsOpened, endpointsOpened metric.Int64Counter + rounds metric.Int64Gauge + openingDone, longestGap metric.Float64Gauge + + // attrs is on every point; success and failure add the result. + attrs, success, failure metric.MeasurementOption +} + +// NewRecorder creates the instruments for the loop req starts, on a private +// MeterProvider with readers, which tests use to read the instruments. +func NewRecorder(ctx context.Context, req contract.StartRequest, readers ...sdkmetric.Reader) (*Recorder, error) { + opts := make([]sdkmetric.Option, 0, len(readers)) + for _, rd := range readers { + opts = append(opts, sdkmetric.WithReader(rd)) + } + r := &Recorder{provider: sdkmetric.NewMeterProvider(opts...), hists: make(map[contract.Distribution]metric.Float64Histogram, len(contract.DistributionNames))} + withResult := func(result string) metric.MeasurementOption { + return metric.WithAttributeSet(attribute.NewSet(attribute.String("result", result))) + } + r.attrs, r.success, r.failure = metric.WithAttributeSet(attribute.NewSet()), withResult("success"), withResult("error") + if err := r.instruments(r.provider.Meter(meterName)); err != nil { + _ = r.provider.Shutdown(context.Background()) + return nil, err + } + return r, nil +} + +// instruments creates the eleven instruments on meter. +func (r *Recorder) instruments(meter metric.Meter) error { + var err error + counter := func(name, desc string) metric.Int64Counter { + c, cerr := meter.Int64Counter(name, metric.WithDescription(desc)) + err = errors.Join(err, cerr) + return c + } + r.requests = counter(requestsName, "Requests the loop finished, by result.") + r.requestErrors = counter(requestErrorsName, "Failed requests, by class.") + r.connsOpened = counter(connsOpenedName, "Requests sent on a freshly dialed connection.") + r.endpointsOpened = counter(endpointsOpenedName, "Endpoints whose first request has completed.") + var gerr error + r.rounds, gerr = meter.Int64Gauge(roundsName, metric.WithDescription("Requests completed by the least-visited endpoint.")) + err = errors.Join(err, gerr) + r.openingDone, gerr = meter.Float64Gauge(openingDoneName, metric.WithUnit("s"), metric.WithDescription("Seconds from /start to the last endpoint's open; zero until then.")) + err = errors.Join(err, gerr) + r.longestGap, gerr = meter.Float64Gauge(longestGapName, metric.WithUnit("s"), metric.WithDescription("Longest gap between two consecutive requests on one endpoint.")) + err = errors.Join(err, gerr) + for _, name := range contract.DistributionNames { + h, herr := meter.Float64Histogram(string(name), metric.WithUnit("s"), metric.WithExplicitBucketBoundaries(contract.Bounds...)) + err = errors.Join(err, herr) + r.hists[name] = h + } + if err != nil { + return fmt.Errorf("creating the instruments: %w", err) + } + return nil +} + +// Record adds one sample, in seconds, to the named distribution. +func (r *Recorder) Record(ctx context.Context, name contract.Distribution, seconds float64) { + r.hists[name].Record(ctx, seconds, r.attrs) +} + +// Request counts one finished request by result. +func (r *Recorder) Request(ctx context.Context, ok bool) { + if ok { + r.requests.Add(ctx, 1, r.success) + } else { + r.requests.Add(ctx, 1, r.failure) + } +} + +// RequestError counts one failed request under its class. +func (r *Recorder) RequestError(ctx context.Context, class string) { + r.requestErrors.Add(ctx, 1, r.attrs, metric.WithAttributes(attribute.String("class", class))) +} + +// ConnectionOpened counts one request sent on a freshly dialed connection. +func (r *Recorder) ConnectionOpened(ctx context.Context) { r.connsOpened.Add(ctx, 1, r.attrs) } + +// EndpointOpened counts one endpoint whose first request completed. +func (r *Recorder) EndpointOpened(ctx context.Context) { r.endpointsOpened.Add(ctx, 1, r.attrs) } + +// Gauges records the three derived values of the Stats. +func (r *Recorder) Gauges(ctx context.Context, rounds int64, openingDone, longestGap float64) { + r.rounds.Record(ctx, rounds, r.attrs) + r.openingDone.Record(ctx, openingDone, r.attrs) + r.longestGap.Record(ctx, longestGap, r.attrs) +} + +// ForceFlush exports what every reader holds, now. +func (r *Recorder) ForceFlush(ctx context.Context) error { return r.provider.ForceFlush(ctx) } + +// Shutdown exports once more and releases the provider. Every method is a +// no-op afterwards. +func (r *Recorder) Shutdown(ctx context.Context) error { return r.provider.Shutdown(ctx) } diff --git a/cmd/benchmarking/egress/internal/cmd/actor/recorder_test.go b/cmd/benchmarking/egress/internal/cmd/actor/recorder_test.go new file mode 100644 index 0000000000..cba501aa0c --- /dev/null +++ b/cmd/benchmarking/egress/internal/cmd/actor/recorder_test.go @@ -0,0 +1,35 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package actor + +import ( + "context" + "testing" + + "go.ate.dev/substrate/cmd/benchmarking/egress/internal/contract" +) + +func BenchmarkRecord(b *testing.B) { + r, err := NewRecorder(context.Background(), contract.StartRequest{}) + if err != nil { + b.Fatal(err) + } + defer r.Shutdown(context.Background()) + ctx := context.Background() + for i := 0; b.Loop(); i++ { + r.Request(ctx, true) + r.Record(ctx, contract.Latency, float64(i%1000)*1e-5) + } +} diff --git a/cmd/benchmarking/egress/main.go b/cmd/benchmarking/egress/main.go index 34bbe057da..044276fb79 100644 --- a/cmd/benchmarking/egress/main.go +++ b/cmd/benchmarking/egress/main.go @@ -26,6 +26,8 @@ import ( "syscall" "github.com/spf13/cobra" + + "go.ate.dev/substrate/cmd/benchmarking/egress/internal/cmd/actor" ) func main() { @@ -42,6 +44,8 @@ happened; the target is the endpoint every tunnel leads to.`, // Nobody types these commands often enough to want shell completion. CompletionOptions: cobra.CompletionOptions{DisableDefaultCmd: true}, } + root.AddCommand(actor.Command()) + ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM) defer stop() if err := root.ExecuteContext(ctx); err != nil { From 48d32fbf961f96850f6efda1dbe1c483219c25f6 Mon Sep 17 00:00:00 2001 From: Yuan Gao Date: Thu, 22 Oct 2026 10:15:39 -0700 Subject: [PATCH 5/6] benchmark-egress: target subcommand The target is the endpoint every tunnel leads to: a 200 with a small fixed body over plain HTTP and, given a certificate, HTTPS. One Deployment sits behind all of the test's Services, each on its own backend port, so two Services on distinct ClusterIPs cannot hand it two connections with the same 4-tuple. --- .../egress/internal/cmd/target/target.go | 208 ++++++++++++++++++ .../egress/internal/cmd/target/target_test.go | 159 +++++++++++++ cmd/benchmarking/egress/main.go | 3 +- 3 files changed, 369 insertions(+), 1 deletion(-) create mode 100644 cmd/benchmarking/egress/internal/cmd/target/target.go create mode 100644 cmd/benchmarking/egress/internal/cmd/target/target_test.go diff --git a/cmd/benchmarking/egress/internal/cmd/target/target.go b/cmd/benchmarking/egress/internal/cmd/target/target.go new file mode 100644 index 0000000000..1932184105 --- /dev/null +++ b/cmd/benchmarking/egress/internal/cmd/target/target.go @@ -0,0 +1,208 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// Package target is the endpoint every tunnel of the egress scale test leads +// to: the subcommand that answers each request with a small fixed body. +package target + +import ( + "bytes" + "context" + "crypto/tls" + "errors" + "fmt" + "log/slog" + "net" + "net/http" + "strconv" + "strings" + "sync" + "time" + + "github.com/spf13/cobra" +) + +// targetOptions are the flags of the target subcommand. +type targetOptions struct { + listen, listenPorts string + tlsListen, tlsListenPorts string + tlsCertFile, tlsKeyFile string + responseBytes int +} + +// Command is the endpoint the benchmark-egress actors call: every request +// gets a 200 with a small fixed body, over plain HTTP and, when given a +// certificate, HTTPS. One Deployment of it sits behind all of the test's +// Services. --listen-ports and --tls-listen-ports give each Service its own +// backend port: Services on distinct ClusterIPs that share one backend port +// can hand the target two connections with the same 4-tuple. +func Command() *cobra.Command { + var o targetOptions + cmd := &cobra.Command{ + Use: "target", + Short: "Answer every request the actors' tunnels carry with a small fixed body", + Args: cobra.NoArgs, + RunE: func(cmd *cobra.Command, _ []string) error { + return runTarget(cmd.Context(), o) + }, + } + cmd.Flags().StringVar(&o.listen, "listen", ":8080", "Address to serve plain HTTP on.") + cmd.Flags().StringVar(&o.listenPorts, "listen-ports", "", "Also serve plain HTTP on each port of this range, A-B.") + cmd.Flags().StringVar(&o.tlsListen, "tls-listen", "", "Address to serve HTTPS on; empty serves no HTTPS.") + cmd.Flags().StringVar(&o.tlsListenPorts, "tls-listen-ports", "", "Also serve HTTPS on each port of this range, A-B.") + cmd.Flags().StringVar(&o.tlsCertFile, "tls-cert-file", "", "PEM certificate to serve HTTPS with.") + cmd.Flags().StringVar(&o.tlsKeyFile, "tls-key-file", "", "PEM private key of --tls-cert-file.") + cmd.Flags().IntVar(&o.responseBytes, "response-bytes", 16, "Size of the body every request gets back.") + return cmd +} + +func newHandler(responseBytes int) http.Handler { + body := bytes.Repeat([]byte("x"), responseBytes) + mux := http.NewServeMux() + mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, _ *http.Request) { + w.Write([]byte("ok\n")) + }) + mux.HandleFunc("GET /", func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "text/plain") + w.Write(body) + }) + return mux +} + +// newTLSConfig serves the certificate in certFile with the key in keyFile. +func newTLSConfig(certFile, keyFile string) (*tls.Config, error) { + cert, err := tls.LoadX509KeyPair(certFile, keyFile) + if err != nil { + return nil, fmt.Errorf("loading TLS certificate: %w", err) + } + return &tls.Config{Certificates: []tls.Certificate{cert}, MinVersion: tls.VersionTLS12}, nil +} + +// portRange parses "A-B" or "A" into the ports A through B; "" is none. +func portRange(s string) ([]int, error) { + if s == "" { + return nil, nil + } + lo, hi, found := strings.Cut(s, "-") + if !found { + hi = lo + } + a, errA := strconv.Atoi(lo) + b, errB := strconv.Atoi(hi) + if errA != nil || errB != nil || a < 1 || b > 65535 || a > b { + return nil, fmt.Errorf("%q is not a port or a port range A-B", s) + } + ports := make([]int, 0, b-a+1) + for p := a; p <= b; p++ { + ports = append(ports, p) + } + return ports, nil +} + +// openListeners opens addr, when set, and every port in ports on all interfaces. +// On an error it closes what it opened. +func openListeners(addr, ports string) ([]net.Listener, error) { + addrs, err := listenAddrs(addr, ports) + if err != nil { + return nil, err + } + return listenAll(addrs) +} + +// listenAddrs is addr, when set, followed by every port of ports on the +// wildcard address. +func listenAddrs(addr, ports string) ([]string, error) { + ps, err := portRange(ports) + if err != nil { + return nil, err + } + var addrs []string + if addr != "" { + addrs = append(addrs, addr) + } + for _, p := range ps { + addrs = append(addrs, ":"+strconv.Itoa(p)) + } + return addrs, nil +} + +// listenAll listens on every address, or on none. +func listenAll(addrs []string) ([]net.Listener, error) { + lns := make([]net.Listener, 0, len(addrs)) + for _, a := range addrs { + ln, err := net.Listen("tcp", a) + if err != nil { + for _, l := range lns { + l.Close() + } + return nil, fmt.Errorf("listening on %s: %w", a, err) + } + lns = append(lns, ln) + } + return lns, nil +} + +// runTarget serves until ctx ends or a listener fails. +func runTarget(ctx context.Context, o targetOptions) error { + handler := newHandler(o.responseBytes) + plain := &http.Server{Handler: handler, ReadHeaderTimeout: 10 * time.Second} + plainLns, err := openListeners(o.listen, o.listenPorts) + if err != nil { + return fmt.Errorf("opening HTTP listeners: %w", err) + } + servers := []*http.Server{plain} + var tlsSrv *http.Server + var tlsLns []net.Listener + if o.tlsListen != "" || o.tlsListenPorts != "" { + cfg, err := newTLSConfig(o.tlsCertFile, o.tlsKeyFile) + if err != nil { + return err + } + tlsSrv = &http.Server{Handler: handler, TLSConfig: cfg, ReadHeaderTimeout: 10 * time.Second} + if tlsLns, err = openListeners(o.tlsListen, o.tlsListenPorts); err != nil { + return fmt.Errorf("opening HTTPS listeners: %w", err) + } + servers = append(servers, tlsSrv) + } + slog.Info("benchmark-egress target serving", "http_ports", len(plainLns), "https_ports", len(tlsLns)) + + // The first listener to fail stops them all, as the signal does. + failed := make(chan error, len(plainLns)+len(tlsLns)) + var wg sync.WaitGroup + serve := func(ln net.Listener, run func(net.Listener) error) { + wg.Go(func() { + if err := run(ln); !errors.Is(err, http.ErrServerClosed) { + failed <- fmt.Errorf("serving on %s: %w", ln.Addr(), err) + } + }) + } + for _, ln := range plainLns { + serve(ln, plain.Serve) + } + for _, ln := range tlsLns { + serve(ln, func(ln net.Listener) error { return tlsSrv.ServeTLS(ln, "", "") }) + } + + select { + case <-ctx.Done(): + case err = <-failed: + } + shutdownCtx, cancelShutdown := context.WithTimeout(context.Background(), 5*time.Second) + defer cancelShutdown() + for _, srv := range servers { + _ = srv.Shutdown(shutdownCtx) + } + wg.Wait() + return err +} diff --git a/cmd/benchmarking/egress/internal/cmd/target/target_test.go b/cmd/benchmarking/egress/internal/cmd/target/target_test.go new file mode 100644 index 0000000000..a1e44911a7 --- /dev/null +++ b/cmd/benchmarking/egress/internal/cmd/target/target_test.go @@ -0,0 +1,159 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package target + +import ( + "context" + "crypto/tls" + "crypto/x509" + "errors" + "io" + "net" + "net/http" + "net/http/httptest" + "path/filepath" + "slices" + "testing" + "time" + + "go.ate.dev/substrate/cmd/benchmarking/egress/internal/contract" + "go.ate.dev/substrate/cmd/benchmarking/egress/internal/targetcert" +) + +func TestHandler(t *testing.T) { + t.Parallel() + h := newHandler(32) + for _, tc := range []struct { + path string + wantBody int + }{ + {"/", 32}, + {"/anything", 32}, + {"/healthz", len("ok\n")}, + } { + t.Run(tc.path, func(t *testing.T) { + t.Parallel() + rec := httptest.NewRecorder() + h.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, tc.path, nil)) + if rec.Code != http.StatusOK || rec.Body.Len() != tc.wantBody { + t.Errorf("GET %s = %d with %d bytes, want 200 with %d", tc.path, rec.Code, rec.Body.Len(), tc.wantBody) + } + }) + } +} + +func TestHandlerOverTLS(t *testing.T) { + t.Parallel() + bundle, err := targetcert.Generate(time.Now()) + if err != nil { + t.Fatal(err) + } + dir := t.TempDir() + if err := bundle.Write(dir); err != nil { + t.Fatal(err) + } + cfg, err := newTLSConfig(filepath.Join(dir, "tls.crt"), filepath.Join(dir, "tls.key")) + if err != nil { + t.Fatal(err) + } + srv := httptest.NewUnstartedServer(newHandler(32)) + srv.TLS = cfg + srv.StartTLS() + t.Cleanup(srv.Close) + + roots := x509.NewCertPool() + if !roots.AppendCertsFromPEM(bundle.CA) { + t.Fatal("bundle CA does not parse") + } + // Dial the test server whatever the URL names, so the name can be an + // endpoint's while the TLS check runs for real. + client := &http.Client{Transport: &http.Transport{ + TLSClientConfig: &tls.Config{RootCAs: roots}, + DialContext: func(ctx context.Context, network, _ string) (net.Conn, error) { + return (&net.Dialer{}).DialContext(ctx, network, srv.Listener.Addr().String()) + }, + }} + + resp, err := client.Get(contract.EndpointURL(42, contract.SchemeHTTPS)) + if err != nil { + t.Fatalf("GET over TLS as an endpoint name: %v", err) + } + body, _ := io.ReadAll(resp.Body) + resp.Body.Close() + if resp.StatusCode != http.StatusOK || len(body) != 32 { + t.Errorf("GET over TLS = %d with %d bytes, want 200 with 32", resp.StatusCode, len(body)) + } + + _, err = client.Get("https://example.com/") + var hostErr x509.HostnameError + if !errors.As(err, &hostErr) { + t.Errorf("GET over TLS as another name = %v, want a hostname mismatch", err) + } +} + +func TestPortRange(t *testing.T) { + t.Parallel() + for _, tc := range []struct { + in string + want []int + wantErr bool + }{ + {"", nil, false}, + {"8080", []int{8080}, false}, + {"10000-10002", []int{10000, 10001, 10002}, false}, + {"10002-10000", nil, true}, + {"0-3", nil, true}, + {"65535-65536", nil, true}, + {"a-b", nil, true}, + {"10000-", nil, true}, + } { + t.Run(tc.in, func(t *testing.T) { + t.Parallel() + got, err := portRange(tc.in) + if (err != nil) != tc.wantErr || !slices.Equal(got, tc.want) { + t.Errorf("portRange(%q) = %v, %v; want %v, error %v", tc.in, got, err, tc.want, tc.wantErr) + } + }) + } +} + +func TestListenAddrs(t *testing.T) { + t.Parallel() + got, err := listenAddrs("127.0.0.1:0", "10000-10002") + if err != nil { + t.Fatal(err) + } + if want := []string{"127.0.0.1:0", ":10000", ":10001", ":10002"}; !slices.Equal(got, want) { + t.Errorf("listenAddrs() = %v, want %v", got, want) + } + if _, err := listenAddrs("", "a-b"); err == nil { + t.Error("listenAddrs accepted a bad port range") + } +} + +func TestListenAllClosesWhatItOpenedOnFailure(t *testing.T) { + t.Parallel() + held, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { held.Close() }) + if lns, err := listenAll([]string{"127.0.0.1:0", held.Addr().String()}); err == nil { + for _, ln := range lns { + ln.Close() + } + t.Fatal("listenAll bound a port that was taken") + } +} diff --git a/cmd/benchmarking/egress/main.go b/cmd/benchmarking/egress/main.go index 044276fb79..fa3f068bb6 100644 --- a/cmd/benchmarking/egress/main.go +++ b/cmd/benchmarking/egress/main.go @@ -28,6 +28,7 @@ import ( "github.com/spf13/cobra" "go.ate.dev/substrate/cmd/benchmarking/egress/internal/cmd/actor" + "go.ate.dev/substrate/cmd/benchmarking/egress/internal/cmd/target" ) func main() { @@ -44,7 +45,7 @@ happened; the target is the endpoint every tunnel leads to.`, // Nobody types these commands often enough to want shell completion. CompletionOptions: cobra.CompletionOptions{DisableDefaultCmd: true}, } - root.AddCommand(actor.Command()) + root.AddCommand(actor.Command(), target.Command()) ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM) defer stop() From 2276b937eaf9583443fbe9288ba35f77feabaadc Mon Sep 17 00:00:00 2001 From: Yuan Gao Date: Thu, 22 Oct 2026 10:15:40 -0700 Subject: [PATCH 6/6] benchmark-egress: export the actor's measurements over OTLP The actor's instruments stayed inside the process, so a driver could poll the counters and nothing else. With OTEL_EXPORTER_OTLP_ENDPOINT set, the Recorder's MeterProvider now pushes every instrument over OTLP, cumulative, at the interval the environment names, through the same gateway the actor measures, to the collector the benchmark deploys; POST /stop flushes the last export before it answers. /start names the actor, the cell and the run, attributes on every point so the fleet's series are told apart once Prometheus holds them, and each /start draws a new service.instance.id so a restarted actor never continues the old series. The README gains the metric names, the export environment and the policy rule the push needs, the queries a run computes its results from, and the limits of reading through a scrape. The tests gain an in-process OTLP receiver that checks what arrives. --- cmd/benchmarking/egress/README.md | 176 +++++++++-- .../egress/internal/cmd/actor/actor_test.go | 10 +- .../egress/internal/cmd/actor/export_test.go | 276 ++++++++++++++++++ .../egress/internal/cmd/actor/recorder.go | 59 +++- .../internal/cmd/actor/recorder_test.go | 195 ++++++++++++- .../egress/internal/contract/api.go | 11 + .../egress/internal/contract/api_test.go | 39 ++- 7 files changed, 716 insertions(+), 50 deletions(-) create mode 100644 cmd/benchmarking/egress/internal/cmd/actor/export_test.go diff --git a/cmd/benchmarking/egress/README.md b/cmd/benchmarking/egress/README.md index 9cbf1e2cd6..d4426cec2f 100644 --- a/cmd/benchmarking/egress/README.md +++ b/cmd/benchmarking/egress/README.md @@ -111,7 +111,7 @@ idle again with nothing kept. | `GET /readyz` | 200 | 200 | | `POST /start` | 204 | 409 | | `GET /stats` | 404 | 200, section 4 | -| `POST /stop` | 200 with the last loop's final counters, 404 if none | 200 with the final counters once every goroutine has exited | +| `POST /stop` | 200 with the last loop's final counters, 404 if none | 200 with the final counters once every goroutine has exited and the last export is out | Every answer carries `Benchmark-Egress-Protocol: 1`; the driver requires that value, and the header is what lets a later contract be refused. @@ -125,16 +125,22 @@ value, and the header is what lets a later contract be refused. | `offsetMs` | Delay before endpoint 0 opens; endpoint *i* opens `offsetMs` + *i* × `intervalMs` after `/start`. Lets a driver give each actor its own phase, so a fleet started at once does not send in lockstep; see below for how a driver picks it. | Negative, or more than C × `intervalMs`. | | `scheme` | `http` or `https`; empty means `http`. | Any other value. | | `requestTimeoutMs` | Bound on one request, handshake included; zero means 5000. | Negative. | +| `actor` | This actor's name, `/`. An attribute on every metric the loop exports; the fleet's series are told apart by it. | Missing. | +| `cell` | The cell this loop belongs to. An attribute on every metric. | Missing. | +| `run` | The run the cell belongs to. An attribute on every metric; a rerun of a cell gets a new `run`. | Missing. | ```json -{"endpoints": 1000, "intervalMs": 100, "scheme": "https"} +{"endpoints": 1000, "intervalMs": 100, "scheme": "https", + "actor": "benchmark-workloads/egress-0017", "cell": "1000x1000-https", "run": "20261022-1"} ``` 204 means the goroutines exist, not that any request has been sent. A 400 -carries a one-line reason. `/start` does not probe the endpoints: a missing -egress policy or Service shows up as errors in `/stats`, not as a failed -`/start`. The instruments are created at `/start` and released at `/stop`, so -every counter counts from `/start` and a later `/start` begins from zero. +carries a one-line reason. `/start` does not probe the endpoints or the +collector: a missing egress policy or Service shows up as errors in `/stats`, +not as a failed `/start`, and a collector that cannot be reached shows up as +absent series in Prometheus. The instruments are created at `/start` and +released at `/stop`, so every counter counts from `/start` and a later `/start` +begins from zero under a new instance id. The loop then runs one goroutine per endpoint: @@ -147,10 +153,11 @@ The loop then runs one goroutine per endpoint: one request is in flight per endpoint. A request that outlasts the period delays only its own endpoint, whose next request starts as soon as the slow one returns. -4. Stop on `POST /stop`, which waits for every goroutine and answers with the - final counters of section 4. A request cut off by the stop is not counted. - The actor keeps those counters until the next `/start`, so a `/stop` whose - answer was lost is asked again and answered again. +4. Stop on `POST /stop`, which waits for every goroutine, flushes the last + export, and answers with the final counters of section 4. A request cut off + by the stop is not counted. The actor keeps those counters until the next + `/start`, so a `/stop` whose answer was lost is asked again and answered + again. The client keeps one connection per endpoint and never closes it for idling, so a redial, one more in `newConns`, means the gateway cut the tunnel or a @@ -177,11 +184,14 @@ and holds C tunnels. ## 4. What the actor reports -`GET /stats` answers one small JSON object, the counters cumulative since -`/start`, for a driver's gates and progress: poll it, it consumes nothing. The -actor records the same counts and the four latency distributions as OTel -instruments on a private MeterProvider; they are the measurement, the counters -are the view a driver polls. +The actor reports on two paths. `GET /stats` answers one small JSON object, +the counters cumulative since `/start`, for a driver's gates and progress: +poll it, it consumes nothing. Everything a result is computed from, the same +counters and the four latency distributions, the actor exports as +OpenTelemetry metrics over OTLP to the collector, once per export interval, and +the driver reads them from Prometheus. The actor keeps no window state: it has +no cut to serve and nothing to replay, so a lost export costs one interval of +staleness and nothing else. ### 4.1 `GET /stats` and `POST /stop` @@ -215,6 +225,79 @@ more. } ``` +### 4.2 Metrics + +The loop records with these instruments. Names are chosen so that the +collector's Prometheus exporter serves exactly the series the queries of +section 5 read: a counter gains `_total`, a unit of seconds gains `_seconds`, +a histogram gains `_bucket`, `_sum` and `_count`. + +| Instrument | Kind, unit | Attributes | Prometheus series | What it holds | +|---|---|---|---|---| +| `requests` | counter | `result` = `success` or `error` | `requests_total` | Requests finished, by outcome. | +| `request_errors` | counter | `class` | `request_errors_total` | Failures by class, as `errors` in `/stats`. | +| `connections_opened` | counter | | `connections_opened_total` | Requests sent on a freshly dialed connection, as `newConns`. | +| `endpoints_opened` | counter | | `endpoints_opened_total` | Endpoints whose first request completed, as `opened`. | +| `rounds` | gauge | | `rounds` | Requests completed by the least-visited endpoint. | +| `opening_done` | gauge, s | | `opening_done_seconds` | Seconds from `/start` to the last open; zero until then. | +| `longest_gap` | gauge, s | | `longest_gap_seconds` | Longest gap between two requests on one endpoint. | +| `opening` | histogram, s | | `opening_seconds_*` | Successful first requests: tunnel setup latency. | +| `latency` | histogram, s | | `latency_seconds_*` | Successful repeat requests: the settled quantiles, no handshakes in them. | +| `dns`, `tls` | histogram, s | | `dns_seconds_*`, `tls_seconds_*` | Lookup and handshake times, when a request did one. Both stop moving once every tunnel is up. | + +Every point carries `actor`, `cell` and `run` from `/start`. The resource is +`service.name` = `benchmark-egress-actor`, the actor's default unless +`OTEL_RESOURCE_ATTRIBUTES` sets another, which the exporter writes as `job`, +and `service.instance.id`, which the actor draws at `/start`, so a restarted +actor is a new `instance` and never continues the old series. The per-endpoint +rows are not exported: at C = 1024 across a thousand actors they would be three +million series. They stay on `/stats?endpoints=1` for a person debugging one +actor. + +The histograms share one set of bounds, 20 per decade from 1 µs to 63.1 s, +157 values, the same in every actor and every image, so `sum by (le)` across +the fleet is meaningful. Neighboring bounds differ by about 12%, which sets the +resolution: a quantile is within one bucket step of the true value, and two +runs whose quantiles fall within one step are indistinguishable. There is no +exact maximum; the overflow bucket is the last. + +One actor exports 646 series, about 90 KB of protobuf per interval. A fleet of +a thousand at the default interval is 646 thousand series and 130 thousand +points per second into the collector, and one more CONNECT tunnel per actor +through the gateway. + +### 4.3 Export + +The actor is configured by the standard OpenTelemetry environment, set on the +ActorTemplate: + +| Variable | Value | Meaning | +|---|---|---| +| `OTEL_EXPORTER_OTLP_ENDPOINT` | `http://:4317` | Where to push, OTLP over gRPC, plaintext. Unset means the loop exports nothing and `/start` still works. | +| `OTEL_RESOURCE_ATTRIBUTES` | `service.name=benchmark-egress-actor` | The `job` the series are stored under. Optional: this value is the actor's default, and the variable overrides it. | +| `OTEL_METRIC_EXPORT_INTERVAL` | `5000` | Milliseconds between exports; the default of the SDK is 60000, too coarse for a window of a few minutes. | +| `OTEL_METRIC_EXPORT_TIMEOUT` | `5000` | A failed export must not delay the next. | + +Temporality is cumulative, the SDK's default, which is what the collector's +Prometheus exporter expects. The push goes through the egress gateway like +every other byte the actor sends, so the actor's egress policy needs one more +rule, cleartext HTTP to the collector on its port: + +```yaml +- http: + hostnames: + - telemetry-meter.benchmarking.svc.cluster.local + - telemetry-meter.benchmarking.svc + - telemetry-meter.benchmarking + ports: + numbers: [4317] +``` + +Both the full name and the short forms are needed: the sandbox's resolver +completes the short name to the full one, which the DNS relay forwards only +when a rule names it, while the gateway matches the request's authority, which +is the short name the SDK dialed. A rule without `ports` admits port 80 only. + ## 5. What a run aggregates The driver is the next PR. This section is the contract it implements. @@ -224,7 +307,34 @@ instants: `t0`, the first `/start`; `t1`, when every actor's `opened` is C, the end of the blended window; `t2`, when every actor's `rounds` is at least two and `newConns` is flat between two polls, the start of the settled window; `t3`, `t2` plus the cell's duration, when it posts `/stop`. Every result is a -difference between two instants. +difference between two instants, read from Prometheus with the instant query +API, `time=` set to the later instant and the earlier one as an `@` anchor, so +the query is about the instants and not about when the driver asked. The +driver reads at `t` plus a lag of one export interval, one scrape interval and +a margin, so the samples at `t` exist. + +Series selector `S` is `{job="benchmark-egress-actor", cell="", run=""}`. +The merge across actors happens in the query: + +| Rule | Series | +|---|---| +| `sum` of `@`-differences | `requests_total`, `request_errors_total`, `connections_opened_total`, `endpoints_opened_total`, `latency_seconds_count`, `latency_seconds_sum` | +| `sum by (le)` of `@`-differences, then `histogram_quantile` | `latency_seconds_bucket`, `opening_seconds_bucket` | +| `max` | `opening_done_seconds`, `longest_gap_seconds` | +| `min` | `rounds` | +| `count by (actor)` | presence: every actor's series exist at both instants | + +| Result | Query at `time=t_end` | +|---|---| +| Requests in a window | `sum(requests_total S) - (sum(requests_total S @ t_start) or vector(0))` | +| Success rate | the same pair with `result="success"`, over the total | +| Settled p50, p99, p99.9 | `histogram_quantile(q, sum by (le) (latency_seconds_bucket S) - (sum by (le) (latency_seconds_bucket S @ t_start) or sum by (le) (latency_seconds_bucket S) * 0))` | +| Blended end | per actor, its `/start` time plus `opening_done_seconds`; the latest is `t1` | +| Gateway CPU in a window | `sum(container_cpu_usage_seconds_total{pod=~"atenet-egress.*", container=~"envoy|ext-proc|sdsmint"})` as the same `@`-difference, from the cAdvisor scrape of the gateway's node | +| Presence | `count(count by (actor) (requests_total S)) == A` at `t2` and `t3` | + +`rate` and `increase` are not used for results: they extrapolate over the +range and lose the first sample. They serve only the collector's own counters. The results of a cell come from two windows. The blended window runs from the first `/start` to the latest `openingDone`, each actor's shifted by its own @@ -270,16 +380,34 @@ rather than two every 100 ms; without it every send of actor 2 would coincide with one of actor 1's. The gateway's CPU in that window paid for four tunnel setups and for every request in it; the settled rate prices every request, the first on each tunnel included, and what is left is the setups. `t1` is the -edge the driver reads the gateway's CPU against, and the requests completed -over `t0..t1` are the window's request count; `requestsAtOpeningDone`, summed -from `/stats`, is the count at each actor's own opening end, for the opening -alone. +edge the driver reads the gateway's CPU against, and the `@`-difference of +`requests_total` over `t0..t1` is the window's request count; +`requestsAtOpeningDone`, summed from `/stats`, is the count at each actor's +own opening end, for the opening alone. | Result | Source | How it is computed | |---|---|---| -| Settled req/s per gateway core | actors + gateway | Requests completed in the settled window divided by the gateway pod's CPU-seconds in the same window, `container_cpu_usage_seconds_total` of the envoy, ext-proc and sdsmint containers over `t2..t3`. | +| Settled req/s per gateway core | actors + gateway | Requests completed in the settled window divided by the gateway pod's CPU-seconds in the same window, both as `@`-differences over `t2..t3` from the same Prometheus. | | Gateway CPU per tunnel opened | actors + gateway | The blended window's gateway CPU, minus every request in it priced at the settled per-request cost, divided by the tunnels opened. What remains is setup alone, independent of pacing, on the assumption that a request on an open tunnel costs the same in either window. Computed after the settled window, since it needs the settled rate. | | Envoy heap | gateway | The maximum of `server.memory_allocated` in the cell; per-tunnel cost is the slope across fleet sizes. | -| Settled p50, p99, p99.9 | actors | The settled window's `latency` histogram, to one bucket step. | -| Success rate | actors | `successes` over `requests` in the window. | -| Run validity | actors + gateway | `newConns` equals `opened` and stays flat; `longestGap` stays under the stream idle timeout; `dns` and `tls` do not move over `t2..t3`; `cluster.mitm_internal.upstream_cx_total` equals A × C and stays flat; `http.egress_connect.downstream_rq_idle_timeout`, `downstream_cx_idle_timeout` and `downstream_cx_protocol_error` do not move. | +| Settled p50, p99, p99.9 | actors | `histogram_quantile` on the settled window's bucket differences, reported as interpolated with a bucket resolution of 12%. | +| Success rate | actors | `result="success"` over all `requests_total` in the window. | +| Run validity | actors + gateway | `endpoints_opened_total` equals A × C and `connections_opened_total` does not move over `t2..t3` (a first request that failed counts its endpoint as opened and dials its tunnel one period later, so the two counters need not be equal); `max(longest_gap_seconds)` stays under the stream idle timeout; `dns_seconds_count` and `tls_seconds_count` do not move over `t2..t3`; presence is A at `t2` and `t3`; `cluster.mitm_internal.upstream_cx_total` equals A × (C + 1) and stays flat; `http.egress_connect.downstream_rq_idle_timeout`, `downstream_cx_idle_timeout` and `downstream_cx_protocol_error` do not move; the collector's `otelcol_receiver_refused_metric_points` does not move. | + +Three limits of this design are accepted and stated here rather than worked +around: + +- **Instant reads lag.** A sample at `t` holds what the actor exported up to + one export interval before `t`, scraped up to one scrape interval later. In + a difference between two instants of a steady window the lag is the same at + both ends and cancels; measured on GKE, a 300 s settled window agreed with + exact cuts within 0.1% of requests. Where the rate changes between the + instants, at the start, the stop, or a restart, the error is up to export + plus scrape interval per end. +- **A restart inside a window invalidates the cell.** A `requests` that went + down between two `/stats` polls, or a 404, means the actor restarted. Its new + instance starts from zero under the same `actor`; the old instance's series + stop and expire from the collector after its expiration time. The driver + marks the cell invalid rather than reasoning about the overlap. +- **No maximum.** The histogram carries no exact maximum; the largest latency + is known to one bucket step. diff --git a/cmd/benchmarking/egress/internal/cmd/actor/actor_test.go b/cmd/benchmarking/egress/internal/cmd/actor/actor_test.go index ff22e57edd..4d485c4b0e 100644 --- a/cmd/benchmarking/egress/internal/cmd/actor/actor_test.go +++ b/cmd/benchmarking/egress/internal/cmd/actor/actor_test.go @@ -238,10 +238,10 @@ const ( stopPath = contract.StopRoute + "?endpoints=1" ) -// start posts /start with req. +// start posts /start with req, identified. func start(t *testing.T, h http.Handler, req contract.StartRequest) *httptest.ResponseRecorder { t.Helper() - return post(t, h, contract.StartRoute, req) + return post(t, h, contract.StartRoute, identified(req)) } // readStats polls the counters. @@ -447,10 +447,12 @@ func TestStartStopLifecycle(t *testing.T) { func TestStartRejectsBadRequests(t *testing.T) { t.Parallel() h := newServer().handler() + const named = `"actor":"benchmark-workloads/egress-0","cell":"cell-1","run":"run-1"` for _, tc := range []struct{ name, body string }{ {"not json", `not json`}, {"empty object", `{}`}, - {"invalid", `{"endpoints":1,"scheme":"ftp"}`}, + {"invalid", `{"endpoints":1,"scheme":"ftp",` + named + `}`}, + {"no identity", `{"endpoints":1}`}, } { t.Run(tc.name, func(t *testing.T) { t.Parallel() @@ -1277,7 +1279,7 @@ func TestEveryRouteCarriesProtocolHeader(t *testing.T) { func TestPayloadBudget(t *testing.T) { t.Parallel() const c = contract.MaxEndpoints - run, err := newLoopRun(context.Background(), contract.StartRequest{Endpoints: c, Scheme: contract.SchemeHTTPS}, nil) + run, err := newLoopRun(context.Background(), identified(contract.StartRequest{Endpoints: c, Scheme: contract.SchemeHTTPS}), nil) if err != nil { t.Fatal(err) } diff --git a/cmd/benchmarking/egress/internal/cmd/actor/export_test.go b/cmd/benchmarking/egress/internal/cmd/actor/export_test.go new file mode 100644 index 0000000000..e2a243fc35 --- /dev/null +++ b/cmd/benchmarking/egress/internal/cmd/actor/export_test.go @@ -0,0 +1,276 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package actor + +import ( + "context" + "crypto/ecdsa" + "crypto/elliptic" + "crypto/rand" + "crypto/tls" + "crypto/x509" + "crypto/x509/pkix" + "maps" + "math/big" + "net" + "net/http" + "net/http/httptest" + "net/url" + "slices" + "sync" + "testing" + "time" + + collectormetrics "go.opentelemetry.io/proto/otlp/collector/metrics/v1" + commonpb "go.opentelemetry.io/proto/otlp/common/v1" + metricspb "go.opentelemetry.io/proto/otlp/metrics/v1" + "google.golang.org/grpc" + + "go.ate.dev/substrate/cmd/benchmarking/egress/internal/contract" +) + +// fakeCollector is an OTLP/gRPC metrics receiver on a loopback listener +// that keeps every export it was sent. +type fakeCollector struct { + collectormetrics.UnimplementedMetricsServiceServer + addr string + mu sync.Mutex + exports []*collectormetrics.ExportMetricsServiceRequest +} + +func (f *fakeCollector) Export(_ context.Context, req *collectormetrics.ExportMetricsServiceRequest) (*collectormetrics.ExportMetricsServiceResponse, error) { + f.mu.Lock() + defer f.mu.Unlock() + f.exports = append(f.exports, req) + return &collectormetrics.ExportMetricsServiceResponse{}, nil +} + +// received returns the exports so far. +func (f *fakeCollector) received() []*collectormetrics.ExportMetricsServiceRequest { + f.mu.Lock() + defer f.mu.Unlock() + return slices.Clone(f.exports) +} + +func startFakeCollector(t *testing.T) *fakeCollector { + t.Helper() + lis, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + f := &fakeCollector{addr: lis.Addr().String()} + srv := grpc.NewServer() + collectormetrics.RegisterMetricsServiceServer(srv, f) + go srv.Serve(lis) + t.Cleanup(srv.Stop) + return f +} + +// exportEnv points the SDK at f with a short interval, as the template's +// environment points the actor at the collector. The environment is process +// wide, so a test that calls it cannot be parallel. +func exportEnv(t *testing.T, f *fakeCollector) { + t.Helper() + t.Setenv(endpointEnv, "http://"+f.addr) + t.Setenv("OTEL_METRIC_EXPORT_INTERVAL", "100") + t.Setenv("OTEL_METRIC_EXPORT_TIMEOUT", "2000") + t.Setenv("OTEL_RESOURCE_ATTRIBUTES", "service.name="+serviceName) +} + +// attr returns the string attribute key of kvs, "" when absent. +func attr(kvs []*commonpb.KeyValue, key string) string { + for _, kv := range kvs { + if kv.Key == key { + return kv.Value.GetStringValue() + } + } + return "" +} + +// metricsOf flattens one export into its metrics by name, checking the +// resource and the scope on the way. +func metricsOf(t *testing.T, req *collectormetrics.ExportMetricsServiceRequest) map[string]*metricspb.Metric { + t.Helper() + out := map[string]*metricspb.Metric{} + for _, rm := range req.ResourceMetrics { + if name := attr(rm.Resource.Attributes, "service.name"); name != serviceName { + t.Errorf("resource service.name = %q, want %q", name, serviceName) + } + if attr(rm.Resource.Attributes, "service.instance.id") == "" { + t.Errorf("resource has no service.instance.id: %v", rm.Resource.Attributes) + } + for _, sm := range rm.ScopeMetrics { + if sm.Scope.Name != meterName { + t.Errorf("scope %q, want %q", sm.Scope.Name, meterName) + } + for _, m := range sm.Metrics { + out[m.Name] = m + } + } + } + return out +} + +// instanceOf is the service.instance.id of one export. +func instanceOf(req *collectormetrics.ExportMetricsServiceRequest) string { + if len(req.ResourceMetrics) == 0 { + return "" + } + return attr(req.ResourceMetrics[0].Resource.Attributes, "service.instance.id") +} + +// sumOf adds the integer points of a counter. +func sumOf(m *metricspb.Metric) int64 { + var n int64 + for _, p := range m.GetSum().GetDataPoints() { + n += p.GetAsInt() + } + return n +} + +// localhostTLSTarget is an HTTPS target with a certificate for localhost, so a +// loop that names it does a lookup and a handshake on every open. +func localhostTLSTarget(t *testing.T, h http.HandlerFunc) (host string, cert *x509.Certificate) { + t.Helper() + key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + if err != nil { + t.Fatal(err) + } + tmpl := &x509.Certificate{ + SerialNumber: big.NewInt(1), + Subject: pkix.Name{CommonName: "localhost"}, + DNSNames: []string{"localhost"}, + NotBefore: time.Now().Add(-time.Hour), + NotAfter: time.Now().Add(time.Hour), + KeyUsage: x509.KeyUsageDigitalSignature, + ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth}, + } + der, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, &key.PublicKey, key) + if err != nil { + t.Fatal(err) + } + cert, err = x509.ParseCertificate(der) + if err != nil { + t.Fatal(err) + } + srv := httptest.NewUnstartedServer(h) + srv.TLS = &tls.Config{Certificates: []tls.Certificate{{Certificate: [][]byte{der}, PrivateKey: key, Leaf: cert}}} + srv.Config.ErrorLog = quietLog + srv.StartTLS() + t.Cleanup(srv.Close) + u, err := url.Parse(srv.URL) + if err != nil { + t.Fatal(err) + } + return "localhost:" + u.Port(), cert +} + +// TestExportReachesTheCollector runs a loop against a fake OTLP receiver, as +// the actor runs against the benchmark's collector, and pins what arrives: +// the resource, the eleven names, the three attributes on every point, +// cumulative temporality, and a final export after /stop that agrees with +// the counters /stop answered. +func TestExportReachesTheCollector(t *testing.T) { + f := startFakeCollector(t) + exportEnv(t, f) + okHost, okCert := localhostTLSTarget(t, okHandler) + failHost, failCert := localhostTLSTarget(t, func(w http.ResponseWriter, _ *http.Request) { http.Error(w, "denied", http.StatusServiceUnavailable) }) + roots := x509.NewCertPool() + roots.AddCert(okCert) + roots.AddCert(failCert) + h := serveTLS(roots, okHost, failHost) + + req := contract.StartRequest{Endpoints: 2, Scheme: contract.SchemeHTTPS, IntervalMs: 2} + final := runLoop(t, h, req, func(s *contract.Stats) bool { return s.Rounds >= 3 && s.Errors["HTTP 503"] > 0 }) + exports := f.received() + if len(exports) == 0 { + t.Fatal("the collector received nothing") + } + last := metricsOf(t, exports[len(exports)-1]) + for _, name := range metricNames { + if _, ok := last[name]; !ok { + t.Errorf("the final export lacks %s; it has %v", name, slices.Sorted(maps.Keys(last))) + } + } + for name, m := range last { + var points []*commonpb.KeyValue + var temporality metricspb.AggregationTemporality + switch d := m.Data.(type) { + case *metricspb.Metric_Sum: + temporality = d.Sum.AggregationTemporality + for _, p := range d.Sum.DataPoints { + points = append(points, p.Attributes...) + } + case *metricspb.Metric_Histogram: + temporality = d.Histogram.AggregationTemporality + for _, p := range d.Histogram.DataPoints { + points = append(points, p.Attributes...) + if len(p.ExplicitBounds) != len(contract.Bounds) || len(p.BucketCounts) != len(contract.Bounds)+1 { + t.Errorf("%s point has %d bounds and %d buckets, want %d and %d", name, len(p.ExplicitBounds), len(p.BucketCounts), len(contract.Bounds), len(contract.Bounds)+1) + } + } + case *metricspb.Metric_Gauge: + temporality = metricspb.AggregationTemporality_AGGREGATION_TEMPORALITY_CUMULATIVE + for _, p := range d.Gauge.DataPoints { + points = append(points, p.Attributes...) + } + default: + t.Errorf("%s arrived as %T", name, m.Data) + } + if temporality != metricspb.AggregationTemporality_AGGREGATION_TEMPORALITY_CUMULATIVE { + t.Errorf("%s arrived with temporality %v, want cumulative", name, temporality) + } + if a, c, r := attr(points, "actor"), attr(points, "cell"), attr(points, "run"); a != identity.Actor || c != identity.Cell || r != identity.Run { + t.Errorf("%s points carry actor %q cell %q run %q, want %q %q %q", name, a, c, r, identity.Actor, identity.Cell, identity.Run) + } + } + if got := sumOf(last[requestsName]); got != final.Requests { + t.Errorf("final export: requests %d, /stop answered %d", got, final.Requests) + } + if got := sumOf(last[requestErrorsName]); got != final.Requests-final.Successes { + t.Errorf("final export: request_errors %d, /stop answered %d failures", got, final.Requests-final.Successes) + } + if got := sumOf(last[endpointsOpenedName]); got != final.Opened { + t.Errorf("final export: endpoints_opened %d, /stop answered %d", got, final.Opened) + } + + // A second loop in the same process is a new instance from zero. + first := instanceOf(exports[len(exports)-1]) + before := len(exports) + runLoop(t, h, req, atLeast(1)) + exports = f.received() + if len(exports) <= before { + t.Fatal("the second loop exported nothing") + } + if second := instanceOf(exports[len(exports)-1]); second == first || second == "" { + t.Errorf("second loop exported as instance %q, first as %q; want two ids", second, first) + } +} + +// TestNoEndpointMeansNoExport pins the actor without the template's +// environment: /start works, the loop runs, and nothing is pushed anywhere. +func TestNoEndpointMeansNoExport(t *testing.T) { + f := startFakeCollector(t) + exportEnv(t, f) + t.Setenv(endpointEnv, "") + h := serveTargets(t, 2) + final := runLoop(t, h, contract.StartRequest{Endpoints: 2}, atLeast(5)) + if final.Requests < 5 || final.Successes != final.Requests { + t.Errorf("loop without an endpoint: %d requests, %d successes", final.Requests, final.Successes) + } + if got := f.received(); len(got) != 0 { + t.Errorf("the collector received %d exports with the endpoint env unset", len(got)) + } +} diff --git a/cmd/benchmarking/egress/internal/cmd/actor/recorder.go b/cmd/benchmarking/egress/internal/cmd/actor/recorder.go index 99de7879bd..0448a57c41 100644 --- a/cmd/benchmarking/egress/internal/cmd/actor/recorder.go +++ b/cmd/benchmarking/egress/internal/cmd/actor/recorder.go @@ -21,10 +21,17 @@ import ( "context" "errors" "fmt" + "log/slog" + "os" + "github.com/google/uuid" "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc" "go.opentelemetry.io/otel/metric" sdkmetric "go.opentelemetry.io/otel/sdk/metric" + "go.opentelemetry.io/otel/sdk/metric/metricdata" + "go.opentelemetry.io/otel/sdk/resource" + semconv "go.opentelemetry.io/otel/semconv/v1.40.0" "go.ate.dev/substrate/cmd/benchmarking/egress/internal/contract" ) @@ -43,9 +50,18 @@ const ( longestGapName = "longest_gap" ) +// serviceName is the resource's service.name unless the environment sets +// one; the collector's exporter writes it as the job label. +const serviceName = "benchmark-egress-actor" + +// endpointEnv is the OTLP endpoint the SDK reads; set, the loop exports. +const endpointEnv = "OTEL_EXPORTER_OTLP_ENDPOINT" + // Recorder holds one loop's instruments on a private MeterProvider: the -// counters that mirror its Stats and the four latency histograms. Without a -// reader every method is a cheap no-op. +// counters that mirror its Stats and the four latency histograms, every point +// under the loop's actor, cell and run. With OTEL_EXPORTER_OTLP_ENDPOINT set +// the provider pushes them over OTLP at the interval the environment names; +// without it the provider has no reader and every method is a cheap no-op. type Recorder struct { provider *sdkmetric.MeterProvider hists map[contract.Distribution]metric.Float64Histogram @@ -58,18 +74,39 @@ type Recorder struct { attrs, success, failure metric.MeasurementOption } -// NewRecorder creates the instruments for the loop req starts, on a private -// MeterProvider with readers, which tests use to read the instruments. +// NewRecorder creates the instruments for the loop req starts, with the +// OTLP reader when the environment names an endpoint, plus readers, which +// tests use to read the instruments without an exporter. The resource takes +// service.name from the environment and a fresh service.instance.id, so a +// restarted actor never continues the old series. func NewRecorder(ctx context.Context, req contract.StartRequest, readers ...sdkmetric.Reader) (*Recorder, error) { - opts := make([]sdkmetric.Option, 0, len(readers)) + res, err := resource.New(ctx, + resource.WithAttributes(semconv.ServiceName(serviceName)), + resource.WithFromEnv(), + resource.WithAttributes(semconv.ServiceInstanceID(uuid.NewString())), + ) + if errors.Is(err, resource.ErrPartialResource) { + slog.WarnContext(ctx, "partial telemetry resource", "err", err) + } else if err != nil { + return nil, fmt.Errorf("building the resource: %w", err) + } + opts := []sdkmetric.Option{sdkmetric.WithResource(res)} + if os.Getenv(endpointEnv) != "" { + exp, err := otlpmetricgrpc.New(ctx, otlpmetricgrpc.WithTemporalitySelector(cumulative)) + if err != nil { + return nil, fmt.Errorf("creating the OTLP exporter: %w", err) + } + opts = append(opts, sdkmetric.WithReader(sdkmetric.NewPeriodicReader(exp))) + } for _, rd := range readers { opts = append(opts, sdkmetric.WithReader(rd)) } r := &Recorder{provider: sdkmetric.NewMeterProvider(opts...), hists: make(map[contract.Distribution]metric.Float64Histogram, len(contract.DistributionNames))} + loop := attribute.NewSet(attribute.String("actor", req.Actor), attribute.String("cell", req.Cell), attribute.String("run", req.Run)) withResult := func(result string) metric.MeasurementOption { - return metric.WithAttributeSet(attribute.NewSet(attribute.String("result", result))) + return metric.WithAttributeSet(attribute.NewSet(append(loop.ToSlice(), attribute.String("result", result))...)) } - r.attrs, r.success, r.failure = metric.WithAttributeSet(attribute.NewSet()), withResult("success"), withResult("error") + r.attrs, r.success, r.failure = metric.WithAttributeSet(loop), withResult("success"), withResult("error") if err := r.instruments(r.provider.Meter(meterName)); err != nil { _ = r.provider.Shutdown(context.Background()) return nil, err @@ -77,6 +114,14 @@ func NewRecorder(ctx context.Context, req contract.StartRequest, readers ...sdkm return r, nil } +// cumulative is the SDK's default temporality, set explicitly so that +// OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE in the environment cannot +// switch the exporter to delta, which would break every difference between two +// instants a run computes. +func cumulative(sdkmetric.InstrumentKind) metricdata.Temporality { + return metricdata.CumulativeTemporality +} + // instruments creates the eleven instruments on meter. func (r *Recorder) instruments(meter metric.Meter) error { var err error diff --git a/cmd/benchmarking/egress/internal/cmd/actor/recorder_test.go b/cmd/benchmarking/egress/internal/cmd/actor/recorder_test.go index cba501aa0c..fd065ba997 100644 --- a/cmd/benchmarking/egress/internal/cmd/actor/recorder_test.go +++ b/cmd/benchmarking/egress/internal/cmd/actor/recorder_test.go @@ -16,13 +16,206 @@ package actor import ( "context" + "reflect" + "slices" "testing" + "go.opentelemetry.io/otel/attribute" + sdkmetric "go.opentelemetry.io/otel/sdk/metric" + "go.opentelemetry.io/otel/sdk/metric/metricdata" + "go.ate.dev/substrate/cmd/benchmarking/egress/internal/contract" ) +// identity is the actor, cell and run every test loop starts under. +var identity = contract.StartRequest{Actor: "benchmark-workloads/egress-0", Cell: "cell-1", Run: "run-1"} + +// identified is req with the identity fields filled where a test left them +// empty, so a test states only what it is about. +func identified(req contract.StartRequest) contract.StartRequest { + if req.Actor == "" { + req.Actor = identity.Actor + } + if req.Cell == "" { + req.Cell = identity.Cell + } + if req.Run == "" { + req.Run = identity.Run + } + return req +} + +// loopAttrs are the attributes every point of a loop started with identity +// carries. +var loopAttrs = attribute.NewSet(attribute.String("actor", identity.Actor), attribute.String("cell", identity.Cell), attribute.String("run", identity.Run)) + +// metricNames are the eleven instruments of the contract, as the SDK names +// them before the collector adds its suffixes. +var metricNames = []string{requestsName, requestErrorsName, connsOpenedName, endpointsOpenedName, roundsName, openingDoneName, longestGapName, + string(contract.Opening), string(contract.Latency), string(contract.DNS), string(contract.TLS)} + +// newTestRecorder is a Recorder for identity with a manual reader on it. +func newTestRecorder(t *testing.T) (*Recorder, *sdkmetric.ManualReader) { + t.Helper() + reader := sdkmetric.NewManualReader() + r, err := NewRecorder(context.Background(), identity, reader) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = r.Shutdown(context.Background()) }) + return r, reader +} + +func collect(t *testing.T, reader *sdkmetric.ManualReader) metricdata.ResourceMetrics { + t.Helper() + var rm metricdata.ResourceMetrics + if err := reader.Collect(context.Background(), &rm); err != nil { + t.Fatal(err) + } + return rm +} + +// metricsByName indexes the one scope's metrics, failing on any other scope. +func metricsByName(t *testing.T, rm metricdata.ResourceMetrics) map[string]metricdata.Metrics { + t.Helper() + out := map[string]metricdata.Metrics{} + for _, sm := range rm.ScopeMetrics { + if sm.Scope.Name != meterName { + t.Errorf("scope %q, want only %q", sm.Scope.Name, meterName) + } + for _, m := range sm.Metrics { + out[m.Name] = m + } + } + return out +} + +// recordOneOfEach puts one measurement into every instrument. +func recordOneOfEach(ctx context.Context, r *Recorder) { + r.Request(ctx, true) + r.Request(ctx, false) + r.RequestError(ctx, "timeout") + r.ConnectionOpened(ctx) + r.EndpointOpened(ctx) + r.Gauges(ctx, 2, 1.5, 0.25) + for i, name := range contract.DistributionNames { + r.Record(ctx, name, float64(1+i)*1e-3) + } +} + +// pointAttrs returns the attribute set of every data point of m. +func pointAttrs(t *testing.T, m metricdata.Metrics) []attribute.Set { + t.Helper() + var sets []attribute.Set + switch d := m.Data.(type) { + case metricdata.Sum[int64]: + for _, p := range d.DataPoints { + sets = append(sets, p.Attributes) + } + case metricdata.Gauge[int64]: + for _, p := range d.DataPoints { + sets = append(sets, p.Attributes) + } + case metricdata.Gauge[float64]: + for _, p := range d.DataPoints { + sets = append(sets, p.Attributes) + } + case metricdata.Histogram[float64]: + for _, p := range d.DataPoints { + sets = append(sets, p.Attributes) + } + default: + t.Errorf("%s has data of type %T", m.Name, m.Data) + } + return sets +} + +// TestInstrumentsCarryTheLoopAttributes pins the wire contract of the +// recorder: the eleven names, the units that become _seconds, the shared +// bounds, and actor, cell and run on every point, with result and class +// beside them where the contract says so. +func TestInstrumentsCarryTheLoopAttributes(t *testing.T) { + t.Parallel() + r, reader := newTestRecorder(t) + recordOneOfEach(context.Background(), r) + byName := metricsByName(t, collect(t, reader)) + + var names []string + for name := range byName { + names = append(names, name) + } + slices.Sort(names) + if want := slices.Sorted(slices.Values(metricNames)); !slices.Equal(names, want) { + t.Errorf("instruments = %v, want %v", names, want) + } + for name, m := range byName { + wantUnit := "" + if slices.Contains([]string{openingDoneName, longestGapName}, name) || slices.Contains(contract.DistributionNames, contract.Distribution(name)) { + wantUnit = "s" + } + if m.Unit != wantUnit { + t.Errorf("%s has unit %q, want %q", name, m.Unit, wantUnit) + } + sets := pointAttrs(t, m) + if len(sets) == 0 { + t.Errorf("%s has no data point", name) + } + for _, set := range sets { + for _, kv := range loopAttrs.ToSlice() { + if v, ok := set.Value(kv.Key); !ok || v != kv.Value { + t.Errorf("%s point %v lacks %s=%s", name, set.Encoded(attribute.DefaultEncoder()), kv.Key, kv.Value.AsString()) + } + } + extra := set.Len() - loopAttrs.Len() + switch name { + case requestsName: + if v, ok := set.Value("result"); extra != 1 || !ok || (v.AsString() != "success" && v.AsString() != "error") { + t.Errorf("requests point has %v, want the loop attributes plus result", set.Encoded(attribute.DefaultEncoder())) + } + case requestErrorsName: + if v, ok := set.Value("class"); extra != 1 || !ok || v.AsString() != "timeout" { + t.Errorf("request_errors point has %v, want the loop attributes plus class", set.Encoded(attribute.DefaultEncoder())) + } + default: + if extra != 0 { + t.Errorf("%s point has %v, want the loop attributes alone", name, set.Encoded(attribute.DefaultEncoder())) + } + } + } + } + if sum, ok := byName[requestsName].Data.(metricdata.Sum[int64]); !ok || len(sum.DataPoints) != 2 || !sum.IsMonotonic || sum.Temporality != metricdata.CumulativeTemporality { + t.Errorf("requests = %+v, want a cumulative monotonic sum with a success and an error point", byName[requestsName].Data) + } + if g, ok := byName[roundsName].Data.(metricdata.Gauge[int64]); !ok || g.DataPoints[0].Value != 2 { + t.Errorf("rounds = %+v, want a gauge at 2", byName[roundsName].Data) + } + if g, ok := byName[openingDoneName].Data.(metricdata.Gauge[float64]); !ok || g.DataPoints[0].Value != 1.5 { + t.Errorf("opening_done = %+v, want a gauge at 1.5", byName[openingDoneName].Data) + } + for i, name := range contract.DistributionNames { + h, ok := byName[string(name)].Data.(metricdata.Histogram[float64]) + if !ok || len(h.DataPoints) != 1 { + t.Fatalf("%s = %+v, want a histogram with one point", name, byName[string(name)].Data) + } + p := h.DataPoints[0] + if !reflect.DeepEqual(p.Bounds, contract.Bounds) || p.Count != 1 || p.Sum != float64(1+i)*1e-3 || len(p.BucketCounts) != len(contract.Bounds)+1 { + t.Errorf("%s point: %d bounds, count %d, sum %v; want contract.Bounds, 1, %v", name, len(p.Bounds), p.Count, p.Sum, float64(1+i)*1e-3) + } + } +} + +// TestResourceDefaultsTheServiceName pins the recorder's own service.name, +// the job the series are stored under when the environment sets none. +func TestResourceDefaultsTheServiceName(t *testing.T) { + t.Setenv("OTEL_RESOURCE_ATTRIBUTES", "") + _, reader := newTestRecorder(t) + if name, _ := collect(t, reader).Resource.Set().Value("service.name"); name.AsString() != serviceName { + t.Errorf("service.name without the env = %q, want %q", name.AsString(), serviceName) + } +} + func BenchmarkRecord(b *testing.B) { - r, err := NewRecorder(context.Background(), contract.StartRequest{}) + r, err := NewRecorder(context.Background(), identity) if err != nil { b.Fatal(err) } diff --git a/cmd/benchmarking/egress/internal/contract/api.go b/cmd/benchmarking/egress/internal/contract/api.go index 59611fc9e7..c45df1dcd4 100644 --- a/cmd/benchmarking/egress/internal/contract/api.go +++ b/cmd/benchmarking/egress/internal/contract/api.go @@ -77,6 +77,12 @@ type StartRequest struct { OffsetMs int64 `json:"offsetMs,omitempty"` // Scheme is SchemeHTTP or SchemeHTTPS; empty means SchemeHTTP. Scheme string `json:"scheme,omitempty"` + // Actor, Cell and Run name this loop: the actor as /, the + // cell it runs in, and the run the cell belongs to. All three are + // attributes on every metric the loop exports and all three are required. + Actor string `json:"actor"` + Cell string `json:"cell"` + Run string `json:"run"` } // URLScheme is the scheme the loop requests endpoints over. @@ -107,5 +113,10 @@ func (r *StartRequest) Validate() error { if r.Scheme != "" && r.Scheme != SchemeHTTP && r.Scheme != SchemeHTTPS { return fmt.Errorf("scheme must be %s or %s, got %q", SchemeHTTP, SchemeHTTPS, r.Scheme) } + for _, f := range []struct{ name, value string }{{"actor", r.Actor}, {"cell", r.Cell}, {"run", r.Run}} { + if f.value == "" { + return fmt.Errorf("%s is required", f.name) + } + } return nil } diff --git a/cmd/benchmarking/egress/internal/contract/api_test.go b/cmd/benchmarking/egress/internal/contract/api_test.go index e833923002..d367d1fad0 100644 --- a/cmd/benchmarking/egress/internal/contract/api_test.go +++ b/cmd/benchmarking/egress/internal/contract/api_test.go @@ -16,6 +16,13 @@ package contract import "testing" +// named is a StartRequest with the three identity fields filled, so a case +// states only what it is about. +func named(r StartRequest) StartRequest { + r.Actor, r.Cell, r.Run = "benchmark-workloads/egress-0", "cell-1", "run-1" + return r +} + func TestStartRequestValidate(t *testing.T) { t.Parallel() for _, tc := range []struct { @@ -23,20 +30,24 @@ func TestStartRequestValidate(t *testing.T) { req StartRequest wantErr bool }{ - {"ok", StartRequest{Endpoints: 10}, false}, - {"http", StartRequest{Endpoints: 10, Scheme: SchemeHTTP}, false}, - {"https", StartRequest{Endpoints: 10, Scheme: SchemeHTTPS}, false}, - {"bad scheme", StartRequest{Endpoints: 10, Scheme: "ftp"}, true}, - {"most endpoints", StartRequest{Endpoints: MaxEndpoints}, false}, - {"no endpoints", StartRequest{}, true}, - {"too many endpoints", StartRequest{Endpoints: MaxEndpoints + 1}, true}, - {"negative timeout", StartRequest{Endpoints: 1, RequestTimeoutMs: -1}, true}, - {"negative interval", StartRequest{Endpoints: 1, IntervalMs: -1}, true}, - {"offset", StartRequest{Endpoints: 3, IntervalMs: 100, OffsetMs: 50}, false}, - {"offset of one period", StartRequest{Endpoints: 3, IntervalMs: 100, OffsetMs: 300}, false}, - {"offset over one period", StartRequest{Endpoints: 3, IntervalMs: 100, OffsetMs: 301}, true}, - {"negative offset", StartRequest{Endpoints: 1, IntervalMs: 100, OffsetMs: -1}, true}, - {"offset without interval", StartRequest{Endpoints: 1, OffsetMs: 1}, true}, + {"ok", named(StartRequest{Endpoints: 10}), false}, + {"http", named(StartRequest{Endpoints: 10, Scheme: SchemeHTTP}), false}, + {"https", named(StartRequest{Endpoints: 10, Scheme: SchemeHTTPS}), false}, + {"bad scheme", named(StartRequest{Endpoints: 10, Scheme: "ftp"}), true}, + {"most endpoints", named(StartRequest{Endpoints: MaxEndpoints}), false}, + {"no endpoints", named(StartRequest{}), true}, + {"too many endpoints", named(StartRequest{Endpoints: MaxEndpoints + 1}), true}, + {"negative timeout", named(StartRequest{Endpoints: 1, RequestTimeoutMs: -1}), true}, + {"negative interval", named(StartRequest{Endpoints: 1, IntervalMs: -1}), true}, + {"offset", named(StartRequest{Endpoints: 3, IntervalMs: 100, OffsetMs: 50}), false}, + {"offset of one period", named(StartRequest{Endpoints: 3, IntervalMs: 100, OffsetMs: 300}), false}, + {"offset over one period", named(StartRequest{Endpoints: 3, IntervalMs: 100, OffsetMs: 301}), true}, + {"negative offset", named(StartRequest{Endpoints: 1, IntervalMs: 100, OffsetMs: -1}), true}, + {"offset without interval", named(StartRequest{Endpoints: 1, OffsetMs: 1}), true}, + {"no identity", StartRequest{Endpoints: 1}, true}, + {"no actor", StartRequest{Endpoints: 1, Cell: "cell-1", Run: "run-1"}, true}, + {"no cell", StartRequest{Endpoints: 1, Actor: "benchmark-workloads/egress-0", Run: "run-1"}, true}, + {"no run", StartRequest{Endpoints: 1, Actor: "benchmark-workloads/egress-0", Cell: "cell-1"}, true}, } { t.Run(tc.name, func(t *testing.T) { t.Parallel()