Repository navigation
benchmarking: add the egress-tunnels workload - #2716
Yuan Gao (ygao-g) wants to merge 5 commits into
Conversation
Yuan Gao (ygao-g)
added
area/benchmarking
kind/feature
c2054a9 to
9cc5d9f
Compare
9cc5d9f to
629c673
Compare
Lior Lieberman (LiorLieberman)
left a comment
Lior Lieberman (LiorLieberman)
left a comment
There was a problem hiding this comment.
Thx Yuan, defering to Bowei Du (@bowei) on this one. My 2c is why do we need to add more actors, cant we reuse images of existing ones?
e9156ff to
8c66e38
Compare
f7d637b to
f27c177
Compare
| // Package egresstunnels is shared by the egress-tunnels driver, actor and | ||
| // target: the routes the actor serves, the request that starts its loop, the | ||
| // stats it reports, the endpoint names, and the target's certificate. | ||
| package egresstunnels |
There was a problem hiding this comment.
We should keep all of the benchmarking code into the cmd/benchmarking/... package instead of at the top level
There was a problem hiding this comment.
Done. The package now lives at cmd/benchmarking/internal/egresstunnels. PTAL
f27c177 to
3452267
Compare
|
Consolidate this into one executable to reduce the number of executables we need to build. Use Cobra to create subcommands. Use this package structure: cmd/benchmarking/egress/main.go Use Cobra flags to create two subcommands for the executable:
Not clear why we need to split off into a separate package. Can we keep everything in cmd/benchmarking/egress/xxx.go? Make sure you unexport all of the symbols. |
| // Histogram counts durations in fixed log-spaced buckets. Every Histogram | ||
| // shares the same buckets, so histograms from many actors merge exactly. | ||
| // The zero value is empty and ready to use. | ||
| type Histogram struct { |
There was a problem hiding this comment.
Can we reuse the otel.metric data structures vs creating our own?
There was a problem hiding this comment.
Done. Each measurement is an OTel instrument. The OTel Prometheus exporter serves them on GET /metrics. The driver sums the scrapes per le label and reads the quantiles from the sum.
One part stays in histogram.go: the bucket bounds. All actors must use the same bounds, so the scrapes add bucket by bucket. The SDK does not give fixed log-spaced bounds, and math.Pow differs across architectures. The code rounds each bound once to four significant digits. README §4 lists the families.
| @@ -0,0 +1,67 @@ | |||
| # egresstunnels | |||
There was a problem hiding this comment.
Can we work on this README?
Rewrite this readme with the following sections.
- The goal of what the egress benchmark tests
- General topology of the setup, seems like you run
benchmark-egress actorinside the Actor andbenchmark-egress targetfor the traffic sink. - Workflow that the Actor runs.
- Metrics that you are collecting from the benchmark Actor.
- What metrics are aggregated and will be summarized by running the experiment.
After rewriting the readme, you should structure your code etc to match the above description.
There was a problem hiding this comment.
Done. The README has these five sections:
- What the benchmark measures: the goal, the terms tunnel, blended and settled, and the costs a cell reports.
- Topology: driver,
benchmark-egress actorin the Actor, gateway, endpoint Services, andbenchmark-egress targetas the backend. - What the actor does: the routes, the
/startbody, and the per-endpoint loop. - What the actor reports: each family on
/metrics, what it counts, and what it is used for. - What a run aggregates: the merge rules, the two windows, and how each result is computed.
bbcbdbe to
fd05f73
Compare
97c0e7f to
ee00514
Compare
Done.
|
The egress scale test holds many CONNECT tunnels open through the egress gateway at a fixed pace and measures what they cost. Its programs are subcommands of one binary, so there is one image to build and ship. This is the Cobra root; the subcommands follow.
ee00514 to
4f01533
Compare
Every measurement the actor reports is an OTel instrument: counters and gauges for the counts, and Float64Histograms for opening, repeat, lookup and handshake latency. The OTel Prometheus exporter serves them on GET /metrics, text or protobuf by Accept, so the actor defines no report type of its own and no merge: the driver sums scrapes per le label and reads quantiles from the sum. This commit defines the families, their names and labels, the route, and the gather and encode steps; the actor commit records into them. The histograms share fixed bounds, 20 per decade from 1 us to 60 s plus overflow, so scrapes from many actors add bucket by bucket. math.Pow is not bit-identical across architectures, so each bound is rounded once to four significant digits; an amd64 actor and an arm64 driver then emit the same le labels. The per-endpoint families live in a second registry and are gathered only on ?endpoints=1: at C = 1024 they outweigh the rest of the scrape more than five to one. prometheus/client_model and prometheus/common move from indirect to direct dependencies for the dto types and the exposition codecs.
4f01533 to
8d08431
Compare
The actor is the workload inside the sandbox. On POST /start it runs one goroutine per endpoint: endpoint i opens its tunnel i intervals after the start, then requests the endpoint once per period of C intervals, so the actor sends one request per interval and a request that outlasts the period delays only its own tunnel. Each request records into the instruments of the previous commit; GET /metrics serves them and POST /stop ends the loop. The loop starts on request rather than at boot because every actor restores from a golden snapshot taken before any egress policy exists. Every answer carries Benchmark-Egress-Protocol: 1. Snapshots outlive images, so the driver uses the header to refuse an actor built before this contract instead of misreading its report. With it come the parts of its contract the driver and the target share: the start request and routes, the endpoint names the loop calls and the egress policy allows, classify's low-cardinality error classes, and the name-constrained CA and certificate the target serves and the actor's HTTPS tests verify against.
The target is the endpoint every tunnel leads to: a 200 with a small fixed body over plain HTTP and, given a certificate, HTTPS. One Deployment sits behind all of the test's Services, each on its own backend port, so two Services on distinct ClusterIPs cannot hand it two connections with the same 4-tuple.
8d08431 to
272c65f
Compare
The README is the contract the driver and the result reader hold the workload to: what the benchmark measures, the topology, what the actor does on each route and in its loop, what each family it reports counts, and how a run sums the scrapes into results. Facts only a maintainer of the code needs stay as comments in the source.
272c65f to
8ed6f08
Compare
3 participants
Part of #2715
First of the PRs that measure how many CONNECT tunnels the egress gateway can hold and what each costs. This PR adds
benchmark-egress, one binary with two subcommands:actor, which runs inside each Actor and holds one tunnel per endpoint through the gateway, andtarget, the Deployment behind the endpoint Services. The driver, the deploy script and the orchestrator entry follow in later PRs.cmd/benchmarking/egress/README.mdis the contract: what the benchmark measures (tunnel, blended, settled), the topology, what the actor does, what it reports, and what a run aggregates. More specifically,/start, then each is requested once per period of C ×intervalMs.GET /metrics. The driver sums scrapes perlelabel and reads quantiles from the sum.Overview of Commits.
/metricsand the exposition codecs, the reproducible bounds/startcontract, the loop, recording into the instruments, the protocol header,classify, the target's name-constrained certificateTesting
Ran on GKE with 1000 actors x 1000 HTTPS endpoints through one c3-standard-44 gateway, with the driver in the next PR. Against the README's contract:
The target served 1000 Services on the port range from one image.
Prepared with help from an AI coding agent; reviewed and verified by the author.