From fbbff7d0afa752ead08f2f67558ff30e4bd8a589 Mon Sep 17 00:00:00 2001 From: Seer Date: Wed, 29 Jul 2026 09:53:09 -0400 Subject: [PATCH 1/8] docs: add Redis performance testing guide --- .../redis/Redis_Performance_Testing_Guide.md | 618 ++++++++++++++++++ 1 file changed, 618 insertions(+) create mode 100644 docs/en/solutions/ecosystem/redis/Redis_Performance_Testing_Guide.md diff --git a/docs/en/solutions/ecosystem/redis/Redis_Performance_Testing_Guide.md b/docs/en/solutions/ecosystem/redis/Redis_Performance_Testing_Guide.md new file mode 100644 index 000000000..f3d70b1b8 --- /dev/null +++ b/docs/en/solutions/ecosystem/redis/Redis_Performance_Testing_Guide.md @@ -0,0 +1,618 @@ +--- +products: + - Alauda Application Services +kind: + - Solutions +--- + +# Redis Cluster and Sentinel Performance Testing Guide + +## Purpose and scope + +This guide defines a reproducible, customer-executable method for measuring Redis data-plane throughput, latency, and resource use on Kubernetes. It covers: + +- **Redis Cluster mode**: three or more primary shards, optionally with replicas. The load generator must use `redis-benchmark --cluster` so requests reach all hash slots. +- **Redis Sentinel mode**: one writable primary with replicas and Sentinel processes. Benchmark the read-write data Service on port `6379`, not the Sentinel Service on port `26379`. + +Run this procedure only against a dedicated test instance. It issues writes and can consume all assigned CPU, memory, network, and storage bandwidth. It does not test failover; run failover as a separate workload because a topology change invalidates a steady-state throughput result. + +There is no hardware-independent "best" result. Define the workload, durability policy, topology, and latency objective before testing. Compare only runs in which these inputs are identical, as required by the [Redis benchmarking guidance](https://redis.io/docs/latest/operate/oss_and_stack/management/optimization/benchmarks/). + +Use one of two access-path profiles and never merge their results: + +- **In-cluster capacity profile (recommended)**: run the load Pod in the Redis cluster and connect through the ClusterIP Service. This follows the current Redis Operator suite and removes external load-balancer/wide area network (WAN) capacity from the Redis result. +- **External end-to-end profile**: run the same open-source client image from the customer-selected source cluster or host through NodePort or LoadBalancer. This follows the cross-cluster capability of the earlier validation suite. Record the client location, access type, round-trip latency, packet loss, and every network hop. For Cluster mode, every node address announced by Cluster discovery must be reachable from the load generator; reaching only the seed endpoint is insufficient. + +## Test design + +Use the following workload matrix for both Cluster and Sentinel. + +Redis Database (RDB) persistence, Append Only File (AOF) persistence, and a non-persistent cache are the three profiles. Select a product parameter template first, keep it unchanged for the baseline run, and then change only one tuning axis at a time. The value-size, pipeline, command, and keyspace rows reproduce the Redis Operator performance suite and earlier validation cases. + +| Axis | Standard values | Purpose | +|---|---|---| +| Parameter template | RDB persistence; AOF persistence; no persistence (cache) | Quantifies durability and replication cost separately using shipped product baselines | +| Value size (`-d`) | `512` bytes; `4096` bytes (4 KiB) | Matches the latest performance e2e matrix for small and medium values | +| Pipeline (`-P`) | `1`, `16` | `1` measures non-pipelined behavior; `16` measures a throughput-oriented workload | +| Commands (`-t`) | `set,get` | Produces separate write and read results | +| Random keyspace (`-r`) | `100000` | Avoids benchmarking a single hot key | +| Redis I/O threads | Selected-template baseline (`io-threads=4`, threaded reads disabled), followed by a CPU-specific sweep | Quantifies socket I/O and protocol-processing scalability without mixing it with other tuning | + +### Select a product parameter template + +This procedure uses the shipped Redis 7.2 templates as its only parameter baseline. In the Redis create form, choose Redis 7.2 and the topology first. Under **Parameter Templates** in the English interface or **参数模板** in the Chinese interface, select the exact identifier that matches both. The identifier displayed in the selector is `metadata.name` and is the same in both languages; only the description is localized. The bilingual profile names below describe the intent and are not alternative resource names. + +| Profile name (中文 / English) | Redis 7.2 Sentinel | Redis 7.2 Cluster | +|---|---|---| +| RDB 持久化 / RDB persistence | `system-rdb-redis-7.2-sentinel` | `system-rdb-redis-7.2-cluster` | +| 无持久化（缓存） / No persistence (cache) | `system-diskless-redis-7.2-sentinel` | `system-diskless-redis-7.2-cluster` | +| AOF 持久化 / AOF persistence | `system-aof-redis-7.2-sentinel` | `system-aof-redis-7.2-cluster` | + +The product selects the corresponding RDB template by default. Select the AOF or no-persistence template explicitly when that is the intended scenario. Some package revisions contain `RBD` in the localized RDB description; this is a description typo. Use the `...rdb...` identifier shown above. This guide uses the canonical term **RDB**. + +The shipped Redis 7.2 templates establish these test baselines: + +| Profile | Effective persistence settings | Intended use | +|---|---|---| +| RDB persistence | `appendonly=no`; `save="60 10000 300 100 600 1"` | Snapshot-based persistence; possible data loss since the last completed snapshot | +| AOF persistence | `appendonly=yes`; `appendfsync=everysec`; `save=""` | AOF durability/throughput balance | +| No persistence (cache) | `appendonly=no`; `save=""`; `repl-diskless-sync=yes`; `repl-backlog-size=50mb` | Cache workloads that can tolerate data loss | + +All listed templates leave `maxmemory` unset and set `maxmemory-policy=noeviction`, `repl-diskless-sync=yes`, `io-threads=4`, and `io-threads-do-reads=no`. With this combination, the Operator generates `maxmemory` as 80% of the Redis data container's memory limit. This is Operator behavior, not the native Redis default. Because installed package revisions can differ, export the installed template and verify the effective configuration rather than reproducing these values manually. + +For Sentinel topology, template selection also supplies Sentinel-process parameters. Keep them unchanged during a data-plane throughput comparison; Sentinel processes do not serve the benchmark traffic, and failover testing is outside this steady-state procedure. + +The no-persistence profile is not "diskless persistence": it disables local persistence and uses diskless synchronization only for replication. Use it only when the application can tolerate data loss. Redis documents the durability and performance trade-offs of RDB, AOF, and no persistence in [Redis persistence](https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/). + +List and export the installed templates before testing. Discover the namespace instead of assuming one: + +```bash +kubectl get paramtemplate.middleware.alauda.io -A \ + -l component=redis,support_version=7.2 + +TEMPLATE_NS=replace-with-template-namespace +TEMPLATE=system-rdb-redis-7.2-cluster +kubectl -n "${TEMPLATE_NS}" get paramtemplate.middleware.alauda.io \ + "${TEMPLATE}" -o yaml > "paramtemplate-${TEMPLATE}.yaml" +``` + +### Other Redis versions + +Template identifiers, supported parameters, defaults, persistence behavior, and apply strategies can differ by Redis and product package version. For any version other than 7.2, select a template whose labels match that exact version and topology, export the installed `ParamTemplate`, and build a separately named test matrix from its effective values. Do not reuse the Redis 7.2 identifiers or assume that its resource, storage, persistence, or I/O-thread defaults apply. + +## Prerequisites + +Prepare the following before the test window: + +- A Kubernetes cluster with the Redis Operator installed and a qualified `StorageClass` for RDB and AOF profiles. +- The Redis 7.2 system `ParamTemplate` resources installed for the topology under test. +- A namespace dedicated to the test, for example `redis-perf`. +- Product-console permission to create Redis instances and select parameter templates, plus `kubectl` access that can create the load Pod and read Redis resources and metrics. Install `jq` for producing allowlisted evidence exports. +- A Redis password Secret with key `password`. Reuse the instance Secret or create it from a protected file; never place the password in a manifest or report: + + ```bash + kubectl -n redis-perf create secret generic redis-perf-auth \ + --from-file=password=/secure/path/redis-password + ``` + + The protected file must contain only the password bytes, without an unintended trailing newline. The current Redis admission policy requires 8-32 allowed characters and rejects passwords that do not include letters, digits, and a special character. Confirm the installed rule with `kubectl explain redis.spec.passwordSecret --api-version=middleware.alauda.io/v1` because package revisions can differ. + + Use a distinct password Secret for each fresh Redis instance. The current Operator can attach the supplied Secret to an instance-owned `RedisUser`; deleting the instance then removes that Secret through Kubernetes ownership. Before creating the next matrix instance, create its new Secret and recreate the load-generator Pod so its projected volume references the new name. Do not assume that a Secret shared with a deleted instance will remain available. + +- An approved [Redis Docker Official Image](https://hub.docker.com/_/redis/) containing `redis-benchmark` and `redis-cli`. Match the server major/minor version where practical, mirror the exact image into the customer registry for an air-gapped cluster, and record its immutable digest. Do not use an Alauda-internal load-generator image. +- Metrics Server for sampled Pod CPU/memory, or Prometheus for time-series CPU, memory, network, storage, and Redis exporter metrics. + +Before applying examples, confirm the installed schema with: + +```bash +kubectl explain redis.spec --api-version=middleware.alauda.io/v1 +``` + +### Qualify the StorageClass + +Use one of these storage profiles and report it explicitly: + +- **Performance-ceiling profile**: prefer low-latency local block storage backed by dedicated SSD or NVMe media. An installed TopoLVM-backed class such as `sc-topolvm` is a suitable candidate, but the name alone is not a performance guarantee: [TopoLVM](https://github.com/topolvm/topolvm) provisions local Logical Volume Manager (LVM) volumes, so verify its device class and physical media. For topology-constrained local storage, use `volumeBindingMode: WaitForFirstConsumer` so volume provisioning follows Pod scheduling constraints, as described in [Kubernetes Storage Classes](https://kubernetes.io/docs/concepts/storage/storage-classes/). Because local volumes are tied to nodes, this profile does not prove cross-node recovery behavior; test recovery separately with the intended production storage. +- **Production-representative profile**: use the exact production `StorageClass`, even when it is slower, and report the result as end-to-end production storage performance rather than the Redis ceiling. Never compare this result directly with the local-block profile. + +Do not use Network File System (NFS), other network-attached storage, or CephFS for the performance-ceiling profile. Redis explicitly warns against placing RDB or AOF files on NFS or network-attached storage because the storage path can add uncontrolled network latency and bandwidth contention. Do not group all Ceph storage under the same rule: Ceph block storage is distributed block storage rather than a shared file system. Its network, replication, recovery, and quality-of-service settings can still dominate latency, so use Ceph block storage only for the production-representative profile or after storage qualification demonstrates sufficient headroom. See [Redis benchmark factors](https://redis.io/docs/latest/operate/oss_and_stack/management/optimization/benchmarks/) and [Ceph architecture](https://docs.ceph.com/en/latest/architecture/). + +Before deploying Redis, qualify a disposable volume with the same `StorageClass`, capacity, filesystem, and node type. Record 4 KiB random-write operation rate, sequential write throughput, and synchronous-write latency percentiles. There is no universal pass threshold: the volume must meet the customer's storage service objective and retain headroom at the Redis peak without throttling, prolonged queueing, recovery traffic, or unrelated I/O. A storage-saturated run measures the storage path, not the Redis performance ceiling. + +## Record the environment before testing + +Create a run directory and capture facts that affect comparability. Do not export Secrets. + +```bash +RUN_ID="$(date -u +%Y%m%dT%H%M%SZ)" +OUT="redis-perf-${RUN_ID}" +mkdir -p "${OUT}" + +kubectl version -o yaml > "${OUT}/kubernetes-version.yaml" +kubectl get nodes -o wide > "${OUT}/nodes.txt" +kubectl get nodes -o json > "${OUT}/nodes.json" +kubectl get storageclass -o yaml > "${OUT}/storageclasses.yaml" +kubectl -n redis-perf get redis,pod,svc,pvc -o wide > "${OUT}/placement.txt" +kubectl -n redis-perf get pvc -o yaml > "${OUT}/persistent-volume-claims.yaml" + +SC_NAME=sc-topolvm # Example; replace with the selected StorageClass. +kubectl get storageclass "${SC_NAME}" -o yaml \ + > "${OUT}/storageclass-${SC_NAME}.yaml" + +TEMPLATE_NS=replace-with-template-namespace +TEMPLATE=replace-with-selected-template-identifier +kubectl -n "${TEMPLATE_NS}" get paramtemplate.middleware.alauda.io \ + "${TEMPLATE}" -o yaml > "${OUT}/paramtemplate-${TEMPLATE}.yaml" +``` + +Also record the Redis Operator version, Redis server version, image IDs, node CPU model and frequency policy, network link speed, StorageClass/Container Storage Interface (CSI) implementation, volume performance class, service mesh status, and test time zone. Redis recommends isolated hardware, stable CPU frequency, known client/server network latency, and no unrelated storage I/O for reproducible results. Verify `vm.overcommit_memory=1` and Transparent Huge Pages are disabled on Redis nodes according to [Redis administration guidance](https://redis.io/docs/latest/operate/oss_and_stack/management/admin/); platform administrators must apply host changes through the supported node-management process. + +## Configure Redis instances + +Create each test instance through the product console so the selected system template is applied through the supported workflow: + +1. Choose Redis 7.2 and select **Cluster** or **Sentinel** topology. +2. Select the exact **Parameter Templates / 参数模板** identifier from the table above. Do this before manually changing Redis parameters. +3. The listed system templates assign limits of `4` CPU and `8Gi` memory to each Redis server, matching the standard capacity profile. Confirm those values after selection and set requests equal to limits where the form exposes both. Configure three primary shards and one replica per primary for Cluster, or one primary, one replica, and three Sentinel processes for Sentinel. Treat any resource or topology change as a separately named profile. +4. For RDB or AOF, choose the qualified `StorageClass` for the declared performance-ceiling or production-representative profile. For both topologies, the shipped templates prefill `24Gi` for RDB and `32Gi` for AOF. Treat these values as minimum starting capacities, calculate the requirement as described below, and use the larger value. The no-persistence templates do not require persistent data storage. +5. Enable the Redis exporter. Expand **Parameters / 参数配置** and verify `appendonly`, `appendfsync`, `save`, `maxmemory`, `maxmemory-policy`, `repl-diskless-sync`, `io-threads`, and `io-threads-do-reads` against the exported template. An empty template value for `maxmemory` is intentional; verify the Operator-derived effective value after creation. +6. After correcting resource and storage capacity for the target dataset, create the baseline with the Redis parameters unchanged. For a tuned comparison, start again from the same template and change only the documented target parameter. If the installed platform supports customer-owned templates, save the tuned copy under a new name; never edit the shipped system template in place. + +Template selection materializes the template values into the Redis instance configuration; the Redis custom resource does not retain a template-name reference. Therefore, the report must preserve both the exact exported `ParamTemplate` and the resulting Redis custom resource. Do not invent a `spec.paramTemplate` field for automation. + +### Set persistent volume capacity for the standard matrix + +Do not require the test operator to infer capacity from observed persistent volume claim (PVC) usage. Calculate it before deployment from the test matrix. Use the same per-Pod calculation for Sentinel and Cluster: Sentinel data nodes hold the complete dataset, while applying the complete-workload allowance to every Cluster data Pod avoids relying on perfectly even slot or key distribution. + +Let `N` be the number of `SET` requests per measured point, `K` the random keyspace, `D` the value size in bytes, and `S` the number of `SET` passes retained since the last successful AOF rewrite. `GET` and pipeline depth do not increase persistent data volume. + +```text +RDB calculated peak = 2 * K * D + # Existing RDB plus the temporary RDB created by BGSAVE. + +AOF calculated peak = (S * N + 2 * K) * D + # Retained SET passes, the existing compacted dataset, + # and the new base file created by BGREWRITEAOF. + +required capacity = ceil_Gi(max(template capacity, calculated peak * safety factor)) +``` + +`ceil_Gi` means round up to the next whole GiB expressed as a Kubernetes `Gi` value. + +Use a safety factor of `1.5` for the bounded standard matrix when the command mix is `SET,GET`, `S` is known, and AOF housekeeping completes between points. Use `2.0` when key lengths or command framing differ materially, the number of retained writes is less predictable, or the test intentionally continues writing during persistence work. For write commands other than `SET`, first replace `S * N * D` with that workload's maximum serialized write volume; increasing the factor alone does not correct an invalid payload model. These factors are this guide's engineering safety policy, not Redis-defined constants. If the write volume cannot be bounded, stop and define a maximum duration or request count; no finite percentage can prevent an indefinitely growing AOF from filling its volume. + +For the standard matrix (`N=1,000,000`, `K=100,000`, `S=1`), the calculation is: + +| Persistence | Value size | Calculated peak | Peak × `1.5` | Template minimum | Provision per data Pod | +|---|---:|---:|---:|---:|---:| +| RDB | 512 B | 0.095 GiB | 0.143 GiB | 24Gi | **24Gi** | +| RDB | 4096 B | 0.763 GiB | 1.145 GiB | 24Gi | **24Gi** | +| AOF | 512 B | 0.572 GiB | 0.858 GiB | 32Gi | **32Gi** | +| AOF | 4096 B | 4.578 GiB | 6.867 GiB | 32Gi | **32Gi** | + +The resulting Kubernetes-requested capacity for the standard topology is: + +| Topology | Redis data Pods | RDB total | AOF total | +|---|---:|---:|---:| +| Sentinel: one primary + one replica | 2 | **48Gi** | **64Gi** | +| Cluster: three primaries + one replica per primary | 6 | **144Gi** | **192Gi** | + +The standard matrix fits within the template capacities because it has a bounded keyspace, one million requests, and one retained AOF pass. This does not make `24Gi` or `32Gi` generally sufficient. For example, with `N=10,000,000`, `K=100,000`, `D=4096`, and `S=1`, the AOF calculated peak is `38.910 GiB`; applying `1.5` requires `59Gi` after rounding up. Round further when the storage class supports only fixed allocation increments. + +Redis serialization, AOF protocol framing, key names, and other files are not included in the payload-only peak. The safety factor provides space for these items and for variation around the bounded test. Redis writes a temporary RDB before replacing the old file. Redis 7.2 multipart AOF retains the current file set while rewrite creates a new base and continues accepting writes, which is why both generations are included before applying the factor. + +Use a fresh instance for every parameter-template, value-size, pipeline, and Redis I/O-thread scenario, as the Operator performance suite does. During a client-concurrency sweep on an AOF instance, either recreate the instance for every measured point or run `BGREWRITEAOF` after warm-up and after each point, wait for `aof_rewrite_in_progress:0`, and require `aof_last_bgrewrite_status:ok` before continuing. Run this housekeeping outside the measured interval. + +If `-n`, `-r`, `-d`, or `S` changes, recompute the capacity before creating the instance. Do not reuse the standard values for the earlier 5-30-million-request Ares cases or a customer workload without recalculation. Multiply the per-Pod result by two for the standard Sentinel topology or by six for the standard three-primary, one-replica-per-primary Cluster topology to obtain total Kubernetes-requested capacity. Storage-system replication, thin-provisioning reserve, and filesystem overhead are additional. For local block storage, every eligible node must have enough allocatable local capacity for the Redis data Pods that can be scheduled there. Provision the calculated capacity before testing; do not resize a PVC during a measured run. + +Wait for `.status.phase` to become `Ready`, then verify Pod placement: + +```bash +kubectl -n redis-perf wait --for=jsonpath='{.status.phase}'=Ready \ + redis/replace-with-instance-name --timeout=30m +kubectl -n redis-perf get pod -o wide +``` + +For valid results, place the load generator on a node that does not host a Redis server. Spread Sentinel data nodes across hosts. Cluster placement is shard-aware, so always record the actual node of every primary and replica rather than assuming separation. + +### Redis configuration rules + +- For the unchanged Redis 7.2 template baseline, leave `maxmemory` empty. Because the templates set `maxmemory-policy=noeviction`, the Operator writes an effective `maxmemory` equal to 80% of the Redis data container memory limit. The standard `8Gi` limit therefore produces approximately `6.4Gi` of `maxmemory`. Confirm the exact byte value with `CONFIG GET maxmemory` on every data Pod; do not report the empty template field as an unlimited Redis configuration. +- The remaining 20% is process headroom, not additional dataset capacity. It must absorb replication/AOF buffers, allocator fragmentation, client buffers, and copy-on-write pages during RDB save or AOF rewrite. Redis warns that a write-heavy background save can temporarily require substantially more memory. If this headroom is insufficient under the target workload, increase the container limit and rerun the resource profile instead of raising `maxmemory` toward the limit without evidence. +- An explicit `maxmemory` overrides the Operator-derived value and is a separately named tuning profile. In the current Operator implementation, leaving `maxmemory` empty with a configured policy other than `noeviction` derives 70% of the memory limit instead of 80%. Therefore, a `maxmemory-policy` comparison must either pin the same explicit `maxmemory` in every candidate or report that usable dataset capacity also changed. +- Keep `maxmemory-policy: noeviction` for a capacity test. An out-of-memory write then fails visibly instead of silently replacing keys and making the workload incomparable. If production uses eviction, run an additional profile with the production policy. +- Do not remove replicas merely to inflate throughput. Sentinel with zero replicas no longer represents its high-availability purpose; each Cluster shard/replica count must match production. +- Keep the selected template unchanged for the baseline: the listed Redis 7.2 templates use `io-threads=4` and `io-threads-do-reads=no`. Do not change `hz`, backlog, persistence, or kernel settings in the same comparison; change one setting at a time and retain before/after evidence. +- Keep the selected `StorageClass` and storage profile constant across a comparison. Do not present results from NFS, other network-attached storage, or CephFS as a Redis performance ceiling. Treat Ceph block storage or other network block storage as production-representative unless its qualification evidence shows that it retains headroom throughout the run. + +### Tune Redis I/O threads + +Redis 7.2 can use I/O threads for client socket work, while command execution remains mostly on the main thread. `io-threads` counts the main thread: `1` means the normal single-threaded I/O path. Values greater than `1` offload socket writes. Set `io-threads-do-reads=yes` only as a separate profile to also offload socket reads and protocol parsing; the Redis 7.2 reference configuration says threaded reads usually provide little benefit. Redis 7.2 also states that threaded I/O does not work with TLS. Override the template to `io-threads=1` for a TLS profile and record that deviation. + +Use the CPU available to the Redis container, not the Kubernetes node's total CPU count. Always measure the unchanged product-template value of `4` first. The following controlled sweeps include both the product baseline and upstream comparison points; they are not universal optimums: + +| Redis version | Redis container CPU | Product baseline | Controlled `io-threads` sweep | `io-threads-do-reads` | +|---|---:|---|---|---| +| 7.2 | 4 vCPU | `4` | `1`, `2`, `3`, `4` | `no` first; test `yes` separately | +| 7.2 | 8 vCPU | `4` after copying the template | `1`, `4`, `6` | `no` first; test `yes` separately | + +For other CPU allocations, retain `4` as the template-baseline result, then compare `1` and gradually increasing values that leave CPU headroom. Changing from 4 to 8 vCPU is a separate resource profile; do not merge those results. Do not set more I/O threads than the Redis container can run without sustained CPU throttling. Guaranteed quality of service (QoS) requires equal CPU and memory requests and limits for every container in the Pod. Dedicated CPUs additionally require the cluster's CPU Manager static policy and an integer Guaranteed CPU request; record whether that policy is active. + +Use the parameter editor to apply one candidate to a new copy of the selected template. For example, keep the first pair for the baseline and use the second pair only for a separately named 4-vCPU comparison: + +```text +# Unchanged product-template baseline +io-threads 4 +io-threads-do-reads no + +# Example comparison profile; all other template values remain unchanged +io-threads 3 +io-threads-do-reads no +``` + +The product `ParamDefinition` classifies both keys as `RestartApply`. Use a fresh instance for the cleanest comparison, or allow the managed restart to finish, wait for `Ready`, and confirm that all expected primaries and replicas have returned before loading data. Verify the effective values on every Redis data Pod with the collection helper below; a mixed value across nodes invalidates the run. + +Calibrate the load generator before comparing server candidates. Redis 7.2 recommends running `redis-benchmark` in threaded mode when evaluating I/O threads. Set client `--threads` high enough that the client retains CPU and network headroom at the largest server candidate, then hold the client setting constant across the `io-threads` comparison. The load-generator thread count and the server I/O thread count are different parameters and must be reported separately. + +For each candidate, repeat the client-concurrency sweep. Select the setting with the highest reproducible throughput that still meets the agreed p99 latency, error-rate, replication-lag, and CPU-throttling limits. Stop increasing threads when throughput no longer improves materially or latency/throttling worsens; retaining only the highest one-off result is not a valid tuning decision. + +## Prepare an open-source load generator + +Use the official Redis image on a dedicated load-generator node. The following reusable Pod reads the password from a Secret volume. Replace the image with the exact mirrored image and update the anti-affinity instance name for each topology. If a dedicated node is unavailable, remove the selector/toleration but treat any client/server co-location as an invalid run. + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: redis-perf-client + namespace: redis-perf + labels: + app.kubernetes.io/name: redis-perf-client +spec: + restartPolicy: Never + nodeSelector: + redis-perf/loadgen: "true" + tolerations: + - key: redis-perf/loadgen + operator: Equal + value: "true" + effect: NoSchedule + affinity: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchLabels: + middleware.instance/name: redis-perf-cluster + topologyKey: kubernetes.io/hostname + containers: + - name: client + image: redis:7.2.15 + imagePullPolicy: IfNotPresent + command: ["sh", "-c", "while true; do sleep 3600; done"] + resources: + requests: {cpu: "8", memory: 8Gi} + limits: {cpu: "8", memory: 8Gi} + volumeMounts: + - name: auth + mountPath: /auth + readOnly: true + volumes: + - name: auth + secret: + secretName: redis-perf-auth +``` + +In an isolated test cluster, reserve a node before applying the Pod: + +```bash +kubectl label node replace-with-loadgen-node redis-perf/loadgen=true +kubectl taint node replace-with-loadgen-node redis-perf/loadgen=true:NoSchedule +kubectl apply -f redis-perf-client.yaml +kubectl -n redis-perf wait --for=condition=Ready pod/redis-perf-client --timeout=10m +kubectl -n redis-perf exec redis-perf-client -- redis-benchmark --version \ + | tee "${OUT}/redis-benchmark-version.txt" +kubectl -n redis-perf exec redis-perf-client -- redis-benchmark --help \ + > "${OUT}/redis-benchmark-help.txt" +``` + +Use only options shown by the exact client binary. In particular, the Redis 7.2.15 +binary does not expose `-e`; preserve benchmark standard error and collect Redis +`INFO errorstats` and rejected-connection counters instead of copying that option +from documentation for a different client version. + +## Validate connectivity and topology + +Use the Service name reported by the Redis custom resource. The data port is `6379` for both topologies. + +```bash +NS=redis-perf +REDIS_NAME=redis-perf-cluster +SERVICE="$(kubectl -n "${NS}" get redis "${REDIS_NAME}" \ + -o jsonpath='{.status.serviceName}')" +HOST="${SERVICE}.${NS}.svc" + +kubectl -n "${NS}" exec redis-perf-client -- sh -eu -c ' + redis-cli -h "$1" -p 6379 -a "$(cat /auth/password)" PING +' sh "${HOST}" +``` + +Expected result: `PONG`. For Cluster, also require `cluster_state:ok`, all `16384` slots assigned, and zero `cluster_slots_pfail`/`cluster_slots_fail`: + +```bash +kubectl -n "${NS}" exec redis-perf-client -- sh -eu -c ' + redis-cli -h "$1" -p 6379 -a "$(cat /auth/password)" CLUSTER INFO +' sh "${HOST}" | tee "${OUT}/cluster-info-before.txt" +``` + +Before load, also require the expected replicas to be connected, no initial synchronization in progress, and no RDB save or AOF rewrite in progress. Record an intentional background persistence operation as part of the scenario instead of allowing it to occur in only one compared run. + +For Sentinel, use the Redis read-write Service for load. Separately query the Sentinel Service and record `SENTINEL MASTER`, `SENTINEL REPLICAS`, `SENTINEL SENTINELS`, and `SENTINEL CKQUORUM` output. The monitored master name and Sentinel Service are deployment-specific; obtain them from the instance status/services rather than assuming a name. The [Redis Sentinel documentation](https://redis.io/docs/latest/operate/oss_and_stack/management/sentinel/) defines these commands and health fields. + +## Run the benchmark + +The command below accepts topology and load parameters as positional arguments. The password is read inside the Pod and is not stored in the manifest or report. Because `redis-benchmark` accepts authentication through `-a`, the password is present in that process's arguments while the test runs; restrict Pod exec/debug permissions, do not enable shell tracing, and delete the client Pod after evidence collection. + +```bash +ARCH=cluster # cluster or sentinel +CLIENTS=100 +REQUESTS=1000000 +DATA_SIZE=512 +PIPELINE=1 +THREADS=4 # redis-benchmark client threads, not Redis io-threads +KEYSPACE=100000 +RESULT_ID="${ARCH}-d${DATA_SIZE}-p${PIPELINE}-c${CLIENTS}" +RAW_OUTPUT="${OUT}/${RESULT_ID}.raw.txt" +CSV_OUTPUT="${OUT}/${RESULT_ID}.csv" + +set -o pipefail +kubectl -n "${NS}" exec redis-perf-client -- sh -eu -c ' + arch="$1"; host="$2"; clients="$3"; requests="$4" + data_size="$5"; pipeline="$6"; threads="$7"; keyspace="$8" + set -- redis-benchmark -h "$host" -p 6379 \ + -a "$(cat /auth/password)" \ + -c "$clients" -n "$requests" -d "$data_size" \ + -P "$pipeline" --threads "$threads" -r "$keyspace" \ + -t set,get --csv + if [ "$arch" = cluster ]; then set -- "$@" --cluster; fi + "$@" +' sh "${ARCH}" "${HOST}" "${CLIENTS}" "${REQUESTS}" \ + "${DATA_SIZE}" "${PIPELINE}" "${THREADS}" "${KEYSPACE}" \ + | tee "${RAW_OUTPUT}" + +# Redis 7.2.15 Cluster mode prints node-discovery lines before the quoted CSV rows. +# Preserve the raw evidence and create a machine-readable CSV containing only those rows. +awk '/^"/ {print}' "${RAW_OUTPUT}" > "${CSV_OUTPUT}" +test "$(wc -l < "${CSV_OUTPUT}")" -ge 3 +``` + +`redis-benchmark` parameters: + +| Parameter | Meaning and selection rule | +|---|---| +| `-c` | Parallel connections. Start at `50` (the community default) or `100` (the Operator suite default), then increase. Earlier internal validation used `200` for Cluster and `1000` for Sentinel as high-concurrency reference points. | +| `-n` | Requests per selected operation. Use `1,000,000` for a quick measured run. Earlier internal validation used 5-30 million depending on topology/value size. Increase it until the measurement contains enough monitoring samples and reaches steady state. | +| `-d` | `SET`/`GET` value size in bytes. Use `512` and `4096` (4 KiB) for the standard matrix. Add separately named sizes when the application's payload distribution differs. | +| `-P` | Pipeline depth. `1` is no pipelining; `16` is the standard throughput-oriented point. Use the application's average pipeline depth for a production-realistic test. Never compare `P=1` and `P=16` as if they were the same workload. | +| `--threads` | Load-generator threads, independent of Redis `io-threads`. Begin with `4`; increase only if the client Pod is CPU-bound before Redis. Earlier internal reference values were `16` for Cluster and `8` for Sentinel. During a server I/O-thread sweep, provision enough client threads for the largest candidate and then hold this value constant. | +| `-r` | Random keyspace. `100000` is the implemented baseline; enlarge it when the real dataset has a larger working set. | +| `-t` | Commands to test. `set,get` is the standard comparison. Add a separate scenario for the application's actual command mix; do not mix incomparable command sets in one result table. | +| `--cluster` | Required for Redis Cluster so the tool discovers and routes across Cluster nodes. Omit for Sentinel. All announced Cluster node addresses must be reachable from the client Pod. | +| `--csv` | Requests CSV-formatted result rows. Redis 7.2.15 Cluster mode also writes node-discovery text to standard output, so preserve the raw file and extract the quoted rows as shown above. Output fields can differ between client versions; mark unavailable latency percentiles as `N/A`, never as zero. | + +If an Access Control List (ACL) username is required, add `--user ` while continuing to read the password from the Secret. For TLS-enabled instances, mount the approved certificate authority (CA) and client certificate Secrets and use the TLS flags supported by the exact `redis-benchmark --help` output; record TLS as a separate test profile because it changes CPU and latency. + +## Find the saturation point + +Use a controlled sweep; do not jump directly to the largest client count. + +1. Run a short connectivity/warm-up pass. Do not include it in the report. +2. Measure `P=1` with `c=50`, then `100`, `200`, `400`, continuing upward while throughput rises and the load generator has spare CPU/network capacity. +3. At every point, inspect `SET`/`GET` requests per second, p99 latency, errors, Redis CPU, client CPU, CPU throttling, network throughput, memory, and persistence activity. +4. When more clients no longer produce a material throughput increase, or latency/errors grow rapidly, repeat the last stable point and the first overloaded point. The last reproducible point that meets the agreed latency/error objective is the usable capacity; the highest reproducible, error-free throughput regardless of latency is the throughput ceiling. Report both when they differ. +5. If the client Pod reaches its CPU limit or network limit while Redis still has headroom, raise client resources or use multiple client Pods. Redis explicitly warns that the benchmark client can become the bottleneck before the server. +6. Repeat the sweep with `P=16` only when the application can pipeline, or to document the protocol throughput ceiling. Keep it separate from the non-pipelined result. +7. For Cluster, inspect each shard. A single hot shard, unequal key distribution, unreachable announced address, or client-side bottleneck is not Cluster saturation. For Sentinel, additional replicas do not add write capacity; they add replication work and must remain in the production-representative topology. + +Repeat runs until results are reproducible and retain all runs. Do not report only the best sample. + +## Collect monitoring data + +### Kubernetes resource metrics + +The Operator test harness samples the standard `metrics.k8s.io` API every five seconds and records maximum/average CPU and memory. Metrics Server does not provide network, storage, or historical series; use Prometheus for those metrics. + +```bash +kubectl -n redis-perf top pod --containers +kubectl get --raw '/apis/metrics.k8s.io/v1beta1/namespaces/redis-perf/pods' \ + > "${OUT}/pod-metrics.json" +``` + +Export range data covering at least the benchmark start/end time. Adapt label selectors to the installed monitoring stack: + +```promql +sum by (pod) (rate(container_cpu_usage_seconds_total{namespace="redis-perf",container!=""}[5m])) +sum by (pod) (container_memory_working_set_bytes{namespace="redis-perf",container!=""}) +sum by (pod) (rate(container_network_receive_bytes_total{namespace="redis-perf"}[5m])) +sum by (pod) (rate(container_network_transmit_bytes_total{namespace="redis-perf"}[5m])) +``` + +The `rate()` range must contain multiple scrape samples. Use `[5m]` when the +installed monitoring interval is 60 seconds; a `[1m]` range can return no data. +Adjust the range when the installed interval differs, and record both values in +the report. + +Calculate utilization against declared container limits with a ratio of sums: + +```promql +100 * +sum(rate(container_cpu_usage_seconds_total{namespace="redis-perf",container="redis"}[5m])) +/ +sum(max by (namespace,pod,container,resource,unit) ( + kube_pod_container_resource_limits{namespace="redis-perf",container="redis",resource="cpu",unit="core"} +)) + +100 * +sum(container_memory_working_set_bytes{namespace="redis-perf",container="redis"}) +/ +sum(max by (namespace,pod,container,resource,unit) ( + kube_pod_container_resource_limits{namespace="redis-perf",container="redis",resource="memory",unit="byte"} +)) +``` + +Use the load-generator container selector in separate queries. Do not average +per-Pod percentages because differently sized Pods would receive equal weight. +Label these results **CPU limit utilization** and **memory limit utilization**; +they are not node utilization, CPU request utilization, Redis `maxmemory` +utilization, or Redis resident set size (RSS). If the Redis limit is not +explicitly configured, the product derives `maxmemory` as 80% of the container +memory limit, but the Kubernetes working set can still differ from Redis +`used_memory` and `maxmemory` because it includes process and allocator overhead. + +Export client and Redis server Pods separately. Include CPU throttling if exposed, volume latency, storage input/output operation rate, throughput, queue depth, and throttling from CSI, the storage platform, or the node exporter. For Ceph block storage, also record whether recovery, rebalancing, or client quality-of-service limits were active. Include node CPU utilization/frequency, node memory pressure, packet drops/retransmits, and Pod restart/out-of-memory events. Counter metrics require `rate()`/`increase()` rather than direct subtraction, as described by [Prometheus metric guidance](https://prometheus.io/docs/practices/instrumentation/). + +### Redis metrics + +Capture `INFO` before and after every measured run. For Cluster, collect it from every primary and replica, not only the seed Service. Redis `INFO` fields vary by version; preserve unknown fields and mark unavailable fields as `N/A`. + +The following helper discovers only Pods containing the Redis server container and writes one file per node. Run it once with `PHASE=before` and again with `PHASE=after`: + +```bash +PHASE=before +kubectl -n "${NS}" get pod -o json \ + | jq -r --arg name "${REDIS_NAME}" ' + .items[] + | select(.metadata.labels["middleware.instance/name"] == $name) + | select(any(.spec.containers[]; .name == "redis")) + | [.metadata.name, .status.podIP] + | @tsv + ' \ + | while IFS=$'\t' read -r NODE_NAME NODE_IP; do + kubectl -n "${NS}" exec redis-perf-client -- sh -eu -c ' + for section in server clients memory stats persistence replication cpu commandstats latencystats errorstats keyspace; do + echo "INFO ${section}" + redis-cli -h "$1" -p 6379 -a "$(cat /auth/password)" INFO "${section}" + done + for key in save appendonly appendfsync maxmemory maxmemory-policy repl-diskless-sync io-threads io-threads-do-reads hz; do + echo "CONFIG GET ${key}" + redis-cli -h "$1" -p 6379 -a "$(cat /auth/password)" CONFIG GET "${key}" + done + ' sh "${NODE_IP}" \ + > "${OUT}/redis-info-${PHASE}-${NODE_NAME}.txt" + done +``` + +Required sections and fields include: + +- `server`: Redis version, mode, uptime, allocator, input/output thread state. +- `clients`: connected/blocked clients and `maxclients`. +- `memory`: `used_memory`, `used_memory_rss`, peak memory, fragmentation, `maxmemory`, policy, and non-evictable buffer memory. +- `stats`: instantaneous/total operations, input/output bytes, rejected connections, evicted/expired keys, keyspace hits/misses, replication sync counters, `total_reads_processed`, `total_writes_processed`, `io_threaded_reads_processed`, and `io_threaded_writes_processed`. +- `persistence`: `rdb_bgsave_in_progress`, `rdb_last_bgsave_status`, `rdb_last_bgsave_time_sec`, `aof_current_size`, `aof_base_size`, `aof_rewrite_in_progress`, `aof_last_bgrewrite_status`, `aof_last_rewrite_time_sec`, and `aof_delayed_fsync` when present. +- `replication`: role, connected replicas, replication offsets, backlog, and link state. +- `cpu`, `commandstats`, `latencystats`, `errorstats`, and `keyspace`. +- Cluster: `CLUSTER INFO` and `CLUSTER NODES` from each node. +- Sentinel: `SENTINEL MASTER`, `REPLICAS`, `SENTINELS`, and `CKQUORUM`. + +When the product Redis exporter is enabled, export the metrics already used by the shipped dashboard, including: + +```text +redis_commands_processed_total +redis_commands_duration_seconds_total +redis_connected_clients +redis_blocked_clients +redis_memory_used_bytes +redis_memory_used_rss_bytes +redis_memory_used_peak_bytes +redis_keyspace_hits_total +redis_keyspace_misses_total +redis_evicted_keys_total +redis_net_input_bytes_total +redis_net_output_bytes_total +redis_rdb_bgsave_in_progress +redis_rdb_last_bgsave_duration_sec +redis_aof_rewrite_in_progress +redis_aof_last_rewrite_duration_sec +redis_slowlog_length +``` + +Record the raw exporter output or Prometheus range-query result, not screenshots alone. The [Redis `INFO` reference](https://redis.io/docs/latest/commands/info/) defines the server-side fields and their version boundaries. + +The shipped Redis `ServiceMonitor` applies a metric-relabel allowlist. A metric +such as `redis_up` can therefore exist on the raw exporter endpoint without being +stored by Prometheus. Use Prometheus target health or its generated `up` series to +verify scraping, and query the allowlisted Redis metrics above to verify data +retention. Do not assume that every raw exporter metric is retained. + +### Kubernetes evidence + +After each scenario, append the following without collecting Secret objects: + +```bash +kubectl -n redis-perf get redis "${REDIS_NAME}" -o json \ + | jq '{ + apiVersion, kind, + metadata: {name: .metadata.name, namespace: .metadata.namespace}, + spec: (.spec | { + version, arch, resources, persistent, persistentSize, replicas, + affinityPolicy, nodeSelector, tolerations, enableTLS, customConfig + }), + status: (.status | {phase, serviceName, lastShardCount, lastVersion}) + }' \ + > "${OUT}/${REDIS_NAME}.json" +kubectl -n redis-perf get pod -o wide \ + > "${OUT}/pods-after.txt" +kubectl -n redis-perf get events --sort-by=.lastTimestamp \ + > "${OUT}/events.txt" +kubectl -n redis-perf get pod redis-perf-client \ + -o jsonpath='{.status.containerStatuses[0].imageID}{"\n"}' \ + > "${OUT}/loadgen-image-id.txt" +``` + +On failure, also export Redis Pod logs, `kubectl describe` output, volume and `StorageClass` details, and the load-generator output. Redact customer identifiers, addresses, and credentials before sharing the bundle. + +## Report format and validity checks + +Produce JSON or CSV for automation and Markdown/PDF for customer review. One report row represents one topology + Redis version + parameter-template identifier + value size + pipeline + client count + Redis I/O-thread setting + load-generator thread count. + +Use a stable machine-readable header so runs can be compared without parsing prose: + +```text +topology,redis_version,parameter_template,persistence,storage_profile,storage_class,pvc_capacity_gib,value_size_bytes,pipeline,clients,redis_io_threads,redis_io_threads_do_reads,loadgen_threads,requests,keyspace,set_rps,set_p99_ms,get_rps,get_p99_ms,redis_cpu_max_millicores,redis_memory_max_mib,loadgen_cpu_max_millicores,loadgen_memory_max_mib,network_rx_mib_s,network_tx_mib_s,storage_iops,storage_latency_ms,valid,notes +``` + +| Category | Required report fields | +|---|---| +| Environment | Date/time, Kubernetes/Alauda Container Platform (ACP)/Operator/Redis versions, node count/type, CPU model, memory, network, storage profile, `StorageClass`, provisioner, parameters, `volumeBindingMode`, `allowVolumeExpansion`, filesystem/backing media, PVC capacity per Redis Pod, qualification results, image names and immutable digests | +| Topology | Cluster shard/replica count or Sentinel primary/replica/Sentinel count; Pod-to-node placement; Service access path | +| Redis configuration | Exact `ParamTemplate` identifier, `metadata.resourceVersion`, `cpaas.io/description` (Chinese), `cpaas.io/description-en` (English), exported template YAML, and allowlisted effective keys (`save`, `appendonly`, `appendfsync`, `maxmemory`, `maxmemory-policy`, `repl-diskless-sync`, `io-threads`, `io-threads-do-reads`, `hz`); approved tuning delta; TLS/ACL state without credentials | +| Load | Tool version/image digest, client location, access type and round-trip latency, commands, `-c`, `-n`, `-d`, `-P`, `--threads`, `-r`, warm-up method, start/end/duration | +| Results | `SET`/`GET` requests per second; average, min, p50, p95, p99, max latency when emitted; raw CSV | +| Resources | Client and server peak/average CPU/memory, throttling, network receive/transmit, storage operation rate/throughput/latency/queue depth/throttling, storage recovery or rebalancing activity, node pressure | +| Redis health | `INFO` deltas, Cluster slot/health state or Sentinel quorum/role state, errors, evictions, rejected connections, slow log, AOF size fields, persistence status and duration | +| Verdict | Meets/does not meet the pre-agreed throughput, p99, error, and resource-headroom objectives; bottleneck evidence and invalid-run reasons | + +A run is valid only when the benchmark exits successfully with positive results, the load generator is not co-located with a measured Redis server, there are no unexpected failovers, restarts, out-of-memory kills, disk-full events, PVC expansions, or failed RDB saves/AOF rewrites, and topology health remains normal. A load-generator bottleneck, missing Cluster node reachability, unrelated node/storage load, or a persistence rewrite that occurs in only one compared run makes the comparison invalid. A run with storage-capacity or storage-performance saturation remains useful as a storage-limited production-path result, but it must not be reported as the Redis performance ceiling. Performance is a measured result, not a universal pass/fail threshold; define customer acceptance criteria before execution. + +## Cleanup + +Delete only the dedicated test resources after evidence has been exported: + +```bash +kubectl -n redis-perf delete pod redis-perf-client +kubectl -n redis-perf delete redis redis-perf-cluster redis-perf-sentinel --ignore-not-found +kubectl -n redis-perf delete secret redis-perf-auth --ignore-not-found +kubectl taint node replace-with-loadgen-node redis-perf/loadgen=true:NoSchedule- +kubectl label node replace-with-loadgen-node redis-perf/loadgen- +``` + +Confirm the operator's PersistentVolumeClaim retention behavior before deleting volumes. Do not delete retained data unless the namespace and volumes are dedicated to this test and the customer has approved removal. + +## Method and source traceability + +This procedure is traceable to the following implementations; customers do not need these source repositories to run it: + +- Redis Operator: `tests/perf/config.go`, `tests/perf/scenarios.go`, `tests/perf/helpers_test.go`, and `tests/framework/bench/{runner,metrics,parse,report}.go`; `internal/builder/clusterbuilder/configmap.go` and `internal/builder/failoverbuilder/configmap.go` derive an unset `maxmemory` from the Redis container memory limit and configured eviction policy; `internal/builder/clusterbuilder/configmap.go` and `internal/controller/middleware/redis/rediscluster.go` define `io-threads` and `io-threads-do-reads` as restart-required custom configuration. +- Internal validation repository: `performance_case/middleware/redis/test_cluster.py` and `test_sentinel.py` for the 4-CPU/8-GiB sizing, persistence/value/pipeline matrix, long request counts, and CPU/memory/network collection; `performance_case/staging/performance_testing.py` uses `sc-topolvm` as the example performance-test `StorageClass`. +- Product parameter templates: `middleware/charts/redis-param/templates/redis-7.2-paramtemplate-{0,1,2}.yaml` and `redis-7.2-paramdefinition.yaml` define the shipped RDB, no-persistence, and AOF baselines. `redis-frontend/src/app/components/params-template-select/{component.ts,template.html}`, `components/form/component.ts`, and `modules/meepo-shared/src/lib/utils/param-template.ts` define default selection, identifier display, localized descriptions, and application of template values to the form. +- Product schema and monitoring: `api/middleware/v1/redis_types.go`, `config/samples/middleware.alauda.io_v1_redis.yaml`, and `resources/dashboard/data/redis-dashboard.yaml`. +- Community references: [Redis benchmark](https://redis.io/docs/latest/operate/oss_and_stack/management/optimization/benchmarks/), [Redis 7.2 reference configuration](https://github.com/redis/redis/blob/7.2/redis.conf), [Redis `INFO`](https://redis.io/docs/latest/commands/info/), [Redis persistence](https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/), [`BGSAVE`](https://redis.io/docs/latest/commands/bgsave/), [`BGREWRITEAOF`](https://redis.io/docs/latest/commands/bgrewriteaof/), [Redis Cluster health](https://redis.io/docs/latest/commands/cluster-info/), [Redis Sentinel](https://redis.io/docs/latest/operate/oss_and_stack/management/sentinel/), [TopoLVM](https://github.com/topolvm/topolvm), [Kubernetes Storage Classes](https://kubernetes.io/docs/concepts/storage/storage-classes/), [Ceph architecture](https://docs.ceph.com/en/latest/architecture/), [Kubernetes resource metrics](https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/), and [Kubernetes Pod QoS](https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/). From 224a43056ea0ffee9021e38cc883e672b2f3da07 Mon Sep 17 00:00:00 2001 From: Seer Date: Wed, 29 Jul 2026 10:10:28 -0400 Subject: [PATCH 2/8] fix: use supported code fence for PromQL --- .../ecosystem/redis/Redis_Performance_Testing_Guide.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/en/solutions/ecosystem/redis/Redis_Performance_Testing_Guide.md b/docs/en/solutions/ecosystem/redis/Redis_Performance_Testing_Guide.md index f3d70b1b8..e1b462906 100644 --- a/docs/en/solutions/ecosystem/redis/Redis_Performance_Testing_Guide.md +++ b/docs/en/solutions/ecosystem/redis/Redis_Performance_Testing_Guide.md @@ -429,7 +429,7 @@ kubectl get --raw '/apis/metrics.k8s.io/v1beta1/namespaces/redis-perf/pods' \ Export range data covering at least the benchmark start/end time. Adapt label selectors to the installed monitoring stack: -```promql +```text sum by (pod) (rate(container_cpu_usage_seconds_total{namespace="redis-perf",container!=""}[5m])) sum by (pod) (container_memory_working_set_bytes{namespace="redis-perf",container!=""}) sum by (pod) (rate(container_network_receive_bytes_total{namespace="redis-perf"}[5m])) @@ -443,7 +443,7 @@ the report. Calculate utilization against declared container limits with a ratio of sums: -```promql +```text 100 * sum(rate(container_cpu_usage_seconds_total{namespace="redis-perf",container="redis"}[5m])) / From 319e942889bf60748298fd44f885fdf2dc437ba1 Mon Sep 17 00:00:00 2001 From: Seer Date: Wed, 29 Jul 2026 10:13:33 -0400 Subject: [PATCH 3/8] docs: remove private implementation references --- .../redis/Redis_Performance_Testing_Guide.md | 34 +++++++++---------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/docs/en/solutions/ecosystem/redis/Redis_Performance_Testing_Guide.md b/docs/en/solutions/ecosystem/redis/Redis_Performance_Testing_Guide.md index e1b462906..7c496a34a 100644 --- a/docs/en/solutions/ecosystem/redis/Redis_Performance_Testing_Guide.md +++ b/docs/en/solutions/ecosystem/redis/Redis_Performance_Testing_Guide.md @@ -20,14 +20,14 @@ There is no hardware-independent "best" result. Define the workload, durability Use one of two access-path profiles and never merge their results: -- **In-cluster capacity profile (recommended)**: run the load Pod in the Redis cluster and connect through the ClusterIP Service. This follows the current Redis Operator suite and removes external load-balancer/wide area network (WAN) capacity from the Redis result. -- **External end-to-end profile**: run the same open-source client image from the customer-selected source cluster or host through NodePort or LoadBalancer. This follows the cross-cluster capability of the earlier validation suite. Record the client location, access type, round-trip latency, packet loss, and every network hop. For Cluster mode, every node address announced by Cluster discovery must be reachable from the load generator; reaching only the seed endpoint is insufficient. +- **In-cluster capacity profile (recommended)**: run the load Pod in the Redis cluster and connect through the ClusterIP Service. This removes external load-balancer/wide area network (WAN) capacity from the Redis result. +- **External end-to-end profile**: run the same open-source client image from the customer-selected source cluster or host through NodePort or LoadBalancer. Record the client location, access type, round-trip latency, packet loss, and every network hop. For Cluster mode, every node address announced by Cluster discovery must be reachable from the load generator; reaching only the seed endpoint is insufficient. ## Test design Use the following workload matrix for both Cluster and Sentinel. -Redis Database (RDB) persistence, Append Only File (AOF) persistence, and a non-persistent cache are the three profiles. Select a product parameter template first, keep it unchanged for the baseline run, and then change only one tuning axis at a time. The value-size, pipeline, command, and keyspace rows reproduce the Redis Operator performance suite and earlier validation cases. +Redis Database (RDB) persistence, Append Only File (AOF) persistence, and a non-persistent cache are the three profiles. Select a product parameter template first, keep it unchanged for the baseline run, and then change only one tuning axis at a time. | Axis | Standard values | Purpose | |---|---|---| @@ -99,7 +99,7 @@ Prepare the following before the test window: Use a distinct password Secret for each fresh Redis instance. The current Operator can attach the supplied Secret to an instance-owned `RedisUser`; deleting the instance then removes that Secret through Kubernetes ownership. Before creating the next matrix instance, create its new Secret and recreate the load-generator Pod so its projected volume references the new name. Do not assume that a Secret shared with a deleted instance will remain available. -- An approved [Redis Docker Official Image](https://hub.docker.com/_/redis/) containing `redis-benchmark` and `redis-cli`. Match the server major/minor version where practical, mirror the exact image into the customer registry for an air-gapped cluster, and record its immutable digest. Do not use an Alauda-internal load-generator image. +- An approved [Redis Docker Official Image](https://hub.docker.com/_/redis/) containing `redis-benchmark` and `redis-cli`. Match the server major/minor version where practical, mirror the exact image into the customer registry for an air-gapped cluster, and record its immutable digest. Do not depend on a private vendor registry for the load-generator image. - Metrics Server for sampled Pod CPU/memory, or Prometheus for time-series CPU, memory, network, storage, and Redis exporter metrics. Before applying examples, confirm the installed schema with: @@ -201,7 +201,7 @@ The standard matrix fits within the template capacities because it has a bounded Redis serialization, AOF protocol framing, key names, and other files are not included in the payload-only peak. The safety factor provides space for these items and for variation around the bounded test. Redis writes a temporary RDB before replacing the old file. Redis 7.2 multipart AOF retains the current file set while rewrite creates a new base and continues accepting writes, which is why both generations are included before applying the factor. -Use a fresh instance for every parameter-template, value-size, pipeline, and Redis I/O-thread scenario, as the Operator performance suite does. During a client-concurrency sweep on an AOF instance, either recreate the instance for every measured point or run `BGREWRITEAOF` after warm-up and after each point, wait for `aof_rewrite_in_progress:0`, and require `aof_last_bgrewrite_status:ok` before continuing. Run this housekeeping outside the measured interval. +Use a fresh instance for every parameter-template, value-size, pipeline, and Redis I/O-thread scenario. During a client-concurrency sweep on an AOF instance, either recreate the instance for every measured point or run `BGREWRITEAOF` after warm-up and after each point, wait for `aof_rewrite_in_progress:0`, and require `aof_last_bgrewrite_status:ok` before continuing. Run this housekeeping outside the measured interval. If `-n`, `-r`, `-d`, or `S` changes, recompute the capacity before creating the instance. Do not reuse the standard values for the earlier 5-30-million-request Ares cases or a customer workload without recalculation. Multiply the per-Pod result by two for the standard Sentinel topology or by six for the standard three-primary, one-replica-per-primary Cluster topology to obtain total Kubernetes-requested capacity. Storage-system replication, thin-provisioning reserve, and filesystem overhead are additional. For local block storage, every eligible node must have enough allocatable local capacity for the Redis data Pods that can be scheduled there. Provision the calculated capacity before testing; do not resize a PVC during a measured run. @@ -389,11 +389,11 @@ test "$(wc -l < "${CSV_OUTPUT}")" -ge 3 | Parameter | Meaning and selection rule | |---|---| -| `-c` | Parallel connections. Start at `50` (the community default) or `100` (the Operator suite default), then increase. Earlier internal validation used `200` for Cluster and `1000` for Sentinel as high-concurrency reference points. | -| `-n` | Requests per selected operation. Use `1,000,000` for a quick measured run. Earlier internal validation used 5-30 million depending on topology/value size. Increase it until the measurement contains enough monitoring samples and reaches steady state. | +| `-c` | Parallel connections. Start at `50` (the community default) or `100`, then increase until throughput stops scaling or another measured limit is reached. | +| `-n` | Requests per selected operation. Use `1,000,000` for a quick measured run. Increase it until the measurement contains enough monitoring samples and reaches steady state. | | `-d` | `SET`/`GET` value size in bytes. Use `512` and `4096` (4 KiB) for the standard matrix. Add separately named sizes when the application's payload distribution differs. | | `-P` | Pipeline depth. `1` is no pipelining; `16` is the standard throughput-oriented point. Use the application's average pipeline depth for a production-realistic test. Never compare `P=1` and `P=16` as if they were the same workload. | -| `--threads` | Load-generator threads, independent of Redis `io-threads`. Begin with `4`; increase only if the client Pod is CPU-bound before Redis. Earlier internal reference values were `16` for Cluster and `8` for Sentinel. During a server I/O-thread sweep, provision enough client threads for the largest candidate and then hold this value constant. | +| `--threads` | Load-generator threads, independent of Redis `io-threads`. Begin with `4`; increase only if the client Pod is CPU-bound before Redis. During a server I/O-thread sweep, provision enough client threads for the largest candidate and then hold this value constant. | | `-r` | Random keyspace. `100000` is the implemented baseline; enlarge it when the real dataset has a larger working set. | | `-t` | Commands to test. `set,get` is the standard comparison. Add a separate scenario for the application's actual command mix; do not mix incomparable command sets in one result table. | | `--cluster` | Required for Redis Cluster so the tool discovers and routes across Cluster nodes. Omit for Sentinel. All announced Cluster node addresses must be reachable from the client Pod. | @@ -419,7 +419,7 @@ Repeat runs until results are reproducible and retain all runs. Do not report on ### Kubernetes resource metrics -The Operator test harness samples the standard `metrics.k8s.io` API every five seconds and records maximum/average CPU and memory. Metrics Server does not provide network, storage, or historical series; use Prometheus for those metrics. +For short benchmark phases, sample the standard `metrics.k8s.io` API every five seconds and calculate maximum and average CPU and memory. Metrics Server does not provide network, storage, or historical series; use Prometheus for those metrics. ```bash kubectl -n redis-perf top pod --containers @@ -607,12 +607,12 @@ kubectl label node replace-with-loadgen-node redis-perf/loadgen- Confirm the operator's PersistentVolumeClaim retention behavior before deleting volumes. Do not delete retained data unless the namespace and volumes are dedicated to this test and the customer has approved removal. -## Method and source traceability +## Community references -This procedure is traceable to the following implementations; customers do not need these source repositories to run it: - -- Redis Operator: `tests/perf/config.go`, `tests/perf/scenarios.go`, `tests/perf/helpers_test.go`, and `tests/framework/bench/{runner,metrics,parse,report}.go`; `internal/builder/clusterbuilder/configmap.go` and `internal/builder/failoverbuilder/configmap.go` derive an unset `maxmemory` from the Redis container memory limit and configured eviction policy; `internal/builder/clusterbuilder/configmap.go` and `internal/controller/middleware/redis/rediscluster.go` define `io-threads` and `io-threads-do-reads` as restart-required custom configuration. -- Internal validation repository: `performance_case/middleware/redis/test_cluster.py` and `test_sentinel.py` for the 4-CPU/8-GiB sizing, persistence/value/pipeline matrix, long request counts, and CPU/memory/network collection; `performance_case/staging/performance_testing.py` uses `sc-topolvm` as the example performance-test `StorageClass`. -- Product parameter templates: `middleware/charts/redis-param/templates/redis-7.2-paramtemplate-{0,1,2}.yaml` and `redis-7.2-paramdefinition.yaml` define the shipped RDB, no-persistence, and AOF baselines. `redis-frontend/src/app/components/params-template-select/{component.ts,template.html}`, `components/form/component.ts`, and `modules/meepo-shared/src/lib/utils/param-template.ts` define default selection, identifier display, localized descriptions, and application of template values to the form. -- Product schema and monitoring: `api/middleware/v1/redis_types.go`, `config/samples/middleware.alauda.io_v1_redis.yaml`, and `resources/dashboard/data/redis-dashboard.yaml`. -- Community references: [Redis benchmark](https://redis.io/docs/latest/operate/oss_and_stack/management/optimization/benchmarks/), [Redis 7.2 reference configuration](https://github.com/redis/redis/blob/7.2/redis.conf), [Redis `INFO`](https://redis.io/docs/latest/commands/info/), [Redis persistence](https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/), [`BGSAVE`](https://redis.io/docs/latest/commands/bgsave/), [`BGREWRITEAOF`](https://redis.io/docs/latest/commands/bgrewriteaof/), [Redis Cluster health](https://redis.io/docs/latest/commands/cluster-info/), [Redis Sentinel](https://redis.io/docs/latest/operate/oss_and_stack/management/sentinel/), [TopoLVM](https://github.com/topolvm/topolvm), [Kubernetes Storage Classes](https://kubernetes.io/docs/concepts/storage/storage-classes/), [Ceph architecture](https://docs.ceph.com/en/latest/architecture/), [Kubernetes resource metrics](https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/), and [Kubernetes Pod QoS](https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/). +- [Redis benchmark](https://redis.io/docs/latest/operate/oss_and_stack/management/optimization/benchmarks/) +- [Redis 7.2 reference configuration](https://github.com/redis/redis/blob/7.2/redis.conf) +- [Redis `INFO`](https://redis.io/docs/latest/commands/info/) +- [Redis persistence](https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/), [`BGSAVE`](https://redis.io/docs/latest/commands/bgsave/), and [`BGREWRITEAOF`](https://redis.io/docs/latest/commands/bgrewriteaof/) +- [Redis Cluster health](https://redis.io/docs/latest/commands/cluster-info/) and [Redis Sentinel](https://redis.io/docs/latest/operate/oss_and_stack/management/sentinel/) +- [TopoLVM](https://github.com/topolvm/topolvm), [Kubernetes Storage Classes](https://kubernetes.io/docs/concepts/storage/storage-classes/), and [Ceph architecture](https://docs.ceph.com/en/latest/architecture/) +- [Kubernetes resource metrics](https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/) and [Kubernetes Pod QoS](https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/) From f222afd8c89894a2af7c8b9a6a818c4b1374e348 Mon Sep 17 00:00:00 2001 From: Richard Feng Date: Wed, 29 Jul 2026 20:24:54 -0400 Subject: [PATCH 4/8] docs: add aKafka performance testing guide --- .../kafka/aKafka_Performance_Testing_Guide.md | 1035 +++++++++++++++++ 1 file changed, 1035 insertions(+) create mode 100644 docs/zh/solutions/ecosystem/kafka/aKafka_Performance_Testing_Guide.md diff --git a/docs/zh/solutions/ecosystem/kafka/aKafka_Performance_Testing_Guide.md b/docs/zh/solutions/ecosystem/kafka/aKafka_Performance_Testing_Guide.md new file mode 100644 index 000000000..f2735212c --- /dev/null +++ b/docs/zh/solutions/ecosystem/kafka/aKafka_Performance_Testing_Guide.md @@ -0,0 +1,1035 @@ +--- +products: + - Alauda Application Services +kind: + - Solution +--- + +# aKafka 性能测试指南 + +## 1. 目的和适用范围 + +本文用于在 Alauda Container Platform（ACP）上对 aKafka 实例执行可重复、可审计的性能测试，并回答以下问题： + +本文沿用 Apache Kafka 官方术语：服务端代理节点（Broker）、主题（Topic）、控制器（Controller）和同步副本集合（in-sync replicas，ISR）。 + +- 在给定副本、确认和压缩策略下，实例可持续处理的生产、消费吞吐量是多少； +- 在延迟、错误率、副本同步和资源水位满足约束时，实例的可用容量是多少； +- 瓶颈位于客户端、Broker 的 CPU/内存/网络、持久卷，还是分区与副本布局； +- 调整实例或客户端参数后，结果是否可比，收益是否可复现。 + +本文的命令按 Apache Kafka 4.2.0 的脚本接口编写。其他版本必须先保存 `--version` 和 `--help` 输出，再按该版本支持的参数调整。特别是，Kafka 4.2 的 `kafka-consumer-perf-test.sh` 没有旧版本的 `--threads` 参数；需要并发时，应增加独立进程或 Pod。 + +本文不提供跨环境通用的“最佳吞吐量”或“最佳参数”。硬件、存储、网络、消息大小、压缩率、可靠性策略和业务负载任一项变化，结果都可能失去可比性。 + +> **风险提示**：压测会大量写入数据，并可能耗尽磁盘、网络或节点资源。只在获得授权的隔离实例、命名空间和时间窗口内执行。不要在承载生产流量的实例上寻找饱和点。 + +## 2. 结果口径 + +测试前必须选择一种负载模式和一种可靠性配置。不同配置的结果不得合并比较。 + +### 2.1 负载模式 + +| 模式 | 目的 | 执行方式 | 主要结果 | +| --- | --- | --- | --- | +| 生产者单测 | 测量写入能力 | 只运行生产者 | records/s、MiB/s、生产确认延迟 | +| 消费者单测 | 测量读取能力 | 先准备足量存量数据，再运行消费者 | records/s、MiB/s、消费组积压 | +| 生产与消费并发 | 模拟持续流转 | 先启动消费者，再启动生产者 | 两端吞吐量、积压增长、Broker 资源和稳定性 | + +生产者脚本报告的延迟是生产请求确认延迟，不是业务端到端延迟。消费者脚本报告吞吐量，但不提供消息从生产到消费的端到端延迟。端到端延迟必须在消息中写入发送时间戳，并由业务压测程序在消费端计算；不要用两段脚本输出相减代替。 + +### 2.2 可靠性配置 + +| 配置 | Topic | 生产者 | 用途 | +| --- | --- | --- | --- | +| 生产可靠性基线 | `replication.factor=3`、`min.insync.replicas=2` | `acks=all`、`enable.idempotence=true` | 容量规划和生产验收的默认配置 | +| Leader 确认上限 | `replication.factor=3`，其余参数与被测基线一致 | `acks=1`、`enable.idempotence=false` | 估算降低确认强度后的吞吐上限 | + +`acks=1` 只等待 Leader 确认；Leader 随后立即故障时，尚未复制的数据可能丢失。Kafka 的幂等生产要求 `acks=all`，因此执行 `acks=1` 测试时必须显式设置 `enable.idempotence=false`，避免客户端配置冲突。容量结论必须标注可靠性配置，不能把 Leader 确认上限作为生产可靠性容量。 + +### 2.3 访问路径 + +外部连接可能使用传输层安全性（Transport Layer Security，TLS）和简单认证与安全层（Simple Authentication and Security Layer，SASL）；测试必须使用与业务计划相同的安全配置。 + +| 访问路径 | 客户端位置 | 结果包含的约束 | 使用场景 | +| --- | --- | --- | --- | +| 集群内容量 | 业务集群内的专用压测节点，通过内部 Bootstrap Service 连接 | Kafka、集群网络和存储 | 实例容量和调参，本文命令默认使用该路径 | +| 外部端到端 | 客户指定的源集群或主机，通过实际外部监听器连接 | 以上约束，加上负载均衡器、广域网、防火墙、TLS/SASL 和客户端出口 | 业务接入验收 | + +两种路径的结果不能合并。外部端到端测试必须保存客户端位置、监听器、往返时延、丢包和网络路径，并确认客户端不仅能访问 Bootstrap 地址，还能访问 Kafka 元数据中公布的每个 Broker 地址。完成 4.3 节的变量初始化后，从当前资源状态读取监听器地址，不应按命名规则推测： + +```bash +kubectl -n "$NS" get kafka "$CLUSTER" \ + -o jsonpath='{range .status.listeners[*]}{.name}{"\t"}{.bootstrapServers}{"\n"}{end}' +``` + +## 3. 标准测试矩阵 + +先执行标准矩阵以获得横向可比的数据，再补充与真实业务一致的消息模型、连接安全和并发模式。 + +### 3.1 实例和消息矩阵 + +| Broker 资源规格 | Broker 数量 | Topic 分区 | 副本数 | 消息大小 | +| --- | ---: | ---: | ---: | --- | +| 1 vCPU / 2 GiB | 3 | 30 | 3 | 100 B、500 B、1000 B | +| 2 vCPU / 4 GiB | 3 | 30 | 3 | 100 B、500 B、1000 B | +| 4 vCPU / 8 GiB | 3 | 30 | 3 | 100 B、500 B、1000 B | +| 8 vCPU / 16 GiB | 3 | 30 | 3 | 100 B、500 B、1000 B | + +标准点位使用以下记录数： + +- 1、2、4 vCPU：每个消息大小发送 50,000,000 条； +- 8 vCPU：100 B 和 500 B 各发送 100,000,000 条，1000 B 发送 50,000,000 条。 + +标准生产者点先使用 1 个生产者进程；如果第 12 节证明客户端先达到上限，则增加进程并把聚合结果单独标记。标准消费者点的目标并发为 20 个消费者。Kafka 4.2 不再提供旧版脚本的 `--threads 20`，因此应运行 20 个独立进程，并让它们使用同一消费组；这些进程可分布到多个资源充足的 Pod，但必须记录每个 Pod 的进程数和资源。 + +如果测试窗口或磁盘容量不足，可以减少记录数，但每个稳态点应持续至少 10 分钟，并在报告中记录实际记录数和持续时间。短时突发结果不能替代稳态容量。 + +### 3.2 业务补充矩阵 + +标准矩阵之外，至少增加一组与生产计划一致的配置： + +- 消息大小分布，而不是只使用平均值； +- 消息键（record key）分布和分区策略； +- `compression.type`； +- TLS、SASL 等连接安全； +- 生产者和消费者实例数； +- Topic 数量、分区数、副本数和保留时间； +- 实际的突发与空闲周期。 + +如果没有业务分布数据，应将结论标记为“合成负载结果”，不能推断为业务容量。 + +## 4. 测试前置条件 + +### 4.1 环境隔离 + +1. 创建专用命名空间、aKafka 实例、Topic、消费组和压测客户端。 +2. Broker 应分散到不同 Kubernetes 节点；压测客户端不要与 Broker 共用节点，否则客户端争抢资源会污染结果。 +3. 测试期间不得运行扩缩容、版本升级、节点维护、存储迁移或其他高负载任务。 +4. 节点和持久卷类型必须与目标环境一致。生产容量测试优先使用本地块存储或经验证的块存储；不要把未经验证的共享文件存储结果外推到块存储环境。 +5. 每组实例参数测试使用新实例，或在清理 Topic 后等待实例恢复稳定。修改需要重启的参数后，必须等待所有 Pod Ready、副本同步完成，再开始下一轮。 + +### 4.2 必需工具和权限 + +- 可访问业务集群的 `kubectl`； +- 对测试命名空间内 aKafka、Pod、ConfigMap、Secret、Service 和持久卷声明（PersistentVolumeClaim，PVC）的必要权限； +- 可读取平台监控或 Prometheus 时序数据； +- 可从已部署 Broker Pod 读取其 Kafka 镜像名称。本文复用 Operator 管理的 Kafka 镜像，不要求任何私有镜像名称； +- 用于 SASL/TLS 的客户端配置 Secret。报告不得包含密码、令牌、私钥或完整 Secret。 + +### 4.3 建立证据目录 + +在同一个 Bash 会话中执行以下初始化；后续代码块沿用这些变量和错误处理选项： + +```bash +set -euo pipefail + +export KUBECONFIG=/path/to/business-cluster.kubeconfig +export NS=akafka-perf +export CLUSTER=perf-kafka +export TOPIC=perf-rf3-p30 +export GROUP="perf-$(date -u +%Y%m%dT%H%M%SZ)" +export RUN_ID="$(date -u +%Y%m%dT%H%M%SZ)-${CLUSTER}" +export OUT="$(pwd)/akafka-perf-${RUN_ID}" +mkdir -p "$OUT" + +kubectl version -o yaml >"$OUT/kubectl-version.yaml" +kubectl cluster-info >"$OUT/cluster-info.txt" +kubectl get nodes -o wide >"$OUT/nodes.txt" +kubectl get storageclass -o yaml >"$OUT/storageclasses.yaml" +``` + +只导出非敏感元数据。不要把 kubeconfig、Secret 数据、仓库登录信息或客户端密码写入证据目录。 + +## 5. 导出并冻结实例参数 + +aKafka 的通用参数模板名称通常为 `general-kafka`。模板属于管理集群的产品配置，业务集群中的 `RdsKafka` 只保存最终生效的实例配置。不同产品版本的模板可能变化，因此每次测试都应导出当前安装版本，而不是依赖本文表格。 + +如果具备管理集群只读权限，执行： + +```bash +export MGMT_KUBECONFIG=/path/to/management-cluster.kubeconfig + +kubectl --kubeconfig "$MGMT_KUBECONFIG" get paramtemplate -A \ + -l component=kafka + +export TEMPLATE_NS="$(kubectl --kubeconfig "$MGMT_KUBECONFIG" \ + get paramtemplate -A -l component=kafka \ + -o jsonpath='{range .items[?(@.metadata.name=="general-kafka")]}{.metadata.namespace}{"\n"}{end}')" +test -n "$TEMPLATE_NS" +kubectl --kubeconfig "$MGMT_KUBECONFIG" -n "$TEMPLATE_NS" \ + get paramtemplate general-kafka -o yaml \ + >"$OUT/general-kafka-paramtemplate.yaml" +``` + +没有管理集群权限时，在产品控制台选择通用 Kafka 参数模板，并从实例导出最终 `spec.config`。不要根据模板名称推测值。 + +当前标准通用模板应重点核对以下参数；“示例基线”仅用于识别参数，不替代已安装模板： + +| Broker 参数 | 示例基线 | 作用与测试解释 | +| --- | ---: | --- | +| `auto.create.topics.enable` | `false` | 禁止因拼写错误自动建 Topic；测试 Topic 必须显式创建 | +| `default.replication.factor` | `3` | 未显式指定副本数时的默认值；测试仍显式传入 `3` | +| `min.insync.replicas` | `1` | 模板基线；生产可靠性测试应在 Topic 或实例层显式设为 `2` | +| `num.network.threads` | `3` | 处理网络请求的线程数 | +| `num.io.threads` | `8` | 处理请求和磁盘相关工作的线程数 | +| `num.replica.fetchers` | `1` | 每个源 Broker 的副本拉取线程数 | +| `num.recovery.threads.per.data.dir` | `1` | 每个数据目录用于启动恢复和关闭刷新的线程数 | +| `background.threads` | `10` | 后台任务线程池大小 | +| `message.max.bytes` | `1048588` | Broker 接受的最大记录批次大小，不等同于单条消息大小 | +| `compression.type` | `producer` | Broker 保留生产者选择的压缩类型 | +| `log.segment.bytes` | `1073741824` | 日志段大小 | +| `log.retention.hours` | `168` | 基于时间的保留期 | +| `log.roll.hours` | `168` | 日志段最长滚动时间 | +| `delete.topic.enable` | `true` | 允许删除测试 Topic | +| `unclean.leader.election.enable` | `false` | 禁止非 ISR 副本被选为 Leader | + +参数模板的变更应用策略由当前产品版本决定。测试前查询当前版本的参数定义，记录其 `applyStrategy`；如果是 `RestartApply`，把滚动重启时间排除在测量窗口之外。 + +```bash +kubectl --kubeconfig "$MGMT_KUBECONFIG" get paramdefinition kafka-general -o yaml \ + >"$OUT/kafka-general-paramdefinition.yaml" +``` + +## 6. 持久卷容量和存储验证 + +### 6.1 有界合成测试的容量公式 + +对单个 Topic，可先用下式估算每个 Broker 的消息载荷占用： + +```text +每 Broker 载荷字节 ≈ 记录数 × 单条记录字节 × 副本数 ÷ Broker 数 ÷ 实测压缩比 +建议 PVC 下限 = 每 Broker 载荷字节 × 安全系数 + 其他 Topic、索引和运维预留 +``` + +压缩比定义为“未压缩字节 / 落盘字节”。预估阶段没有实测值时取 `1`，不要假设压缩收益。本文对全新、单 Topic、有界合成测试采用 `1.5` 的工程安全系数；该系数不是 Kafka 产品默认值，也不适用于生产容量规划。存在数据倾斜、重分配、多个 Topic 或保留期内持续写入时，应根据实测峰值提高系数，并单独预留迁移空间。 + +在 3 Broker、3 副本、压缩比为 1 的标准矩阵中，每条记录的一个副本平均落在每个 Broker，因此载荷和建议下限如下： + +| 记录数 | 消息大小 | 每 Broker 载荷 | 乘 1.5 后的最低取整值 | +| ---: | ---: | ---: | ---: | +| 50,000,000 | 100 B | 4.66 GiB | 7 GiB | +| 50,000,000 | 500 B | 23.28 GiB | 35 GiB | +| 50,000,000 | 1000 B | 46.57 GiB | 70 GiB | +| 100,000,000 | 100 B | 9.31 GiB | 14 GiB | +| 100,000,000 | 500 B | 46.57 GiB | 70 GiB | + +表中未单独计算 Key、Record Header、批次开销、索引、控制记录、其他 Topic 和重分配临时空间。因此 50 GiB PVC 不足以安全执行 50,000,000 条 1000 B 或 100,000,000 条 500 B 的完整标准点位。无限时长或未知写入量的测试不能用该有界公式，应按写入速率、保留时间和目标磁盘水位计算。 + +独立 KRaft Controller 不保存 Topic 消息载荷，不能套用 Broker 载荷公式。其 PVC 应满足当前产品版本的最小值，并根据 Topic/分区数量变化、元数据日志增长和保留策略实测校准。Controller/Broker 混合角色共用数据卷时，PVC 必须同时满足 Broker 载荷下限和元数据、索引及运维预留。 + +### 6.2 存储检查 + +```bash +kubectl -n "$NS" get pvc -o wide >"$OUT/pvc-before.txt" +kubectl -n "$NS" get pod -o wide >"$OUT/pods-before.txt" + +# 用当前实例实际标签筛选；不要假设 Pod 名称。 +kubectl -n "$NS" get pod \ + -l "strimzi.io/cluster=${CLUSTER},strimzi.io/broker-role=true" \ + -o wide +``` + +确认： + +- PVC 全部为 `Bound`，容量不小于计算值； +- 每个 Broker 的持久卷和节点分布符合测试设计； +- StorageClass、卷类型、文件系统和节点磁盘与目标环境一致； +- 节点不存在持续磁盘压力，且压测前 Kafka 没有待恢复副本。 + +存储延迟是 Kafka 性能的主要约束之一。只报告 StorageClass 名称不足以证明存储等价，应同时保存供应器、介质类型、拓扑和测试期间的 I/O 延迟/利用率证据。 + +## 7. 创建或调整 aKafka 实例 + +通过产品控制台创建专用实例，或修改已有的专用测试实例。控制台字段随产品版本可能变化，最终以业务集群中的 `RdsKafka` 对象和生成的 Strimzi 资源为准。 + +### 7.1 拓扑 + +- 使用 Kafka Raft 元数据模式（KRaft）； +- 至少 3 个 Broker； +- 生产形态使用独立 Controller 时，性能测试也应使用相同拓扑；生产形态使用 Controller/Broker 混合角色时，必须在报告中明确； +- Broker 通过 Pod 反亲和性分散到不同节点和故障域； +- Broker CPU/内存按第 3 节矩阵逐档测试；Controller 资源固定，避免与 Broker 规格同时变化； +- Java 虚拟机（Java Virtual Machine，JVM）堆上限不能占满容器内存，必须为页缓存和容器内的非堆内存留出空间。 + +> Kafka 依赖操作系统页缓存。增大 JVM 堆不等于增大 Kafka 可用缓存；如果堆设置与容器限制过近，可能导致容器内存不足或压缩页缓存空间。 + +控制台提交后，应按当前 `RdsKafka` 自定义资源定义（CustomResourceDefinition，CRD）核对以下字段，不根据控制台显示值推测最终配置： + +| `RdsKafka` 字段 | 标准测试要求 | +| --- | --- | +| `spec.mode` | `KRaft` | +| `spec.replicas` | 3 个 Broker | +| `spec.resources.requests/limits` | 当前矩阵档位；requests 和 limits 均需记录 | +| `spec.storage.class` | 客户选定并已验证的 StorageClass | +| `spec.storage.size` | 不小于第 6 节计算值 | +| `spec.storage.deleteClaim` | 按数据保留和销毁要求选择；不得仅为方便清理而改为 `true` | +| `spec.controller` | 与生产计划一致的角色、数量、资源和存储；独立 Controller 使用不少于 3 个投票节点 | +| `spec.config` | 已导出的 `general-kafka` 基线，加上本轮唯一的参数变更 | +| `spec.kafka.listeners` | 与本轮集群内或外部访问路径一致 | +| `spec.kafkaExporter` | 需要消费组积压时启用，并限制 Topic/消费组正则范围 | + +```bash +kubectl explain rdskafka.spec --recursive \ + >"$OUT/rdskafka-spec-schema.txt" +``` + +### 7.2 固定变量 + +每轮测试只改变一个待评估变量。以下配置必须固定并写入报告： + +- Kafka 与 Operator 版本； +- KRaft 角色布局、Broker/Controller 数量； +- Broker/Controller 的 CPU、内存、JVM 堆、节点选择和反亲和性； +- StorageClass、PVC 容量和存储介质； +- Broker 配置完整快照； +- Topic 分区、副本和 `min.insync.replicas`； +- 客户端镜像版本、资源、节点位置和连接安全； +- 消息大小、记录数、压缩、消息键分布、确认和消费参数。 + +保存实例最终状态： + +```bash +kubectl -n "$NS" get rdskafka "$CLUSTER" -o yaml \ + >"$OUT/rdskafka.yaml" +kubectl -n "$NS" get kafka "$CLUSTER" -o yaml \ + >"$OUT/kafka.yaml" +kubectl -n "$NS" get kafkanodepool -l "strimzi.io/cluster=${CLUSTER}" -o yaml \ + >"$OUT/kafkanodepools.yaml" +kubectl -n "$NS" get pod,pvc,service -o wide \ + >"$OUT/workloads-before.txt" +``` + +任一采集命令失败都应停止并排查权限、资源名称或实例状态，不能用 `|| true` 掩盖错误。 + +### 7.3 监控开关 + +创建实例时启用平台提供的 Kafka 指标采集；需要消费组积压时同时启用 Kafka Exporter，并把 Topic 和消费组正则限制在本次测试范围。开始压测前核对最终资源和 Exporter Pod： + +```bash +kubectl -n "$NS" get rdskafka "$CLUSTER" \ + -o jsonpath='{.spec.kafkaExporter}' \ + >"$OUT/rdskafka-kafka-exporter.json" +kubectl -n "$NS" get kafka "$CLUSTER" \ + -o jsonpath='{.spec.kafka.metricsConfig}' \ + >"$OUT/kafka-metrics-config.json" +kubectl -n "$NS" get pod \ + -l "strimzi.io/cluster=${CLUSTER},strimzi.io/name=${CLUSTER}-kafka-exporter" \ + -o wide +``` + +如果最后一条命令没有找到 Pod，说明 Kafka Exporter 未启用或标签与当前版本不同。先检查 `RdsKafka`、生成的 `Kafka` 资源和实际 Pod 标签，不要把空结果解释为“积压为 0”。 + +## 8. 部署压测客户端 + +### 8.1 复用实例的 Kafka 镜像 + +从 Broker Pod 读取 Operator 实际使用的 Kafka 镜像。这样可保证脚本版本与服务端版本一致，也不需要在外部文档中固化镜像仓库地址。 + +```bash +export KAFKA_IMAGE="$(kubectl -n "$NS" get pod \ + -l "strimzi.io/cluster=${CLUSTER},strimzi.io/broker-role=true" \ + -o jsonpath='{.items[0].spec.containers[?(@.name=="kafka")].image}')" + +test -n "$KAFKA_IMAGE" +export BOOTSTRAP="${CLUSTER}-kafka-bootstrap.${NS}.svc:9092" +``` + +上面的 Service 名称符合 Strimzi 内部 Bootstrap Service 规则，但仍应查询当前实例验证监听器、端口和认证方式： + +```bash +kubectl -n "$NS" get service -l "strimzi.io/cluster=${CLUSTER}" -o wide +kubectl -n "$NS" get kafka "$CLUSTER" \ + -o jsonpath='{.spec.kafka.listeners}' >"$OUT/listeners.json" +``` + +以下示例使用隔离命名空间内的明文内部监听器。启用 TLS/SASL 时，应把完整客户端属性放入受控 Secret，并以只读卷挂载；不要通过命令行参数传递密码。 + +### 8.2 创建客户端属性 + +```bash +kubectl -n "$NS" create configmap akafka-perf-client-config \ + --from-literal=admin.properties='' \ + --from-literal=producer.properties='acks=all +enable.idempotence=true +batch.size=50000 +linger.ms=5 +compression.type=none +buffer.memory=134217728' \ + --from-literal=consumer.properties='auto.offset.reset=earliest +fetch.min.bytes=1 +fetch.max.wait.ms=500 +max.partition.fetch.bytes=1048576 +fetch.max.bytes=52428800' \ + --dry-run=client -o yaml | kubectl apply -f - +``` + +如果使用认证，在本地受控目录中准备包含性能参数和连接参数的完整 `producer.properties`、`consumer.properties`，以 Secret 替代上面的 ConfigMap： + +```bash +kubectl -n "$NS" create secret generic akafka-perf-client-config \ + --from-file=admin.properties=/secure/path/admin.properties \ + --from-file=producer.properties=/secure/path/producer.properties \ + --from-file=consumer.properties=/secure/path/consumer.properties \ + --dry-run=client -o yaml | kubectl apply -f - +``` + +同时把 8.3 节两个 Pod 的卷源从 `configMap` 改为： + +```yaml +secret: + secretName: akafka-perf-client-config +``` + +`admin.properties` 只保存管理脚本所需的连接安全属性；生产者和消费者文件另外保存各自性能参数。属性中的 `security.protocol`、SASL 或 SSL 设置必须与实例监听器一致。报告和证据目录只保存脱敏副本，不得导出 Secret、密码、令牌、私钥或未脱敏的 Java 身份验证和授权服务（Java Authentication and Authorization Service，JAAS）配置。 + +### 8.3 创建客户端 Pod + +```bash +kubectl -n "$NS" apply -f - <"$OUT/producer-perf-help.txt" 2>&1 + +kubectl -n "$NS" exec akafka-consumer-perf -- \ + /opt/kafka/bin/kafka-consumer-perf-test.sh --help \ + >"$OUT/consumer-perf-help.txt" 2>&1 +``` + +如果镜像中的 Kafka 安装目录不是 `/opt/kafka`，先在镜像说明或 Broker 容器环境中确认实际路径，再替换命令。不要下载另一个未知版本的脚本混用。 + +## 9. 创建和验证测试 Topic + +```bash +kubectl -n "$NS" exec akafka-producer-perf -- \ + /opt/kafka/bin/kafka-topics.sh \ + --bootstrap-server "$BOOTSTRAP" \ + --command-config /opt/perf-config/admin.properties \ + --create --if-not-exists \ + --topic "$TOPIC" \ + --partitions 30 \ + --replication-factor 3 \ + --config min.insync.replicas=2 + +kubectl -n "$NS" exec akafka-producer-perf -- \ + /opt/kafka/bin/kafka-topics.sh \ + --bootstrap-server "$BOOTSTRAP" \ + --command-config /opt/perf-config/admin.properties \ + --describe --topic "$TOPIC" \ + | tee "$OUT/topic-before.txt" + +kubectl -n "$NS" exec akafka-producer-perf -- \ + /opt/kafka/bin/kafka-configs.sh \ + --bootstrap-server "$BOOTSTRAP" \ + --command-config /opt/perf-config/admin.properties \ + --entity-type topics --entity-name "$TOPIC" --describe \ + | tee "$OUT/topic-config.txt" +``` + +开始前逐项确认： + +- `PartitionCount=30`、`ReplicationFactor=3`； +- 每个分区有 3 个副本，ISR 数量为 3； +- Leader 在 Broker 间没有明显失衡； +- 动态 Topic 配置中 `min.insync.replicas=2`； +- Topic 是本轮新建或已按测试设计清理，消费组名称没有复用。 + +## 10. 执行单个测试点 + +### 10.1 生产可靠性基线 + +先记录参数，再执行。示例为 1,000,000 条、每条 500 B 的功能验证；正式标准点位替换 `--num-records`，并确保 PVC 容量足够。 + +```bash +export NUM_RECORDS=1000000 +export RECORD_SIZE=500 + +kubectl -n "$NS" exec akafka-producer-perf -- sh -c \ + "cat /opt/perf-config/producer.properties" \ + >"$OUT/producer.properties" + +kubectl -n "$NS" exec akafka-producer-perf -- \ + /opt/kafka/bin/kafka-producer-perf-test.sh \ + --bootstrap-server "$BOOTSTRAP" \ + --topic "$TOPIC" \ + --num-records "$NUM_RECORDS" \ + --record-size "$RECORD_SIZE" \ + --throughput -1 \ + --warmup-records 100000 \ + --reporting-interval 5000 \ + --command-config /opt/perf-config/producer.properties \ + --print-metrics \ + | tee "$OUT/producer-${RECORD_SIZE}B.txt" +``` + +`--throughput -1` 表示客户端不主动限速，用于寻找该客户端并发下的上限。它不保证 Broker 已饱和；必须结合客户端和 Broker 指标判断。正式对比中应固定 `--warmup-records`，并只使用脚本的稳态汇总结果，不把预热阶段混入结果。 + +### 10.2 Leader 确认上限 + +保持其他属性不变，只覆盖确认和幂等配置： + +```properties +acks=1 +enable.idempotence=false +batch.size=50000 +linger.ms=5 +compression.type=none +buffer.memory=134217728 +``` + +Kafka 4.2 可通过 `--command-property` 覆盖配置文件中的这两个值，避免无意改变其他参数： + +```bash +kubectl -n "$NS" exec akafka-producer-perf -- \ + /opt/kafka/bin/kafka-producer-perf-test.sh \ + --bootstrap-server "$BOOTSTRAP" \ + --topic "$TOPIC" \ + --num-records "$NUM_RECORDS" \ + --record-size "$RECORD_SIZE" \ + --throughput -1 \ + --warmup-records 100000 \ + --reporting-interval 5000 \ + --command-config /opt/perf-config/producer.properties \ + --command-property acks=1 \ + --command-property enable.idempotence=false \ + --print-metrics \ + | tee "$OUT/producer-leader-ack-${RECORD_SIZE}B.txt" +``` + +结果必须标记为 `leader-ack`，且不得与 `acks=all` 的容量直接比较。 + +### 10.3 消费者单测 + +消费者单测前先准备不小于 `NUM_RECORDS` 的存量数据，然后使用新的消费组： + +```bash +export NUM_RECORDS=1000000 +export RECORD_SIZE=500 +export GROUP="perf-consumer-$(date -u +%Y%m%dT%H%M%SZ)" + +kubectl -n "$NS" exec akafka-consumer-perf -- sh -c \ + "cat /opt/perf-config/consumer.properties" \ + >"$OUT/consumer.properties" + +kubectl -n "$NS" exec akafka-consumer-perf -- \ + /opt/kafka/bin/kafka-consumer-perf-test.sh \ + --bootstrap-server "$BOOTSTRAP" \ + --topic "$TOPIC" \ + --group "$GROUP" \ + --num-records "$NUM_RECORDS" \ + --fetch-size 200000 \ + --timeout 60000 \ + --show-detailed-stats \ + --reporting-interval 5000 \ + --command-config /opt/perf-config/consumer.properties \ + --print-metrics \ + | tee "$OUT/consumer-${RECORD_SIZE}B.txt" +``` + +Kafka 4.2 中 `--messages` 已弃用，应使用 `--num-records`。`--fetch-size` 限制每次从单个分区获取的数据量；客户端属性中的 `max.partition.fetch.bytes` 也参与限制。必须保存 `--print-metrics` 输出中的 `records-consumed-total`，确认实际消费量达到目标。超时、Topic 数据不足或权限错误都可能使脚本提前结束，不能只看最后一行吞吐量。 + +### 10.4 生产和消费并发 + +1. 使用新消费组启动消费者，并确认其已加入组； +2. 启动生产者； +3. 在整个稳态窗口采集指标； +4. 生产者停止后继续运行消费者，直到积压归零或达到预定义超时； +5. 保存最终消费位点和日志末端偏移量（log end offset，LEO）。 + +并发测试不能直接把单次消费者命令放在前台后再启动生产者。应在两个终端运行，或通过 Job/测试编排器同时启动并分别保存标准输出。任何后台执行方式都必须保存进程退出码；仅有日志文件不足以证明命令成功。 + +### 10.5 Kafka 4.2 的 20 消费者执行示例 + +以下示例在一个客户端 Pod 中启动 20 个独立消费者进程，并检查每个 `kubectl exec` 的退出码。正式测试前必须根据 12.1 节确认该 Pod 的 CPU、内存和网络不会成为瓶颈；资源不足时，复制 8.3 节的消费者 Pod 并把进程分散到多个压测节点。 + +```bash +export CONSUMER_COUNT=20 +export CONSUMER_POD=akafka-consumer-perf +export GROUP="perf-consumer-20-$(date -u +%Y%m%dT%H%M%SZ)" +test "$NUM_RECORDS" -ge "$CONSUMER_COUNT" + +q=$((NUM_RECORDS / CONSUMER_COUNT)) +r=$((NUM_RECORDS % CONSUMER_COUNT)) +pids=() + +for ((i = 1; i <= CONSUMER_COUNT; i++)); do + process_records="$q" + if ((i <= r)); then + process_records=$((q + 1)) + fi + + kubectl -n "$NS" exec "$CONSUMER_POD" -- \ + /opt/kafka/bin/kafka-consumer-perf-test.sh \ + --bootstrap-server "$BOOTSTRAP" \ + --topic "$TOPIC" \ + --group "$GROUP" \ + --num-records "$process_records" \ + --fetch-size 200000 \ + --timeout 60000 \ + --reporting-interval 5000 \ + --command-config /opt/perf-config/consumer.properties \ + --command-property "client.id=perf-consumer-${i}" \ + --print-metrics \ + >"$OUT/consumer-${i}-${RECORD_SIZE}B.txt" 2>&1 & + pids+=("$!") +done + +failed=0 +for pid in "${pids[@]}"; do + wait "$pid" || failed=1 +done +test "$failed" -eq 0 +``` + +把 20 个文件中的实际消费记录数相加，并核对总数等于 `NUM_RECORDS`。聚合吞吐量按公共测量窗口中的总记录数/总字节数计算；各进程起止时间不一致时，不能直接相加各自打印的平均速率。任一进程超时、退出非零或记录数不符，都应使该测试点无效。 + +## 11. 客户端参数说明 + +### 11.1 生产者 + +| 参数 | 标准值 | 机制和边界 | +| --- | ---: | --- | +| `--num-records` | 见矩阵 | 实际发送条数；总载荷约为该值乘 `--record-size` | +| `--record-size` | 100/500/1000 | 合成记录 Value 的字节数；只可与消息载荷（payload）文件模式二选一 | +| `--throughput` | `-1` | 不限速上探；受控负载测试时填写目标 records/s | +| `--warmup-records` | 固定值 | 排除连接建立、元数据和 JVM 预热影响；不计入稳态结果 | +| `acks` | `all` 或 `1` | 确认强度；改变后可靠性和性能口径同时变化 | +| `enable.idempotence` | `true` | 防止重试导致重复；要求兼容的 `acks`、重试和在途请求设置 | +| `batch.size` | `50000` | 每分区批次的目标上限；过大可能增加内存和等待时间 | +| `linger.ms` | `5` | 等待合批的最长时间；Kafka 4.x 默认值与早期版本不同，测试应显式设置 | +| `compression.type` | `none` | 标准矩阵关闭压缩；业务矩阵使用真实算法并记录 CPU 与实测压缩比 | +| `buffer.memory` | `134217728` | 生产者可用于缓冲待发送记录的总内存；不是容器内存需求的全部 | + +每个分区有独立批次，增加生产者数量、分区数、批次或 `linger.ms` 可能提高合批效率，但也可能增加内存和延迟。只有在报告中保持其余变量不变时，参数前后结果才可归因。 + +### 11.2 消费者 + +| 参数 | 标准值 | 机制和边界 | +| --- | ---: | --- | +| `--group` | 每轮唯一 | 同组消费者共同分配分区；复用组可能从旧位点开始 | +| `--num-records` | 与存量数据匹配 | 计划读取的总记录数；必须用客户端指标核对实际值 | +| `--fetch-size` | `200000` | 脚本传入的单分区抓取大小 | +| `--timeout` | `60000` | 两次返回记录之间允许的最长时间，不是整个测试的总时长；超时会使工具提前退出并打印警告 | +| `fetch.min.bytes` | `1` | Broker 返回 Fetch 的最小数据量；增大可提高批量效率，但可能增加等待延迟 | +| `fetch.max.wait.ms` | `500` | 未达到 `fetch.min.bytes` 时的最长等待时间 | +| `max.partition.fetch.bytes` | `1048576` | 每分区每次 Fetch 的返回上限；要能容纳 Broker 允许的最大记录批次 | +| `fetch.max.bytes` | `52428800` | 单次 Fetch 的总体返回上限，仍受服务端和分区级限制 | + +同一消费组的有效并行度不超过可分配分区数。Kafka 4.2 的脚本每次调用运行一个消费者；需要 2、4、8 个消费者时，创建相同配置的独立 Pod 或进程，并使用同一个消费组。消费者数超过分区数时，多出的消费者不会获得分区。 + +矩阵中的 `NUM_RECORDS` 是全部进程的聚合目标，不是每个进程的目标。并发数为 `C` 时，令 `q=NUM_RECORDS/C`、`r=NUM_RECORDS%C`：前 `r` 个进程使用 `q+1`，其余进程使用 `q`，保证各进程目标之和等于 `NUM_RECORDS`。多生产者测试也使用相同分配方法。若每个进程都发送或消费完整的 `NUM_RECORDS`，总数据量已经改变，该结果不能与单进程点直接比较。 + +## 12. 寻找饱和点和可用容量 + +### 12.1 先排除客户端上限 + +对生产者和消费者分别使用 1、2、4、8 个独立客户端进程上探；标准消费者点继续增加到 12、16、20，或在此之前已达到分区并行度/资源上限时停止。客户端应尽量分布在不同压测节点，每个进程使用相同属性，并聚合所有进程的吞吐量。每增加一次并发，至少核对： + +- 客户端 Pod CPU、内存、CPU 节流、网络和重启次数； +- Broker CPU、内存、网络、磁盘延迟/利用率； +- 请求错误、超时、重试和生产确认延迟； +- `UnderReplicatedPartitions`、`UnderMinIsrPartitionCount` 和消费组积压； +- 各 Broker 的分区 Leader、日志字节和流量是否失衡。 + +如果增加客户端资源或进程后吞吐量仍显著增长，前一个结果是客户端上限，不能报告为 Kafka 上限。如果单个压测节点网卡已饱和，应增加客户端节点，而不是继续增加同节点进程。 + +### 12.2 两阶段负载阶梯 + +1. **无节流上探**：使用 `--throughput -1`，逐步增加客户端进程，得到可达吞吐量范围。 +2. **受控负载验证**：以无节流结果为基准，依次施加约 25%、50%、75%、90%、100%、110% 的目标吞吐量。所有生产者的 `--throughput` 之和等于该点目标值。 +3. 每个点先预热，再保持至少 10 分钟稳态；结束后等待积压归零并确认副本健康。 +4. 90% 附近加密测试点；对最后一个稳定点和第一个不稳定点各重复至少 3 次。 +5. 打乱重复测试顺序或重新运行基线点，以识别硬件温度、缓存、邻居负载和时间漂移。 + +百分比只是寻找区间的起始策略，不是产品阈值。如果第一轮已违反健康条件，应降低负载重新建立区间。 + +### 12.3 预先定义判定规则 + +测试开始前由客户根据业务服务等级目标（Service Level Objective，SLO）确定以下阈值： + +- 生产确认 p95/p99 延迟上限； +- 错误、超时和重试率上限； +- 允许的最大消费积压及清空时间； +- Broker、客户端、节点和 PVC 的资源水位； +- 允许的 `UnderReplicatedPartitions` 和 `UnderMinIsrPartitionCount`，通常应始终为 0； +- 稳态窗口和重复次数。 + +如果客户尚无阈值，可把以下规则作为本次测试的工程判定方法，但必须在报告中标记为“测试规则”，不能称为 Kafka 官方阈值： + +- 增加一档负载或并发后，聚合吞吐量增幅连续两档小于 5%；且 +- p99 延迟、错误/超时、积压或资源水位持续恶化；或 +- 出现 `UnderReplicatedPartitions>0`、`UnderMinIsrPartitionCount>0`、离线日志目录、Pod 重启或磁盘压力。 + +满足上述组合条件的第一个点记为**饱和点**。饱和前最后一个满足全部 SLO、重复结果稳定且无健康异常的点记为**可用容量**。不要用瞬时最大值代替可用容量。 + +重复结果的离散程度也要报告。默认可用中位数作为中心值，并同时给出最小值、最大值和变异系数；如果同一测试点吞吐量变异系数超过 5%，应调查环境漂移并增加重复次数。5% 同样是本文的测试规则，不是 Kafka 产品保证。 + +## 13. 实例参数调优 + +先用模板基线完成一轮，再根据监控证据一次只调整一个参数。以下建议是诊断方向，不是固定最优值。 + +| 参数/资源 | 何时考虑调整 | 验证方法 | 风险和停止条件 | +| --- | --- | --- | --- | +| Broker CPU | 请求处理线程忙、CPU 持续接近约束、客户端未饱和 | 增加 CPU 后重复相同负载 | 若吞吐不增而磁盘/网络已满，继续加 CPU 无益 | +| Broker 内存/JVM 堆 | 容器内存压力、频繁垃圾回收（garbage collection，GC），或页缓存不足 | 同时观察堆、GC、常驻内存集（resident set size，RSS）、工作集和磁盘读取 | 堆过大挤压页缓存；堆与容器限制过近可能发生内存不足（out of memory，OOM） | +| `num.network.threads` | `NetworkProcessorAvgIdlePercent` 持续偏低且 CPU 尚有余量 | 小步增加并重复同一点 | 增加线程会增加调度和内存开销；不能修复网卡饱和 | +| `num.io.threads` | `RequestHandlerAvgIdlePercent` 持续偏低，且磁盘/CPU 尚有余量 | 小步增加并比较请求等待和吞吐 | 至少覆盖数据卷数量；过多线程可能增加上下文切换和 I/O 竞争 | +| `num.replica.fetchers` | 副本追赶慢、复制流量未用满资源 | 在故障恢复或高复制负载下验证 | 增加后会提高网络、磁盘和 CPU 消耗 | +| `num.recovery.threads.per.data.dir` | 启动或恢复过慢 | 在受控恢复测试中验证 | 恢复速度提高会加重前台 I/O 干扰 | +| Topic 分区数 | 单分区吞吐成为限制，且有足够消费者并发 | 新建不同分区数的 Topic 对比 | 增加分区改变 Key 顺序范围、元数据和副本开销；不能随意减少 | +| Broker 数量 | 单机资源已接近约束，需要横向扩展 | 扩容并等待副本重分配完成后重测 | 只增加 Broker 不会自动均衡已有分区数据；重分配期间结果无效 | +| `message.max.bytes` | 业务最大记录批次超过默认限制 | 同步核对 Topic、生产者和消费者抓取上限 | 增大后提高内存和网络突发，配置不一致会产生发送或消费失败 | +| 保留与日志段参数 | 磁盘占用、删除粒度或恢复时间不满足需求 | 长时间测试保留和删除行为 | 不应仅为短时跑分改变生产保留策略 | + +Apache Kafka 的线程空闲指标越接近 0 表示越忙。`NetworkProcessorAvgIdlePercent` 或 `RequestHandlerAvgIdlePercent` 长时间低于约 0.3 时，应结合 CPU、网络和存储判断是否出现处理能力不足；该值是诊断信号，不是单独的扩容触发器。 + +每次变更后： + +1. 保存变更前后完整配置差异； +2. 按当前产品版本的应用策略完成滚动重启或动态更新； +3. 等待 Kafka 和 NodePool Ready、ISR 完整、KRaft 控制器仲裁状态稳定； +4. 以相同 Topic 数据状态、客户端并发和负载重新执行至少 3 次； +5. 收益不稳定或健康指标恶化时回到前一个已验证配置。 + +## 14. 监控和证据采集 + +### 14.1 Kafka 健康检查 + +在测前、稳态窗口和测后执行。Broker Pod 名称通过标签发现： + +```bash +export BROKER_POD="$(kubectl -n "$NS" get pod \ + -l "strimzi.io/cluster=${CLUSTER},strimzi.io/broker-role=true" \ + -o jsonpath='{.items[0].metadata.name}')" + +kubectl -n "$NS" exec akafka-producer-perf -- \ + /opt/kafka/bin/kafka-metadata-quorum.sh \ + --bootstrap-server "$BOOTSTRAP" \ + --command-config /opt/perf-config/admin.properties \ + describe --status \ + | tee "$OUT/metadata-quorum-after.txt" + +kubectl -n "$NS" exec akafka-producer-perf -- \ + /opt/kafka/bin/kafka-topics.sh \ + --bootstrap-server "$BOOTSTRAP" \ + --command-config /opt/perf-config/admin.properties \ + --describe --under-replicated-partitions \ + | tee "$OUT/under-replicated-after.txt" + +kubectl -n "$NS" exec akafka-producer-perf -- \ + /opt/kafka/bin/kafka-consumer-groups.sh \ + --bootstrap-server "$BOOTSTRAP" \ + --command-config /opt/perf-config/admin.properties \ + --describe --group "$GROUP" \ + | tee "$OUT/consumer-group-after.txt" + +kubectl -n "$NS" exec akafka-producer-perf -- \ + /opt/kafka/bin/kafka-log-dirs.sh \ + --bootstrap-server "$BOOTSTRAP" \ + --command-config /opt/perf-config/admin.properties \ + --describe --topic-list "$TOPIC" \ + >"$OUT/log-dirs-after.txt" +``` + +`--under-replicated-partitions` 在健康状态下可以没有输出。应同时保存命令退出码或由编排器记录成功状态，避免把连接失败误判为“没有异常分区”。 + +`kafka-log-dirs.sh` 会在 JSON 数据前输出状态文本，因此上例按原始文本保存；需要程序化解析时，应先验证并移除非 JSON 前缀，不要直接把整个文件交给 JSON 解析器。 + +### 14.2 必采 Kafka 指标 + +以下 Java 管理扩展（Java Management Extensions，JMX）管理型 Bean（MBean）或其 Prometheus 映射必须覆盖完整稳态窗口： + +| 类别 | 指标或 JMX MBean | 用途 | +| --- | --- | --- | +| 写入/读取 | `BytesInPerSec`、`BytesOutPerSec`、`MessagesInPerSec` | 与客户端吞吐交叉验证 | +| 副本健康 | `UnderReplicatedPartitions`、`UnderMinIsrPartitionCount` | 前者表示 ISR 数量小于配置副本数，后者表示 ISR 数量小于 `min.insync.replicas` | +| 日志目录 | `OfflineLogDirectoryCount` | 识别不可用存储目录 | +| 网络线程 | `NetworkProcessorAvgIdlePercent` | 判断网络处理线程繁忙程度 | +| 请求线程 | `RequestHandlerAvgIdlePercent` | 判断请求处理线程繁忙程度 | +| 请求延迟 | Produce/Fetch 请求的 `TotalTimeMs`、`RequestQueueTimeMs`、`LocalTimeMs`、`RemoteTimeMs`、`ResponseQueueTimeMs` 和 `ResponseSendTimeMs` | 定位等待发生在哪个阶段 | +| 消费组 | 当前位点、LEO、每分区和总积压（lag） | 判断消费者能否跟上生产流量 | + +JMX Exporter 的 Prometheus 指标名称由实例的指标映射规则决定。不要假设所有环境使用同一名称。先从 Broker 暴露的 `/metrics` 端点保存原始指标并确认名称： + +```bash +kubectl -n "$NS" exec "$BROKER_POD" -- \ + curl -fsS http://127.0.0.1:9404/metrics \ + >"$OUT/broker-jmx-metrics.txt" + +grep -m 100 -E 'bytesin|bytesout|messagesin|underreplicated|underminisr|avgidle' \ + "$OUT/broker-jmx-metrics.txt" +``` + +如果实例启用了 Kafka Exporter，先发现 Pod，再保存其原始指标： + +```bash +export EXPORTER_PODS="$(kubectl -n "$NS" get pod \ + -l "strimzi.io/cluster=${CLUSTER},strimzi.io/name=${CLUSTER}-kafka-exporter" \ + -o jsonpath='{.items[*].metadata.name}')" + +if test -n "$EXPORTER_PODS"; then + export EXPORTER_POD="${EXPORTER_PODS%% *}" + kubectl -n "$NS" exec "$EXPORTER_POD" -- \ + curl -fsS http://127.0.0.1:9404/metrics \ + >"$OUT/kafka-exporter-metrics.txt" +else + printf '%s\n' 'Kafka Exporter Pod not found' \ + >"$OUT/kafka-exporter-not-found.txt" +fi +``` + +原始端点有数据只证明 Exporter 正常暴露指标，不证明平台已经抓取时序数据。正式测试前，应在平台监控或 Prometheus 中查询最近 15 分钟的 Broker 指标，确认每个 Broker 都有连续样本。如果查询为空或存在采集空洞，应停止测试，按当前平台监控配置补齐 ServiceMonitor/PodMonitor 和指标映射；其选择标签由当前 Prometheus 配置决定，不能照抄其他集群的标签。 + +### 14.3 Kubernetes 和基础设施指标 + +至少采集以下时序数据，而不是只保存测后瞬时值： + +- Broker、Controller 和客户端 Pod：CPU 使用、CPU 节流、内存工作集、网络收发、重启和 OOM； +- 节点：CPU、内存、网络带宽/丢包、磁盘吞吐、每秒输入/输出操作数（input/output operations per second，IOPS）、平均和高分位延迟、利用率、队列深度； +- PVC：已用字节、容量、使用率和增长速率； +- Kubernetes 事件、Pod 调度节点和资源 requests/limits。 + +`kubectl top` 只提供近实时快照，可用于快速检查，但不能替代 Prometheus 范围查询： + +```bash +kubectl -n "$NS" top pod --containers \ + | tee "$OUT/top-pods-after.txt" +kubectl -n "$NS" get events --sort-by=.lastTimestamp \ + >"$OUT/events.txt" +kubectl -n "$NS" get pod -o wide \ + >"$OUT/pods-after.txt" +kubectl -n "$NS" get pvc -o wide \ + >"$OUT/pvc-after.txt" +``` + +以下 PromQL 仅表示计算意图。先在当前 Prometheus 中检查实际指标名和标签，再替换 `$NS`、`$TOPIC`、`$GROUP` 和集群标签；保存查询表达式、时间范围、步长和原始响应。 + +```promql +sum(rate(kafka_server_brokertopicmetrics_messagesin_total{namespace="$NS",topic="$TOPIC"}[5m])) + +sum(rate(kafka_server_brokertopicmetrics_bytesin_total{namespace="$NS",topic="$TOPIC"}[5m])) + +sum(rate(kafka_server_brokertopicmetrics_bytesout_total{namespace="$NS",topic="$TOPIC"}[5m])) + +sum(kafka_server_replicamanager_underreplicatedpartitions{namespace="$NS"}) + +sum(kafka_server_replicamanager_underminisrpartitioncount{namespace="$NS"}) + +min(kafka_network_socketserver_networkprocessoravgidle_percent{namespace="$NS"}) + +min(kafka_server_kafkarequesthandlerpool_requesthandleravgidle_percent{namespace="$NS"}) + +sum(kafka_consumergroup_lag{namespace="$NS",consumergroup="$GROUP",topic="$TOPIC"}) + +sum by (pod) ( + rate(container_cpu_usage_seconds_total{namespace="$NS",container!="",image!=""}[5m]) +) + +max by (pod) ( + container_memory_working_set_bytes{namespace="$NS",container!="",image!=""} +) + +sum by (pod) ( + rate(container_network_receive_bytes_total{namespace="$NS"}[5m]) +) + +sum by (pod) ( + rate(container_network_transmit_bytes_total{namespace="$NS"}[5m]) +) + +kubelet_volume_stats_used_bytes{namespace="$NS"} +/ +kubelet_volume_stats_capacity_bytes{namespace="$NS"} +``` + +CPU 节流和节点磁盘指标在不同容器运行时、cgroup 版本和监控栈中的名称可能不同。如果平台没有对应时序指标，应在测试前补齐采集或从节点运行时、cgroup 和存储系统导出等价证据；不能把“查询为空”记为 0。 + +### 14.4 Redis 或其他关联组件 + +如果业务链路还包含 Redis，应为 Redis 使用相同的 `RUN_ID`、UTC 时间窗口和节点事件记录，并附独立的 Redis 性能报告。Kafka 和 Redis 的吞吐量、延迟及饱和点属于不同指标，不能合并，也不能仅凭时间相关性推断一个组件导致另一个组件变慢。Kafka 报告只保留关联报告链接、版本、测试窗口和经验证的依赖关系。 + +## 15. 结果汇总和有效性检查 + +### 15.1 每个测试点的记录字段 + +至少记录： + +```text +run_id,start_utc,end_utc,measurement_seconds,test_mode,reliability_profile, +kafka_version,operator_version,kraft_roles,broker_count,controller_count, +broker_cpu,broker_memory,jvm_xms,jvm_xmx,storage_class,pvc_size, +topic,partitions,replication_factor,min_insync_replicas, +record_count_target,producer_records_sent,consumer_records_consumed, +record_size,compression,compression_ratio,key_distribution, +producer_processes,consumer_processes,acks,idempotence,batch_size,linger_ms, +throughput_target,fetch_min_bytes,fetch_max_wait_ms,max_partition_fetch_bytes, +producer_records_s,producer_mib_s,producer_avg_ms,producer_p50_ms, +producer_p95_ms,producer_p99_ms,producer_p999_ms, +consumer_records_s,consumer_mib_s,max_lag,final_lag,error_count,timeout_count, +broker_cpu_cores_max,broker_memory_bytes_max,broker_network_bps_max,pvc_used_bytes_max, +network_idle_min,request_idle_min,under_replicated_max,under_min_isr_max, +client_cpu_cores_max,client_cpu_throttling_ratio_max,result_valid,invalid_reason +``` + +Kafka 脚本输出使用 `MB/sec` 列名。报告应保留原始列名；需要统一为 MiB/s 时，另行使用 `实际载荷字节 ÷ 秒数 ÷ 2^20` 计算并记录公式，不要只修改列名。 + +资源峰值字段默认记录“单个 Pod 或 PVC 的最大值”；如果使用求和、平均或其他聚合方式，应在字段说明中记录聚合函数、单位、时间窗口和采样步长。 + +### 15.2 有效结果的必要条件 + +仅当以下条件全部满足时，结果才标记为 `valid`： + +- 命令退出码为 0，生产和消费记录数达到目标； +- 预热和稳态窗口按计划完成，时钟范围明确； +- Kafka、Operator、配置、Topic、客户端和基础设施快照完整； +- 未发生 Pod 重启、OOM、节点压力、卷异常或计划外维护； +- `UnderReplicatedPartitions=0`、`UnderMinIsrPartitionCount=0`、`OfflineLogDirectoryCount=0`； +- 生产可靠性基线没有违反 `acks=all`、幂等和最小 ISR 要求； +- 客户端不是已确认的瓶颈，或结果已明确标记为“客户端上限”； +- 监控覆盖完整稳态窗口，并且没有采集空洞； +- 重复测试的差异已经解释，或结论保留不确定性。 + +以下任一情况应标记为 `invalid` 并重测： + +- Topic 已有未知数据、消费组复用或实际记录数不足； +- 测试期间发生滚动重启、重分配、Leader 频繁切换或副本追赶； +- PVC 接近满、存储或节点出现压力； +- 客户端被 CPU 限流、网络饱和或异常退出； +- 不同测试点改变了两个以上关键变量； +- 仅有平均值，没有原始输出、分位延迟或时序监控。 + +### 15.3 报告结论写法 + +结论必须区分： + +- **事实**：原始输出、配置快照和监控直接显示的结果； +- **推断**：由多个指标共同支持的瓶颈判断，并列出反证； +- **建议**：下一步调参、扩容或重测方案； +- **待验证事项**：缺少存储、网络、业务消息模型或长稳测试时尚不能确认的内容。 + +推荐同时报告： + +1. 最后一个满足全部 SLO 的可用容量； +2. 第一个不稳定的饱和点及触发条件； +3. 3 次以上重复结果的中位数、范围和变异系数； +4. 与基线相比的相对变化，并保持可靠性和环境配置相同； +5. 适用边界，特别是消息模型、存储、网络、认证、保留策略和测试持续时间。 + +## 16. 清理 + +先归档并校验 `$OUT`，再删除本轮独立资源。不要使用模糊标签或通配符删除共享资源。 + +```bash +# 仅删除本文创建的专用 Topic；命令执行前再次核对名称。 +kubectl -n "$NS" exec akafka-producer-perf -- \ + /opt/kafka/bin/kafka-topics.sh \ + --bootstrap-server "$BOOTSTRAP" \ + --command-config /opt/perf-config/admin.properties \ + --delete --topic "$TOPIC" + +kubectl -n "$NS" delete pod \ + akafka-producer-perf akafka-consumer-perf --ignore-not-found +kubectl -n "$NS" delete configmap \ + akafka-perf-client-config --ignore-not-found +kubectl -n "$NS" delete secret \ + akafka-perf-client-config --ignore-not-found +``` + +如果整个命名空间都是本轮创建且已确认无共享资源，可以通过变更流程删除命名空间。删除实例或命名空间前，必须核对 PVC 的 `deleteClaim` 和 StorageClass 回收策略，确认数据保留或销毁符合要求。 + +## 17. 参考资料 + +- [Apache Kafka 4.2 Broker 配置](https://kafka.apache.org/42/configuration/broker-configs/) +- [Apache Kafka 4.2 Producer 配置](https://kafka.apache.org/42/configuration/producer-configs/) +- [Apache Kafka 4.2 Consumer 配置](https://kafka.apache.org/42/configuration/consumer-configs/) +- [Apache Kafka 4.2 监控](https://kafka.apache.org/42/operations/monitoring/) +- [Apache Kafka 4.2 硬件与操作系统](https://kafka.apache.org/42/operations/hardware-and-os/) +- [Strimzi 0.48 部署和运行指南](https://strimzi.io/docs/operators/0.48.0/deploying.html) +- [Prometheus Operator API 参考](https://prometheus-operator.dev/docs/api-reference/api/) From 4a0ec2b3930f8497f3fbcc13e1d13d78771613cb Mon Sep 17 00:00:00 2001 From: Richard Feng Date: Wed, 29 Jul 2026 22:04:26 -0400 Subject: [PATCH 5/8] docs: use supported code fence for PromQL --- .../ecosystem/kafka/aKafka_Performance_Testing_Guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/zh/solutions/ecosystem/kafka/aKafka_Performance_Testing_Guide.md b/docs/zh/solutions/ecosystem/kafka/aKafka_Performance_Testing_Guide.md index f2735212c..cf64f24a2 100644 --- a/docs/zh/solutions/ecosystem/kafka/aKafka_Performance_Testing_Guide.md +++ b/docs/zh/solutions/ecosystem/kafka/aKafka_Performance_Testing_Guide.md @@ -891,7 +891,7 @@ kubectl -n "$NS" get pvc -o wide \ 以下 PromQL 仅表示计算意图。先在当前 Prometheus 中检查实际指标名和标签，再替换 `$NS`、`$TOPIC`、`$GROUP` 和集群标签；保存查询表达式、时间范围、步长和原始响应。 -```promql +```text sum(rate(kafka_server_brokertopicmetrics_messagesin_total{namespace="$NS",topic="$TOPIC"}[5m])) sum(rate(kafka_server_brokertopicmetrics_bytesin_total{namespace="$NS",topic="$TOPIC"}[5m])) From d687ffd34f2d0d5f35c6f9ee70c41b095b510666 Mon Sep 17 00:00:00 2001 From: Richard Feng Date: Wed, 29 Jul 2026 22:38:16 -0400 Subject: [PATCH 6/8] docs: publish Kafka performance guide in English --- .../kafka/Kafka_Performance_Testing_Guide.md | 1271 +++++++++++++++++ .../kafka/aKafka_Performance_Testing_Guide.md | 1035 -------------- 2 files changed, 1271 insertions(+), 1035 deletions(-) create mode 100644 docs/en/solutions/ecosystem/kafka/Kafka_Performance_Testing_Guide.md delete mode 100644 docs/zh/solutions/ecosystem/kafka/aKafka_Performance_Testing_Guide.md diff --git a/docs/en/solutions/ecosystem/kafka/Kafka_Performance_Testing_Guide.md b/docs/en/solutions/ecosystem/kafka/Kafka_Performance_Testing_Guide.md new file mode 100644 index 000000000..c2f0bd27c --- /dev/null +++ b/docs/en/solutions/ecosystem/kafka/Kafka_Performance_Testing_Guide.md @@ -0,0 +1,1271 @@ +--- +products: + - Alauda Application Services +kind: + - Solution +--- + +# Kafka Performance Testing Guide + +## 1. Purpose and Scope + +This guide describes how to run repeatable and auditable performance tests for a Kafka instance on Alauda Container Platform (ACP). It is intended to answer the following questions: + +- What sustained producer and consumer throughput can the instance deliver with a specified replication, acknowledgment, and compression policy? +- What usable capacity can the instance deliver while latency, error rate, replica synchronization, and resource utilization remain within their constraints? +- Is the bottleneck in the clients, broker CPU, memory, or network, persistent volumes, or partition and replica placement? +- After an instance or client parameter is changed, are the results comparable and is the improvement reproducible? + +This guide uses Apache Kafka terminology for brokers, topics, and controllers. + +In-sync replica (ISR) terminology follows the Apache Kafka documentation. + +This guide covers both Kafka Raft metadata mode (KRaft) and legacy ZooKeeper-based instances. Treat the metadata mode as part of the test profile. Do not compare or combine results across metadata modes. + +The primary commands use the Apache Kafka 4.2.0 script interfaces. Section 8.5 lists the required command changes for the legacy Kafka 2.x clients shipped with ZooKeeper-based operator releases. For any installed version, first save the `--version` and `--help` output and use only the options shown by that version. + +This guide does not provide a universally applicable "best throughput" or "best configuration." A change in hardware, storage, network, message size, compression ratio, reliability policy, or workload can make results incomparable. + +> **Risk notice:** Performance tests write large amounts of data and can exhaust disk, network, or node resources. Run them only against an authorized, isolated instance, namespace, and test window. Do not search for the saturation point on an instance that serves production traffic. + +## 2. Result Profiles + +Select one workload mode and one reliability profile before testing. Do not combine results from different profiles. + +### 2.1 Workload modes + +| Mode | Purpose | Execution | Primary results | +| --- | --- | --- | --- | +| Producer-only | Measure write capacity | Run producers only | records/s, MiB/s, produce acknowledgment latency | +| Consumer-only | Measure read capacity | Preload enough data, then run consumers | records/s, MiB/s, consumer group lag | +| Concurrent produce and consume | Simulate continuous data flow | Start consumers, then start producers | Throughput on both sides, lag growth, broker resources, and stability | + +The producer script reports produce request acknowledgment latency, not application end-to-end latency. The consumer script reports throughput but does not report the time from message production to consumption. To measure end-to-end latency, write a send timestamp into each message and calculate latency in an application-specific load generator at the consumer. Do not subtract the output of the two scripts. + +### 2.2 Reliability profiles + +| Profile | Topic | Producer | Purpose | +| --- | --- | --- | --- | +| Production reliability baseline | `replication.factor=3`, `min.insync.replicas=2` | `acks=all`, `enable.idempotence=true` | Default for capacity planning and production acceptance | +| Leader acknowledgment ceiling | `replication.factor=3`; all other settings match the tested baseline | `acks=1`, `enable.idempotence=false` | Estimate the throughput ceiling with weaker acknowledgment | + +`acks=1` waits only for the leader to acknowledge the request. If the leader fails immediately afterward, data that has not been replicated can be lost. Kafka idempotent production requires `acks=all`, so an `acks=1` test must explicitly set `enable.idempotence=false` to avoid a conflicting client configuration. Every capacity result must identify its reliability profile. Do not report the leader acknowledgment ceiling as production reliability capacity. + +### 2.3 Access paths + +External connections can use Transport Layer Security (TLS) and Simple Authentication and Security Layer (SASL). Test with the same security configuration planned for the application. + +| Access path | Client location | Constraints included in the result | Use case | +| --- | --- | --- | --- | +| In-cluster capacity | Dedicated load-test nodes in the business cluster, connected through the internal bootstrap Service | Kafka, cluster network, and storage | Instance capacity and tuning; this is the default path in this guide | +| External end-to-end | Customer-selected source cluster or host, connected through the actual external listener | All in-cluster constraints plus the load balancer, wide area network (WAN), firewall, TLS/SASL, and client egress | Application connectivity acceptance | + +Do not combine results from the two paths. For an external end-to-end test, save the client location, listener, round-trip latency, packet loss, and network path. Confirm that the client can reach every broker address advertised in Kafka metadata, not only the bootstrap address. After initializing the variables in section 4.3, read listener addresses from current resource status instead of inferring them from naming rules: + +```bash +kubectl -n "$NS" get kafka "$CLUSTER" \ + -o jsonpath='{range .status.listeners[*]}{.name}{"\t"}{.bootstrapServers}{"\n"}{end}' +``` + +### 2.4 Metadata modes + +| Area | KRaft instance | ZooKeeper-based instance | +| --- | --- | --- | +| Metadata quorum | KRaft controller quorum | ZooKeeper ensemble | +| Controller role | Dedicated or combined controller/broker roles, depending on the deployment | One broker is elected as the active Kafka controller; ZooKeeper nodes are not Kafka controllers | +| Generated workloads | Kafka Pods and, where supported, `KafkaNodePool` resources | Kafka and ZooKeeper StatefulSets | +| Primary health evidence | `kafka-metadata-quorum.sh`, controller metrics, and broker health | ZooKeeper ensemble metrics, exactly one active Kafka controller, and broker health | +| Storage outside broker data volumes | Controller metadata storage, depending on role layout | A separate persistent volume claim (PVC) for each ZooKeeper node | + +Determine the mode from the final generated resources and installed operator schema, not from the Kafka version alone. Record the mode, operator version, Kafka version, and generated resource shape for every result. + +## 3. Standard Test Matrix + +Run the standard matrix first to produce comparable data, then add a workload that matches the actual message model, connection security, and concurrency pattern. Run the complete matrix separately for KRaft and ZooKeeper-based instances; the metadata mode must not change within a comparison series. + +### 3.1 Instance and message matrix + +| Broker resources | Brokers | Topic partitions | Replication factor | Message sizes | +| --- | ---: | ---: | ---: | --- | +| 1 vCPU / 2 GiB | 3 | 30 | 3 | 100 B, 500 B, 1000 B | +| 2 vCPU / 4 GiB | 3 | 30 | 3 | 100 B, 500 B, 1000 B | +| 4 vCPU / 8 GiB | 3 | 30 | 3 | 100 B, 500 B, 1000 B | +| 8 vCPU / 16 GiB | 3 | 30 | 3 | 100 B, 500 B, 1000 B | + +Use these record counts for the standard points: + +- For 1, 2, and 4 vCPU, send 50,000,000 records for each message size. +- For 8 vCPU, send 100,000,000 records for 100 B and 500 B, and 50,000,000 records for 1000 B. + +Start each standard producer point with one producer process. If section 12 shows that the client reaches its limit first, add processes and report the aggregated result separately. The standard consumer point targets 20 consumers. Run 20 independent processes in the same consumer group in every metadata mode. Do not rely on `--threads`: Kafka 4.2 does not provide it, and some legacy Kafka 2.x scripts accept it but ignore it. The processes can be distributed across multiple adequately resourced Pods, but record the number of processes and resources in each Pod. + +If the test window or disk capacity cannot support these record counts, reduce them, but keep each steady-state point running for at least 10 minutes and record the actual count and duration. A short burst result is not a substitute for steady-state capacity. + +### 3.2 Application-specific matrix + +In addition to the standard matrix, test at least one configuration that matches the production plan: + +- message-size distribution, not only an average size; +- record-key distribution and partitioning strategy; +- `compression.type`; +- connection security such as TLS and SASL; +- number of producer and consumer instances; +- topic count, partition count, replication factor, and retention time; +- actual burst and idle cycles. + +If no production workload distribution is available, label the conclusion as a synthetic workload result. Do not present it as application capacity. + +## 4. Prerequisites + +### 4.1 Environment isolation + +1. Create a dedicated namespace, Kafka instance, topic, consumer group, and load-test clients. +2. Distribute brokers across Kubernetes nodes. For a ZooKeeper-based instance, distribute ZooKeeper replicas across nodes and failure domains as well. Preserve the production plan for whether one broker and one ZooKeeper Pod may share a node. +3. Do not place load-test clients on any broker, controller, or ZooKeeper node, because client resource contention will distort results. +4. Do not run scaling, upgrades, node maintenance, storage migration, ZooKeeper reconfiguration, or other high-load tasks during the test. +5. Use the same node and persistent-volume types as the target environment. Prefer local block storage or qualified block storage for production capacity testing. Do not extrapolate results from unqualified shared file storage to a block-storage environment. +6. Use a new instance for each instance-parameter test, or delete the test topic and wait for the instance to stabilize. After changing a parameter that requires a restart, wait until all Pods are Ready, all replicas are synchronized, and the KRaft quorum or ZooKeeper ensemble is stable before starting the next run. + +### 4.2 Required tools and permissions + +- `kubectl` access to the business cluster; +- the required permissions for Kafka and, for a legacy deployment, ZooKeeper resources, as well as Pods, StatefulSets, ConfigMaps, Secrets, Services, and persistent volume claims (PVCs) in the test namespace; +- read access to platform monitoring or Prometheus time-series data; +- permission to read the Kafka image name from a deployed broker Pod. This guide reuses the Kafka image managed by the operator and does not require a private image name; +- a client-configuration Secret for SASL/TLS. Do not include passwords, tokens, private keys, or complete Secrets in the report. + +### 4.3 Create an evidence directory + +Run the following initialization in one Bash session. Later code blocks use these variables and error-handling options: + +```bash +set -euo pipefail + +export KUBECONFIG=/path/to/business-cluster.kubeconfig +export NS=kafka-perf +export CLUSTER=perf-kafka +export TOPIC=perf-rf3-p30 +export METADATA_MODE=kraft +export GROUP="perf-$(date -u +%Y%m%dT%H%M%SZ)" +export RUN_ID="$(date -u +%Y%m%dT%H%M%SZ)-${CLUSTER}" +export OUT="$(pwd)/kafka-perf-${RUN_ID}" +mkdir -p "$OUT" + +kubectl version -o yaml >"$OUT/kubectl-version.yaml" +kubectl cluster-info >"$OUT/cluster-info.txt" +kubectl get nodes -o wide >"$OUT/nodes.txt" +kubectl get storageclass -o yaml >"$OUT/storageclasses.yaml" +``` + +Set `METADATA_MODE=zookeeper` for a ZooKeeper-based instance. Keep this variable fixed for the complete comparison series. + +Export only non-sensitive metadata. Do not write kubeconfig files, Secret data, registry credentials, or client passwords to the evidence directory. + +## 5. Export and Freeze Instance Parameters + +The general Kafka parameter template is usually named `general-kafka`. The template is product configuration in the management cluster. The `RdsKafka` resource in the business cluster stores the final effective instance configuration. Templates can change between product versions, so export the installed version for every test instead of relying on the table in this guide. + +If you have read access to the management cluster, run: + +```bash +export MGMT_KUBECONFIG=/path/to/management-cluster.kubeconfig + +kubectl --kubeconfig "$MGMT_KUBECONFIG" get paramtemplate -A \ + -l component=kafka + +export TEMPLATE_NS="$(kubectl --kubeconfig "$MGMT_KUBECONFIG" \ + get paramtemplate -A -l component=kafka \ + -o jsonpath='{range .items[?(@.metadata.name=="general-kafka")]}{.metadata.namespace}{"\n"}{end}')" +test -n "$TEMPLATE_NS" +kubectl --kubeconfig "$MGMT_KUBECONFIG" -n "$TEMPLATE_NS" \ + get paramtemplate general-kafka -o yaml \ + >"$OUT/general-kafka-paramtemplate.yaml" +``` + +If management-cluster access is unavailable, select the general Kafka parameter template in the product console and export the final `spec.config` from the instance. Do not infer values from the template name. + +Review at least the following parameters in the current standard general template. The example baseline only identifies parameters; it does not replace the installed template. + +| Broker parameter | Example baseline | Effect and test interpretation | +| --- | ---: | --- | +| `auto.create.topics.enable` | `false` | Prevents an accidental topic from being created because of a typo; create test topics explicitly | +| `default.replication.factor` | `3` | Default when replication factor is omitted; the test still specifies `3` explicitly | +| `min.insync.replicas` | `1` | Template baseline; set it explicitly to `2` at the topic or instance level for the production reliability test | +| `num.network.threads` | `3` | Number of threads that process network requests | +| `num.io.threads` | `8` | Number of threads that process requests and disk-related work | +| `num.replica.fetchers` | `1` | Number of replica fetcher threads per source broker | +| `num.recovery.threads.per.data.dir` | `1` | Threads per data directory for startup recovery and shutdown flushes | +| `background.threads` | `10` | Background task thread-pool size | +| `message.max.bytes` | `1048588` | Maximum record-batch size accepted by a broker; not the size of one record | +| `compression.type` | `producer` | Retains the compression type selected by the producer | +| `log.segment.bytes` | `1073741824` | Log-segment size | +| `log.retention.hours` | `168` | Time-based retention period | +| `log.roll.hours` | `168` | Maximum time before a log segment rolls | +| `delete.topic.enable` | `true` | Allows test topics to be deleted | +| `unclean.leader.election.enable` | `false` | Prevents a replica outside the ISR from becoming leader | + +The current product version determines how parameter-template changes are applied. Before testing, query the parameter definition and record its `applyStrategy`. If it is `RestartApply`, exclude the rolling-restart interval from the measurement window. + +```bash +kubectl --kubeconfig "$MGMT_KUBECONFIG" get paramdefinition kafka-general -o yaml \ + >"$OUT/kafka-general-paramdefinition.yaml" +``` + +### 5.1 Additional snapshot for a ZooKeeper-based instance + +The legacy generated `Kafka` resource stores the broker and ZooKeeper specifications separately. Save both before testing: + +```bash +kubectl -n "$NS" get kafka "$CLUSTER" \ + -o jsonpath='{.spec.kafka.version}{"\n"}' \ + >"$OUT/kafka-version.txt" +kubectl -n "$NS" get kafka "$CLUSTER" \ + -o jsonpath='{.spec.kafka.config}' \ + >"$OUT/kafka-broker-config.json" +kubectl -n "$NS" get kafka "$CLUSTER" \ + -o jsonpath='{.spec.zookeeper}' \ + >"$OUT/zookeeper-spec.json" +``` + +Record `log.message.format.version` and `inter.broker.protocol.version` when present. A value that does not match the broker major and minor version can indicate an incomplete upgrade. Do not change either value during a performance comparison. + +ZooKeeper settings under `spec.zookeeper.config` are version-specific, and the operator prevents changes to settings that it owns, including server addresses, data directories, client ports, quorum authentication, and several TLS properties. Query the installed custom resource definition before changing a ZooKeeper setting. Keep the complete ZooKeeper configuration, resources, JVM options, and metrics mapping fixed unless one of them is the single variable under test. + +## 6. Persistent-Volume Capacity and Storage Qualification + +### 6.1 Capacity formula for a bounded synthetic test + +For one topic, estimate the message payload stored by each broker as follows: + +```text +payload bytes per broker ≈ record count × record bytes × replication factor ÷ broker count ÷ measured compression ratio +recommended minimum PVC = payload bytes per broker × safety factor + other topics, indexes, and operational reserve +``` + +The compression ratio is uncompressed bytes divided by bytes stored. Use `1` during estimation when no measured ratio is available; do not assume a compression benefit. This guide uses an engineering safety factor of `1.5` for a new, single-topic, bounded synthetic test. This factor is not a Kafka default and does not apply to production capacity planning. Increase it based on measured peaks when data can be skewed, replicas can be reassigned, multiple topics exist, or writes continue throughout the retention period. Reserve migration space separately. + +With 3 brokers, replication factor 3, and compression ratio 1, one copy of every record is stored on each broker on average. The payload and recommended minimums are: + +| Records | Message size | Payload per broker | Minimum rounded up after multiplying by 1.5 | +| ---: | ---: | ---: | ---: | +| 50,000,000 | 100 B | 4.66 GiB | 7 GiB | +| 50,000,000 | 500 B | 23.28 GiB | 35 GiB | +| 50,000,000 | 1000 B | 46.57 GiB | 70 GiB | +| 100,000,000 | 100 B | 9.31 GiB | 14 GiB | +| 100,000,000 | 500 B | 46.57 GiB | 70 GiB | + +The table does not separately account for keys, record headers, batch overhead, indexes, control records, other topics, or temporary reassignment space. A 50 GiB PVC is therefore insufficient to safely run the complete standard point for either 50,000,000 1000 B records or 100,000,000 500 B records. Do not use this bounded formula for a test with unlimited duration or unknown write volume. Calculate those cases from the write rate, retention time, and target disk utilization. + +A dedicated KRaft controller does not store topic message payload and must not use the broker payload formula. Its PVC must satisfy the minimum for the installed product version and be calibrated from changes in topic and partition count, metadata-log growth, and retention. When controllers and brokers share a data volume in a combined role, the PVC must accommodate the broker payload minimum plus metadata, indexes, and operational reserve. + +A ZooKeeper PVC also does not store topic payload and must not use the broker payload formula. Size it from observed ZooKeeper data and snapshot growth, recovery requirements, and an explicit free-space reserve. Use persistent, low-latency block storage and separate ZooKeeper volumes from Kafka broker data volumes. Do not use ephemeral ZooKeeper storage for a capacity result: a single-replica ensemble with ephemeral storage loses cluster metadata after a restart, and any ephemeral ensemble introduces a recovery condition that is not representative of production. + +For this legacy operator model, storage type and class are not general tuning variables. Persistent-volume expansion is increase-only, depends on the StorageClass, and causes the operator to restart Pods that use the resized volume. Complete any expansion before the measurement window and wait for the full ensemble and all broker sessions to stabilize. + +### 6.2 Storage checks + +```bash +kubectl -n "$NS" get pvc -o wide >"$OUT/pvc-before.txt" +kubectl -n "$NS" get pod -o wide >"$OUT/pods-before.txt" + +# Filter with labels from the current instance; do not assume Pod names. +kubectl -n "$NS" get pod \ + -l "strimzi.io/name=${CLUSTER}-kafka" \ + -o wide +``` + +Confirm that: + +- every PVC is `Bound` and is at least as large as the calculated requirement; +- persistent volumes and nodes are distributed as required by the test design; +- the StorageClass, volume type, file system, and node disks match the target environment; +- no node has sustained disk pressure and Kafka has no replicas awaiting recovery. + +Storage latency is a primary Kafka constraint. A StorageClass name alone does not prove storage equivalence. Also save the provisioner, media type, topology, and I/O latency and utilization during the test. + +## 7. Create or Update the Kafka Instance + +Create a dedicated instance in the product console, or update an existing dedicated test instance. Console fields can change between product versions. Treat the `RdsKafka` resource and generated Strimzi resources in the business cluster as the final state. + +### 7.1 Topology + +- Use at least 3 brokers. +- Distribute brokers across nodes and failure domains with Pod anti-affinity. +- Use the same metadata mode and role layout as production. +- Test the broker CPU and memory sizes in section 3 while keeping metadata-service resources fixed, so broker and metadata-service sizes do not change at the same time. +- Do not allow the Java Virtual Machine (JVM) heap limit to consume all container memory. Reserve memory for page cache and non-heap memory in the container. + +> Kafka relies on the operating system page cache. Increasing the JVM heap does not increase cache available to Kafka. A heap configured too close to the container limit can reduce page-cache capacity. Out of memory (OOM) can also terminate the container. + +For KRaft, match the production use of dedicated controllers or combined controller/broker roles and state the role layout in the report. Keep controller count, resources, and storage fixed while testing broker sizes. + +For ZooKeeper mode: + +- use an odd number of ZooKeeper replicas, with at least 3 for a production-representative test; +- spread ZooKeeper replicas across nodes and failure domains with Pod anti-affinity; +- use persistent, low-latency block storage and keep ZooKeeper resources, JVM options, storage, and configuration fixed; +- record which broker is the active Kafka controller, but do not describe a ZooKeeper Pod as a Kafka controller; +- preserve the production colocation policy between brokers and ZooKeeper Pods. + +After submitting the console form, verify the following KRaft fields against the current `RdsKafka` custom resource definition (CRD). Do not infer the effective configuration from displayed console values: + +| `RdsKafka` field | Standard test requirement | +| --- | --- | +| `spec.mode` | `KRaft` | +| `spec.replicas` | 3 brokers | +| `spec.resources.requests/limits` | Current matrix size; record both requests and limits | +| `spec.storage.class` | Customer-selected and qualified StorageClass | +| `spec.storage.size` | At least the value calculated in section 6 | +| `spec.storage.deleteClaim` | Select according to data-retention and destruction requirements; do not set it to `true` only to simplify cleanup | +| `spec.controller` | Roles, count, resources, and storage match the production plan; use at least 3 voting nodes for dedicated controllers | +| `spec.config` | Exported `general-kafka` baseline plus the one parameter changed in this run | +| `spec.kafka.listeners` | Matches the in-cluster or external access path for this run | +| `spec.kafkaExporter` | Enabled when consumer group lag is required, with topic and group regular expressions limited to the test scope | + +```bash +kubectl explain rdskafka.spec --recursive \ + >"$OUT/rdskafka-spec-schema.txt" +``` + +For a ZooKeeper-based instance, also verify these fields in the generated `Kafka` resource. Field availability is determined by the installed CRD: + +| Generated `Kafka` field | Standard test requirement | +| --- | --- | +| `spec.kafka.version` | Installed and supported Kafka version; keep fixed | +| `spec.kafka.replicas` | 3 brokers | +| `spec.kafka.resources` | Current matrix size; record requests and limits | +| `spec.kafka.storage` | Qualified persistent block storage sized by section 6 | +| `spec.kafka.config` | Frozen broker baseline; record `inter.broker.protocol.version` and `log.message.format.version` when present | +| `spec.zookeeper.replicas` | Odd count, with at least 3 for a production-representative test | +| `spec.zookeeper.resources` | Fixed requests and limits | +| `spec.zookeeper.storage` | Persistent, low-latency block storage with a fixed type, class, and size | +| `spec.zookeeper.jvmOptions` | Fixed heap and JVM settings | +| `spec.zookeeper.config` | Frozen ZooKeeper configuration | +| `spec.zookeeper.metricsConfig` | Java Management Extensions (JMX) Prometheus Exporter mapping enabled for the test | +| `spec.kafkaExporter` | Enabled when consumer group lag is required, with topic and group regular expressions limited to the test scope | + +Do not apply KRaft-only fields such as `spec.mode`, `spec.controller`, or `KafkaNodePool` to a legacy ZooKeeper-based resource. + +### 7.2 Fixed variables + +Change only one evaluated variable in each test cycle. Keep the following fixed and record them in the report: + +- Kafka and operator versions and metadata mode; +- KRaft role placement and broker/controller counts, or ZooKeeper replica count and active broker-controller state; +- broker, controller, and ZooKeeper CPU, memory, JVM heap, node selection, and anti-affinity, as applicable; +- StorageClass, PVC size, and storage media; +- complete broker-configuration snapshot; +- topic partitions, replication factor, and `min.insync.replicas`; +- client image version, resources, node location, and connection security; +- message size, record count, compression, key distribution, acknowledgments, and consumer parameters. + +Save the final instance state: + +```bash +kubectl -n "$NS" get rdskafka "$CLUSTER" -o yaml \ + >"$OUT/rdskafka.yaml" +kubectl -n "$NS" get kafka "$CLUSTER" -o yaml \ + >"$OUT/kafka.yaml" +kubectl -n "$NS" get pod,pvc,service -o wide \ + >"$OUT/workloads-before.txt" +``` + +For KRaft, also save the node pools: + +```bash +kubectl -n "$NS" get kafkanodepool \ + -l "strimzi.io/cluster=${CLUSTER}" -o yaml \ + >"$OUT/kafkanodepools.yaml" +``` + +For ZooKeeper mode, save both generated StatefulSets and the ZooKeeper Pods: + +```bash +kubectl -n "$NS" get statefulset \ + "${CLUSTER}-kafka" "${CLUSTER}-zookeeper" -o yaml \ + >"$OUT/kafka-zookeeper-statefulsets.yaml" +kubectl -n "$NS" get pod \ + -l "strimzi.io/name=${CLUSTER}-zookeeper" -o wide \ + >"$OUT/zookeeper-pods-before.txt" +``` + +If any collection command fails, stop and investigate permissions, resource names, or instance status. Do not conceal the error with `|| true`. + +### 7.3 Monitoring options + +Enable the Kafka metrics collection provided by the platform when creating the instance. If consumer group lag is required, also enable Kafka Exporter and limit its topic and consumer-group regular expressions to this test. Before starting the load test, check the final resources and Exporter Pod: + +```bash +kubectl -n "$NS" get rdskafka "$CLUSTER" \ + -o jsonpath='{.spec.kafkaExporter}' \ + >"$OUT/rdskafka-kafka-exporter.json" +kubectl -n "$NS" get kafka "$CLUSTER" \ + -o jsonpath='{.spec.kafka.metricsConfig}' \ + >"$OUT/kafka-metrics-config.json" +kubectl -n "$NS" get pod \ + -l "strimzi.io/cluster=${CLUSTER},strimzi.io/name=${CLUSTER}-kafka-exporter" \ + -o wide +``` + +For ZooKeeper mode, also save its metrics configuration and confirm that all ensemble Pods exist: + +```bash +kubectl -n "$NS" get kafka "$CLUSTER" \ + -o jsonpath='{.spec.zookeeper.metricsConfig}' \ + >"$OUT/zookeeper-metrics-config.json" +kubectl -n "$NS" get pod \ + -l "strimzi.io/name=${CLUSTER}-zookeeper" -o wide +``` + +If the last command finds no Pod, Kafka Exporter is disabled or the current version uses different labels. Check `RdsKafka`, the generated `Kafka` resource, and actual Pod labels. Do not interpret an empty result as zero lag. + +## 8. Deploy Load-Test Clients + +### 8.1 Reuse the instance Kafka image + +Read the Kafka image used by the operator from a broker Pod. This keeps the script version aligned with the server version without exposing an image-registry address in an external document. + +```bash +export KAFKA_IMAGE="$(kubectl -n "$NS" get pod \ + -l "strimzi.io/name=${CLUSTER}-kafka" \ + -o jsonpath='{.items[0].spec.containers[?(@.name=="kafka")].image}')" + +test -n "$KAFKA_IMAGE" +export BOOTSTRAP="${CLUSTER}-kafka-bootstrap.${NS}.svc:9092" +``` + +The Service name follows the Strimzi internal bootstrap Service convention, but query the current instance to verify the listener, port, and authentication method: + +```bash +kubectl -n "$NS" get service -l "strimzi.io/cluster=${CLUSTER}" -o wide +kubectl -n "$NS" get kafka "$CLUSTER" \ + -o jsonpath='{.spec.kafka.listeners}' >"$OUT/listeners.json" +``` + +The following examples use a plaintext internal listener in an isolated namespace. For TLS/SASL, put all client properties in an access-controlled Secret and mount it as a read-only volume. Do not pass passwords as command-line arguments. + +### 8.2 Create client properties + +```bash +kubectl -n "$NS" create configmap kafka-perf-client-config \ + --from-literal=admin.properties='' \ + --from-literal=producer.properties='acks=all +enable.idempotence=true +batch.size=50000 +linger.ms=5 +compression.type=none +buffer.memory=134217728' \ + --from-literal=consumer.properties='auto.offset.reset=earliest +fetch.min.bytes=1 +fetch.max.wait.ms=500 +max.partition.fetch.bytes=1048576 +fetch.max.bytes=52428800' \ + --dry-run=client -o yaml | kubectl apply -f - +``` + +When authentication is enabled, prepare complete `producer.properties` and `consumer.properties` files containing the performance and connection properties in a controlled local directory. Use a Secret instead of the ConfigMap: + +```bash +kubectl -n "$NS" create secret generic kafka-perf-client-config \ + --from-file=admin.properties=/secure/path/admin.properties \ + --from-file=producer.properties=/secure/path/producer.properties \ + --from-file=consumer.properties=/secure/path/consumer.properties \ + --dry-run=client -o yaml | kubectl apply -f - +``` + +Also replace the `configMap` volume sources in both Pods in section 8.3 with: + +```yaml +secret: + secretName: kafka-perf-client-config +``` + +`admin.properties` contains only connection-security properties required by administrative scripts. The producer and consumer files separately contain their performance properties. `security.protocol`, SASL, and SSL settings must match the instance listener. Store only redacted copies in the report and evidence directory. Do not export Secrets, passwords, tokens, private keys, or unredacted Java Authentication and Authorization Service (JAAS) configuration. + +### 8.3 Create client Pods + +```bash +kubectl -n "$NS" apply -f - <"$OUT/producer-perf-help.txt" 2>&1 + +kubectl -n "$NS" exec kafka-consumer-perf -- \ + /opt/kafka/bin/kafka-consumer-perf-test.sh --help \ + >"$OUT/consumer-perf-help.txt" 2>&1 +``` + +If Kafka is installed at a path other than `/opt/kafka` in the image, confirm the path from the image documentation or broker container environment and update the commands. Do not mix in scripts downloaded from an unknown version. + +### 8.5 Legacy Kafka 2.x script compatibility + +The ZooKeeper-based operator line uses Kafka 2.x client scripts whose interfaces differ from the Kafka 4.2 examples. The reviewed legacy scripts have these differences: + +| Operation | Kafka 4.2 example in this guide | Legacy Kafka 2.x form | +| --- | --- | --- | +| Producer bootstrap and property file | `--bootstrap-server` and `--command-config` | Put `bootstrap.servers=$BOOTSTRAP` in the property file and use `--producer.config` | +| Producer warmup | `--warmup-records` | No equivalent option in the reviewed scripts; run a separate, predefined warmup cycle | +| Producer interval option | `--reporting-interval` | Not exposed by the reviewed producer script; preserve its raw interval output | +| Producer property override | `--command-property` | Use a separate complete `--producer.config` file | +| Consumer record target | `--num-records` | `--messages` | +| Consumer property file | `--command-config` | `--consumer.config` | +| Consumer concurrency | One process per consumer | Use one process per consumer for cross-version consistency | + +Some earlier Kafka 2.x scripts implement `--threads`, while later Kafka 2.x scripts accept it but ignore it. Never use `--threads` to establish the standard concurrency point. Use independent processes or Pods and verify the actual consumer group membership. + +For a legacy test, add the resolved bootstrap address to the producer property file before creating the ConfigMap or Secret: + +```properties +bootstrap.servers=perf-kafka-kafka-bootstrap.kafka-perf.svc:9092 +``` + +Replace the example value with the resolved value of `$BOOTSTRAP`; property files do not expand shell variables. Store connection-security properties in the same protected file when authentication is enabled. + +## 9. Create and Verify the Test Topic + +```bash +kubectl -n "$NS" exec kafka-producer-perf -- \ + /opt/kafka/bin/kafka-topics.sh \ + --bootstrap-server "$BOOTSTRAP" \ + --command-config /opt/perf-config/admin.properties \ + --create --if-not-exists \ + --topic "$TOPIC" \ + --partitions 30 \ + --replication-factor 3 \ + --config min.insync.replicas=2 + +kubectl -n "$NS" exec kafka-producer-perf -- \ + /opt/kafka/bin/kafka-topics.sh \ + --bootstrap-server "$BOOTSTRAP" \ + --command-config /opt/perf-config/admin.properties \ + --describe --topic "$TOPIC" \ + | tee "$OUT/topic-before.txt" + +kubectl -n "$NS" exec kafka-producer-perf -- \ + /opt/kafka/bin/kafka-configs.sh \ + --bootstrap-server "$BOOTSTRAP" \ + --command-config /opt/perf-config/admin.properties \ + --entity-type topics --entity-name "$TOPIC" --describe \ + | tee "$OUT/topic-config.txt" +``` + +Before starting, confirm that: + +- `PartitionCount=30` and `ReplicationFactor=3`; +- every partition has 3 replicas and an ISR count of 3; +- leaders are not materially imbalanced across brokers; +- the dynamic topic configuration contains `min.insync.replicas=2`; +- the topic is new for this run or has been cleaned according to the test design, and the consumer group name has not been reused. + +## 10. Run One Test Point + +### 10.1 Production reliability baseline + +Record the parameters before execution. The example is a functional check with 1,000,000 records of 500 B each. For a formal standard point, replace `--num-records` and ensure the PVC is large enough. + +```bash +export NUM_RECORDS=1000000 +export RECORD_SIZE=500 + +kubectl -n "$NS" exec kafka-producer-perf -- sh -c \ + "cat /opt/perf-config/producer.properties" \ + >"$OUT/producer.properties" + +kubectl -n "$NS" exec kafka-producer-perf -- \ + /opt/kafka/bin/kafka-producer-perf-test.sh \ + --bootstrap-server "$BOOTSTRAP" \ + --topic "$TOPIC" \ + --num-records "$NUM_RECORDS" \ + --record-size "$RECORD_SIZE" \ + --throughput -1 \ + --warmup-records 100000 \ + --reporting-interval 5000 \ + --command-config /opt/perf-config/producer.properties \ + --print-metrics \ + | tee "$OUT/producer-${RECORD_SIZE}B.txt" +``` + +`--throughput -1` disables client-side throttling and searches for the limit at the current client concurrency. It does not prove that the brokers are saturated; evaluate client and broker metrics together. Keep `--warmup-records` fixed in formal comparisons and use only the script's steady-state summary. Do not include warmup in the result. + +For a legacy ZooKeeper-based instance, use the same payload and producer properties with the legacy interface: + +```bash +kubectl -n "$NS" exec kafka-producer-perf -- \ + /opt/kafka/bin/kafka-producer-perf-test.sh \ + --topic "$TOPIC" \ + --num-records "$NUM_RECORDS" \ + --record-size "$RECORD_SIZE" \ + --throughput -1 \ + --producer.config /opt/perf-config/producer.properties \ + --print-metrics \ + | tee "$OUT/producer-${RECORD_SIZE}B.txt" +``` + +Run a separate, fixed warmup cycle before the measured legacy command. Do not count its records in the test target, and use a fresh topic or account explicitly for warmup data before a consumer-only test. + +### 10.2 Leader acknowledgment ceiling + +Keep all other properties unchanged and override only acknowledgment and idempotence: + +```properties +acks=1 +enable.idempotence=false +batch.size=50000 +linger.ms=5 +compression.type=none +buffer.memory=134217728 +``` + +Kafka 4.2 can override these two values from the property file with `--command-property`, avoiding accidental changes to other parameters: + +```bash +kubectl -n "$NS" exec kafka-producer-perf -- \ + /opt/kafka/bin/kafka-producer-perf-test.sh \ + --bootstrap-server "$BOOTSTRAP" \ + --topic "$TOPIC" \ + --num-records "$NUM_RECORDS" \ + --record-size "$RECORD_SIZE" \ + --throughput -1 \ + --warmup-records 100000 \ + --reporting-interval 5000 \ + --command-config /opt/perf-config/producer.properties \ + --command-property acks=1 \ + --command-property enable.idempotence=false \ + --print-metrics \ + | tee "$OUT/producer-leader-ack-${RECORD_SIZE}B.txt" +``` + +Label the result `leader-ack`. Do not compare it directly with `acks=all` capacity. + +For a legacy client, create a separate `producer-leader-ack.properties` file containing the same properties as the baseline except for `acks=1` and `enable.idempotence=false`, then pass it with `--producer.config`. Save a redacted diff between the two files. Do not use an unsupported `--command-property` option. + +### 10.3 Consumer-only test + +Before a consumer-only test, load at least `NUM_RECORDS` records and use a new consumer group: + +```bash +export NUM_RECORDS=1000000 +export RECORD_SIZE=500 +export GROUP="perf-consumer-$(date -u +%Y%m%dT%H%M%SZ)" + +kubectl -n "$NS" exec kafka-consumer-perf -- sh -c \ + "cat /opt/perf-config/consumer.properties" \ + >"$OUT/consumer.properties" + +kubectl -n "$NS" exec kafka-consumer-perf -- \ + /opt/kafka/bin/kafka-consumer-perf-test.sh \ + --bootstrap-server "$BOOTSTRAP" \ + --topic "$TOPIC" \ + --group "$GROUP" \ + --num-records "$NUM_RECORDS" \ + --fetch-size 200000 \ + --timeout 60000 \ + --show-detailed-stats \ + --reporting-interval 5000 \ + --command-config /opt/perf-config/consumer.properties \ + --print-metrics \ + | tee "$OUT/consumer-${RECORD_SIZE}B.txt" +``` + +In Kafka 4.2, `--messages` is deprecated; use `--num-records`. `--fetch-size` limits the data fetched from one partition in one request, and the `max.partition.fetch.bytes` client property also applies. Save `records-consumed-total` from the `--print-metrics` output and confirm that the consumer read the target count. A timeout, insufficient topic data, or authorization error can make the script exit early, so do not rely only on the final throughput line. + +For a legacy ZooKeeper-based instance, use `--messages` and `--consumer.config`: + +```bash +kubectl -n "$NS" exec kafka-consumer-perf -- \ + /opt/kafka/bin/kafka-consumer-perf-test.sh \ + --bootstrap-server "$BOOTSTRAP" \ + --topic "$TOPIC" \ + --group "$GROUP" \ + --messages "$NUM_RECORDS" \ + --fetch-size 200000 \ + --timeout 60000 \ + --show-detailed-stats \ + --reporting-interval 5000 \ + --consumer.config /opt/perf-config/consumer.properties \ + --print-metrics \ + | tee "$OUT/consumer-${RECORD_SIZE}B.txt" +``` + +The legacy script reports messages rather than records in some headings. Preserve its original output labels and normalize only in separate report fields. + +### 10.4 Concurrent produce and consume + +1. Start consumers with a new consumer group and confirm that they have joined the group. +2. Start producers. +3. Collect metrics throughout the steady-state window. +4. After producers stop, keep consumers running until lag reaches zero or a predefined timeout expires. +5. Save the final consumer position and log end offset (LEO). + +Do not run one consumer command in the foreground and then attempt to start the producer in the same terminal. Use two terminals, or start both through Jobs or a test orchestrator and save their standard output separately. Any background execution method must preserve process exit codes. Log files alone do not prove that commands succeeded. + +### 10.5 Example with 20 consumers on Kafka 4.2 + +This example starts 20 independent consumer processes in one client Pod and checks every `kubectl exec` exit code. Before a formal test, use section 12.1 to confirm that Pod CPU, memory, and network are not bottlenecks. If resources are insufficient, replicate the consumer Pod from section 8.3 and distribute the processes across load-test nodes. + +```bash +export CONSUMER_COUNT=20 +export CONSUMER_POD=kafka-consumer-perf +export GROUP="perf-consumer-20-$(date -u +%Y%m%dT%H%M%SZ)" +test "$NUM_RECORDS" -ge "$CONSUMER_COUNT" + +q=$((NUM_RECORDS / CONSUMER_COUNT)) +r=$((NUM_RECORDS % CONSUMER_COUNT)) +pids=() + +for ((i = 1; i <= CONSUMER_COUNT; i++)); do + process_records="$q" + if ((i <= r)); then + process_records=$((q + 1)) + fi + + kubectl -n "$NS" exec "$CONSUMER_POD" -- \ + /opt/kafka/bin/kafka-consumer-perf-test.sh \ + --bootstrap-server "$BOOTSTRAP" \ + --topic "$TOPIC" \ + --group "$GROUP" \ + --num-records "$process_records" \ + --fetch-size 200000 \ + --timeout 60000 \ + --reporting-interval 5000 \ + --command-config /opt/perf-config/consumer.properties \ + --command-property "client.id=perf-consumer-${i}" \ + --print-metrics \ + >"$OUT/consumer-${i}-${RECORD_SIZE}B.txt" 2>&1 & + pids+=("$!") +done + +failed=0 +for pid in "${pids[@]}"; do + wait "$pid" || failed=1 +done +test "$failed" -eq 0 +``` + +Sum the actual records consumed across all 20 files and confirm the total equals `NUM_RECORDS`. Calculate aggregate throughput as total records or bytes divided by the common measurement window. If process start and end times differ, do not add the individual average rates. A timeout, nonzero exit, or record-count mismatch in any process invalidates the test point. + +For legacy Kafka 2.x, use the same 20-process allocation but replace `--num-records` with `--messages` and `--command-config` with `--consumer.config`. Do not add `--threads`; on later Kafka 2.x scripts it is deprecated and ignored. + +## 11. Client Parameter Reference + +### 11.1 Producer + +| Parameter | Standard value | Mechanism and boundary | +| --- | ---: | --- | +| `--num-records` | See the matrix | Records sent; total payload is approximately this value multiplied by `--record-size` | +| `--record-size` | 100/500/1000 | Bytes in the synthetic record value; mutually exclusive with a message payload file | +| `--throughput` | `-1` | Unthrottled limit search; for controlled load, set the target records/s | +| `--warmup-records` | Fixed value | Kafka 4.2 option that excludes connection setup, metadata, and JVM warmup; use a separate fixed warmup cycle on the reviewed legacy scripts | +| `acks` | `all` or `1` | Acknowledgment strength; changing it changes reliability and performance semantics | +| `enable.idempotence` | `true` | Prevents duplicates caused by retries; requires compatible acknowledgments, retries, and in-flight request settings | +| `batch.size` | `50000` | Target maximum batch size per partition; a large value can increase memory use and wait time | +| `linger.ms` | `5` | Maximum wait for batching; Kafka 4.x defaults differ from earlier versions, so set it explicitly | +| `compression.type` | `none` | The standard matrix disables compression; use the production algorithm in the application matrix and record CPU and measured compression ratio | +| `buffer.memory` | `134217728` | Total producer memory available for records waiting to be sent; not the complete container-memory requirement | + +Each partition has an independent batch. More producers, partitions, batch capacity, or linger time can improve batching but can also increase memory use and latency. Attribute a result to a parameter only when all other variables remain fixed. + +### 11.2 Consumer + +| Parameter | Standard value | Mechanism and boundary | +| --- | ---: | --- | +| `--group` | Unique per run | Consumers in the same group share partitions; a reused group can start at old offsets | +| `--num-records` or legacy `--messages` | Matches preloaded data | Planned total records; select the version-supported option and verify the actual value with client metrics | +| `--fetch-size` | `200000` | Per-partition fetch size passed by the script | +| `--timeout` | `60000` | Maximum time between returned records, not total test duration; a timeout makes the tool exit early with a warning | +| `fetch.min.bytes` | `1` | Minimum data before the broker returns a Fetch response; increasing it can improve batching but can add wait latency | +| `fetch.max.wait.ms` | `500` | Maximum wait when `fetch.min.bytes` has not been reached | +| `max.partition.fetch.bytes` | `1048576` | Per-partition limit for one Fetch response; it must accommodate the maximum record batch allowed by the broker | +| `fetch.max.bytes` | `52428800` | Overall limit for one Fetch response, still subject to server and partition limits | + +Effective parallelism in one consumer group cannot exceed the number of assignable partitions. This guide uses one consumer per script process in every Kafka version. To run 2, 4, or 8 consumers, create independent Pods or processes with identical properties and the same consumer group. Consumers beyond the partition count receive no partitions. + +`NUM_RECORDS` in the matrix is the aggregate target across all processes, not a per-process target. With concurrency `C`, set `q=NUM_RECORDS/C` and `r=NUM_RECORDS%C`. Give `q+1` records to the first `r` processes and `q` to the remainder, so the targets sum to `NUM_RECORDS`. Use the same allocation for multi-producer tests. If every process produces or consumes all `NUM_RECORDS`, the total data volume has changed and the result cannot be compared directly with the single-process point. + +## 12. Find the Saturation Point and Usable Capacity + +### 12.1 Exclude client limits first + +Probe producers and consumers separately with 1, 2, 4, and 8 independent client processes. For the standard consumer point, continue to 12, 16, and 20 unless partition parallelism or a resource limit is reached first. Distribute clients across load-test nodes when possible, use identical properties for each process, and aggregate process throughput. At each increase in concurrency, check at least: + +- client Pod CPU, memory, CPU throttling, network, and restart count; +- broker CPU, memory, network, disk latency, and disk utilization; +- request errors, timeouts, retries, and produce acknowledgment latency; +- `UnderReplicatedPartitions`, `UnderMinIsrPartitionCount`, and consumer group lag; +- partition leaders, log bytes, and traffic balance across brokers; +- for ZooKeeper mode, ZooKeeper Pod CPU, memory, disk latency and utilization, request latency, outstanding requests, quorum size, and broker ZooKeeper session state. + +If throughput continues to increase materially after client resources or processes are added, the prior result was a client limit and must not be reported as the Kafka limit. If the network interface on one load-test node is saturated, add client nodes instead of adding more processes on that node. + +### 12.2 Two-phase load staircase + +1. **Unthrottled probe:** Use `--throughput -1` and increase client processes to establish an achievable throughput range. +2. **Controlled-load validation:** Starting from the unthrottled result, apply approximately 25%, 50%, 75%, 90%, 100%, and 110% target throughput. The sum of `--throughput` across all producers equals the target for that point. +3. Warm up each point, then maintain steady state for at least 10 minutes. Afterward, wait for lag to reach zero and verify replica health. +4. Add denser test points near 90%. Repeat both the last stable point and first unstable point at least three times. +5. Randomize repeat order or rerun the baseline to detect thermal effects, cache state, noisy neighbors, and time drift. + +The percentages are only a starting strategy for finding the range; they are not product thresholds. If the first load point violates a health condition, reduce the load and re-establish the range. + +### 12.3 Define acceptance criteria in advance + +Before testing, the customer must set these thresholds from the service level objective (SLO) for the application: + +- p95 and p99 produce acknowledgment latency; +- error, timeout, and retry rates; +- maximum consumer lag and drain time; +- resource-utilization limits for brokers, clients, nodes, and PVCs; +- permitted `UnderReplicatedPartitions` and `UnderMinIsrPartitionCount`, normally both always zero; +- for ZooKeeper mode, stable ensemble membership, exactly one active Kafka controller, acceptable ZooKeeper request latency, no sustained growth in outstanding requests, and no broker ZooKeeper session loss; +- steady-state window and repeat count. + +If the customer has no thresholds, use the following as engineering rules for this test and label them as test rules, not official Kafka thresholds: + +- after a load or concurrency increase, aggregate throughput improves by less than 5% for two consecutive steps; and +- p99 latency, errors/timeouts, lag, or resource utilization continues to worsen; or +- `UnderReplicatedPartitions>0`, `UnderMinIsrPartitionCount>0`, an offline log directory, a Pod restart, or disk pressure occurs; or +- in ZooKeeper mode, ensemble membership changes, the sum of `ActiveControllerCount` across brokers differs from 1, a broker loses its ZooKeeper session, or ZooKeeper request latency or outstanding requests breach the predefined test threshold. + +The first point meeting the combined conditions is the **saturation point**. The last point before saturation that satisfies all SLOs, has stable repeat results, and has no health anomaly is the **usable capacity**. Do not substitute an instantaneous maximum for usable capacity. + +Report repeat variability. By default, use the median as the center and also report the minimum, maximum, and coefficient of variation. If throughput at one test point has a coefficient of variation greater than 5%, investigate environmental drift and add repetitions. The 5% value is also a rule for this test, not a Kafka guarantee. + +## 13. Instance Parameter Tuning + +Complete one run with the template baseline, then change one parameter at a time in response to monitoring evidence. The following entries are diagnostic directions, not fixed optimal values. + +| Parameter or resource | When to consider a change | Validation | Risk and stop condition | +| --- | --- | --- | --- | +| Broker CPU | Request-processing threads are busy, CPU remains near its constraint, and clients are not saturated | Add CPU and repeat the same load | If throughput does not improve while disk or network is full, more CPU will not help | +| Broker memory/JVM heap | Container memory pressure, frequent garbage collection (GC), or insufficient page cache | Observe heap, GC, resident set size (RSS), working set, and disk reads together | An oversized heap displaces page cache; a heap too close to the container limit can cause an out-of-memory (OOM) event | +| `num.network.threads` | `NetworkProcessorAvgIdlePercent` remains low and CPU headroom exists | Increase in small steps and repeat the same point | More threads add scheduling and memory overhead and cannot fix a saturated network interface | +| `num.io.threads` | `RequestHandlerAvgIdlePercent` remains low and disk and CPU headroom exist | Increase in small steps and compare request wait and throughput | Cover at least the number of data volumes; too many threads can increase context switching and I/O contention | +| `num.replica.fetchers` | Replicas catch up slowly and replication traffic has unused resources | Validate during controlled recovery or high replication load | More fetchers increase network, disk, and CPU use | +| `num.recovery.threads.per.data.dir` | Startup or recovery is too slow | Validate in a controlled recovery test | Faster recovery increases interference with foreground I/O | +| Topic partition count | Per-partition throughput is limiting and enough consumer concurrency exists | Compare new topics with different partition counts | More partitions change key-ordering scope and add metadata and replica overhead; partitions cannot be reduced arbitrarily | +| Broker count | A broker has reached its resource constraint and horizontal scaling is required | Scale out, wait for replica reassignment to finish, then retest | Adding brokers alone does not rebalance existing partition data; results during reassignment are invalid | +| `message.max.bytes` | The largest application record batch exceeds the default | Check topic, producer, and consumer fetch limits together | A larger value increases memory and network bursts; inconsistent settings cause produce or consume failures | +| Retention and log-segment parameters | Disk use, deletion granularity, or recovery time does not meet requirements | Test retention and deletion behavior over a long interval | Do not change production retention only to improve a short benchmark | +| ZooKeeper CPU, memory, or JVM heap | ZooKeeper request latency or outstanding requests rise while storage has headroom | Change one resource dimension and repeat the same metadata and traffic load | Oversized heap can increase pause time; resource changes invalidate direct comparison unless they are the tested variable | +| ZooKeeper storage | Request latency correlates with disk latency, queue depth, or synchronous-write pressure | Qualify lower-latency persistent block storage in a separate instance | A storage-class or media change creates a new environment baseline; do not present it as a broker-only tuning gain | +| ZooKeeper replica count | Availability requirements or production topology require a different odd-sized ensemble | Validate quorum health and failover outside the measured capacity window | Replica count is not a routine throughput knob; an even count adds no failure tolerance over the preceding odd count and reconfiguration invalidates the measurement window | + +For Apache Kafka thread-idle metrics, a value closer to zero means the threads are busier. If `NetworkProcessorAvgIdlePercent` or `RequestHandlerAvgIdlePercent` remains below approximately 0.3, evaluate CPU, network, and storage together for insufficient processing capacity. This value is a diagnostic signal, not an independent scaling trigger. + +After each change: + +1. Save the complete configuration diff before and after the change. +2. Apply a rolling restart or dynamic update according to the installed product version. +3. Wait until Kafka resources are Ready and the ISR is complete. For KRaft, wait for stable controller quorum and NodePool status. For ZooKeeper mode, wait for all ZooKeeper Pods, stable ensemble membership, connected broker sessions, and exactly one active Kafka controller. +4. Repeat the test at least three times with the same topic data state, client concurrency, and load. +5. Return to the previous validated configuration if the gain is unstable or health metrics deteriorate. + +## 14. Monitoring and Evidence Collection + +### 14.1 Kafka health checks + +Run these checks before the test, during the steady-state window, and after the test. Discover a broker Pod by label: + +```bash +export BROKER_POD="$(kubectl -n "$NS" get pod \ + -l "strimzi.io/name=${CLUSTER}-kafka" \ + -o jsonpath='{.items[0].metadata.name}')" +test -n "$BROKER_POD" +``` + +For KRaft, record controller-quorum status: + +```bash +kubectl -n "$NS" exec kafka-producer-perf -- \ + /opt/kafka/bin/kafka-metadata-quorum.sh \ + --bootstrap-server "$BOOTSTRAP" \ + --command-config /opt/perf-config/admin.properties \ + describe --status \ + | tee "$OUT/metadata-quorum-after.txt" +``` + +For ZooKeeper mode, `kafka-metadata-quorum.sh` is not applicable. Record the generated StatefulSets, every ZooKeeper Pod, and current resource status instead: + +```bash +kubectl -n "$NS" get statefulset \ + "${CLUSTER}-kafka" "${CLUSTER}-zookeeper" -o wide \ + | tee "$OUT/kafka-zookeeper-statefulsets-after.txt" +kubectl -n "$NS" get pod \ + -l "strimzi.io/name=${CLUSTER}-zookeeper" -o wide \ + | tee "$OUT/zookeeper-pods-after.txt" +kubectl -n "$NS" get kafka "$CLUSTER" -o yaml \ + >"$OUT/kafka-resource-after.yaml" +``` + +Run these broker and topic checks in both metadata modes: + +```bash +kubectl -n "$NS" exec kafka-producer-perf -- \ + /opt/kafka/bin/kafka-topics.sh \ + --bootstrap-server "$BOOTSTRAP" \ + --command-config /opt/perf-config/admin.properties \ + --describe --under-replicated-partitions \ + | tee "$OUT/under-replicated-after.txt" + +kubectl -n "$NS" exec kafka-producer-perf -- \ + /opt/kafka/bin/kafka-consumer-groups.sh \ + --bootstrap-server "$BOOTSTRAP" \ + --command-config /opt/perf-config/admin.properties \ + --describe --group "$GROUP" \ + | tee "$OUT/consumer-group-after.txt" + +kubectl -n "$NS" exec kafka-producer-perf -- \ + /opt/kafka/bin/kafka-log-dirs.sh \ + --bootstrap-server "$BOOTSTRAP" \ + --command-config /opt/perf-config/admin.properties \ + --describe --topic-list "$TOPIC" \ + >"$OUT/log-dirs-after.txt" +``` + +`--under-replicated-partitions` can produce no output in a healthy state. Also save its exit code, or have the orchestrator record successful completion, so a connection failure is not misclassified as no under-replicated partitions. + +`kafka-log-dirs.sh` prints status text before its JSON data, so the example saves raw text. Before parsing it programmatically, validate and remove the non-JSON prefix. Do not pass the whole file directly to a JSON parser. + +### 14.2 Required Kafka metrics + +The following Java Management Extensions (JMX) managed beans (MBeans), or their Prometheus mappings, must cover the complete steady-state window: + +| Category | Metric or JMX MBean | Purpose | +| --- | --- | --- | +| Write/read | `BytesInPerSec`, `BytesOutPerSec`, `MessagesInPerSec` | Cross-check client throughput | +| Replica health | `UnderReplicatedPartitions`, `UnderMinIsrPartitionCount` | The first shows ISR count below the configured replication factor; the second shows ISR count below `min.insync.replicas` | +| Log directory | `OfflineLogDirectoryCount` | Detect unavailable storage directories | +| Network threads | `NetworkProcessorAvgIdlePercent` | Determine network-processor thread utilization | +| Request threads | `RequestHandlerAvgIdlePercent` | Determine request-handler thread utilization | +| Request latency | Produce/Fetch `TotalTimeMs`, `RequestQueueTimeMs`, `LocalTimeMs`, `RemoteTimeMs`, `ResponseQueueTimeMs`, and `ResponseSendTimeMs` | Identify the request stage where time is spent | +| Consumer group | Current position, LEO, per-partition lag, and total lag | Determine whether consumers keep up with production | +| ZooKeeper-mode controller | `ActiveControllerCount`, `OfflinePartitionsCount`, and leader-election metrics | Confirm that exactly one broker is the active controller and detect controller or partition instability | +| ZooKeeper-mode broker session | `SessionState` | Detect a broker that is disconnected from ZooKeeper or has an expired session | + +Prometheus metric names from JMX Exporter depend on the instance metric-mapping rules. Do not assume all environments use the same names. Save raw metrics from the broker `/metrics` endpoint and inspect the names first: + +```bash +kubectl -n "$NS" exec "$BROKER_POD" -- \ + curl -fsS http://127.0.0.1:9404/metrics \ + >"$OUT/broker-jmx-metrics.txt" + +grep -m 100 -E 'bytesin|bytesout|messagesin|underreplicated|underminisr|avgidle' \ + "$OUT/broker-jmx-metrics.txt" +``` + +If Kafka Exporter is enabled, discover its Pod and save its raw metrics: + +```bash +export EXPORTER_PODS="$(kubectl -n "$NS" get pod \ + -l "strimzi.io/cluster=${CLUSTER},strimzi.io/name=${CLUSTER}-kafka-exporter" \ + -o jsonpath='{.items[*].metadata.name}')" + +if test -n "$EXPORTER_PODS"; then + export EXPORTER_POD="${EXPORTER_PODS%% *}" + kubectl -n "$NS" exec "$EXPORTER_POD" -- \ + curl -fsS http://127.0.0.1:9404/metrics \ + >"$OUT/kafka-exporter-metrics.txt" +else + printf '%s\n' 'Kafka Exporter Pod not found' \ + >"$OUT/kafka-exporter-not-found.txt" +fi +``` + +Raw endpoint output proves only that the exporter exposes metrics; it does not prove that the platform is collecting time-series data. Before formal testing, query broker metrics for the last 15 minutes in platform monitoring or Prometheus and confirm continuous samples for every broker. If the query is empty or has collection gaps, stop the test and correct the ServiceMonitor or PodMonitor and metric mapping for the current platform. Selector labels depend on the current Prometheus configuration; do not copy labels from another cluster. + +For ZooKeeper mode, enable the `spec.zookeeper.metricsConfig` mapping and collect at least: + +| ZooKeeper evidence | Purpose | +| --- | --- | +| `QuorumSize` | Confirm the observed ensemble size remains equal to the configured replica count | +| `NumAliveConnections` | Detect connection loss or an unexpected connection increase | +| `OutstandingRequests` | Detect work arriving faster than a server can process it | +| `MinRequestLatency`, `AvgRequestLatency`, `MaxRequestLatency` | Correlate metadata-service latency with Kafka controller events and storage latency | +| `NodeCount` and `WatchCount` | Preserve metadata and watch-set scale for comparability | +| ZooKeeper Pod CPU, memory, restart count, JVM GC, and PVC usage | Exclude resource pressure and restarts | +| ZooKeeper volume latency, utilization, queue depth, and free space | Detect synchronous-write or storage-capacity constraints | + +The exact Prometheus names depend on the installed mapping. In the reviewed legacy mapping they are lower-case names such as `zookeeper_quorumsize`, `zookeeper_numaliveconnections`, `zookeeper_outstandingrequests`, and `zookeeper_avgrequestlatency`. Confirm names from every ZooKeeper Pod's raw endpoint before querying them: + +```bash +export ZOOKEEPER_PODS="$(kubectl -n "$NS" get pod \ + -l "strimzi.io/name=${CLUSTER}-zookeeper" \ + -o jsonpath='{.items[*].metadata.name}')" +test -n "$ZOOKEEPER_PODS" + +for pod in $ZOOKEEPER_PODS; do + kubectl -n "$NS" exec "$pod" -- \ + curl -fsS http://127.0.0.1:9404/metrics \ + >"$OUT/${pod}-jmx-metrics.txt" +done +``` + +Do not copy example alert thresholds into the capacity acceptance criteria. Establish a pre-test baseline and use the customer's SLOs. A request-latency increase, sustained outstanding-request growth, quorum change, broker session loss, or active-controller count other than one invalidates the point even when producer throughput remains high. + +### 14.3 Kubernetes and infrastructure metrics + +Collect at least the following time-series data rather than only a post-test snapshot: + +- broker, controller, ZooKeeper, and client Pods, as applicable: CPU use, CPU throttling, memory working set, network transmit and receive, restarts, and OOM events; +- nodes: CPU, memory, network bandwidth and packet loss, disk throughput, input/output operations per second (IOPS), average and percentile latency, utilization, and queue depth; +- PVCs: used bytes, capacity, utilization, and growth rate; +- Kubernetes events, Pod placement, and resource requests and limits. + +`kubectl top` provides only a near-real-time snapshot. Use it for a quick check, not as a substitute for a Prometheus range query: + +```bash +kubectl -n "$NS" top pod --containers \ + | tee "$OUT/top-pods-after.txt" +kubectl -n "$NS" get events --sort-by=.lastTimestamp \ + >"$OUT/events.txt" +kubectl -n "$NS" get pod -o wide \ + >"$OUT/pods-after.txt" +kubectl -n "$NS" get pvc -o wide \ + >"$OUT/pvc-after.txt" +``` + +The following Prometheus Query Language (PromQL) expressions show calculation intent only. Inspect actual metric names and labels in the current Prometheus instance before replacing `$NS`, `$TOPIC`, `$GROUP`, and cluster labels. Save each expression, time range, step, and raw response. + +```text +sum(rate(kafka_server_brokertopicmetrics_messagesin_total{namespace="$NS",topic="$TOPIC"}[5m])) + +sum(rate(kafka_server_brokertopicmetrics_bytesin_total{namespace="$NS",topic="$TOPIC"}[5m])) + +sum(rate(kafka_server_brokertopicmetrics_bytesout_total{namespace="$NS",topic="$TOPIC"}[5m])) + +sum(kafka_server_replicamanager_underreplicatedpartitions{namespace="$NS"}) + +sum(kafka_server_replicamanager_underminisrpartitioncount{namespace="$NS"}) + +sum(kafka_controller_kafkacontroller_activecontrollercount{namespace="$NS"}) + +min(kafka_network_socketserver_networkprocessoravgidle_percent{namespace="$NS"}) + +min(kafka_server_kafkarequesthandlerpool_requesthandleravgidle_percent{namespace="$NS"}) + +sum(kafka_consumergroup_lag{namespace="$NS",consumergroup="$GROUP",topic="$TOPIC"}) + +sum by (pod) ( + rate(container_cpu_usage_seconds_total{namespace="$NS",container!="",image!=""}[5m]) +) + +max by (pod) ( + container_memory_working_set_bytes{namespace="$NS",container!="",image!=""} +) + +sum by (pod) ( + rate(container_network_receive_bytes_total{namespace="$NS"}[5m]) +) + +sum by (pod) ( + rate(container_network_transmit_bytes_total{namespace="$NS"}[5m]) +) + +kubelet_volume_stats_used_bytes{namespace="$NS"} +/ +kubelet_volume_stats_capacity_bytes{namespace="$NS"} + +max(zookeeper_quorumsize{namespace="$NS",strimzi_io_cluster="$CLUSTER"}) + +max(zookeeper_outstandingrequests{namespace="$NS",strimzi_io_cluster="$CLUSTER"}) + +max(zookeeper_avgrequestlatency{namespace="$NS",strimzi_io_cluster="$CLUSTER"}) +``` + +CPU-throttling and node-disk metric names vary by container runtime, cgroup version, and monitoring stack. If the platform has no corresponding time series, add collection before testing or export equivalent evidence from the node runtime, cgroup, and storage system. Do not record an empty query as zero. + +### 14.4 Redis or other related components + +If the application path also includes Redis, use the same `RUN_ID`, UTC interval, and node-event record for Redis, and attach a separate Redis performance report. Kafka and Redis throughput, latency, and saturation points are different metrics. Do not combine them or infer causation from timing correlation alone. The Kafka report should retain only the related report link, versions, test interval, and verified dependency. + +## 15. Result Summary and Validity Checks + +### 15.1 Fields for each test point + +Record at least: + +```text +run_id,start_utc,end_utc,measurement_seconds,test_mode,reliability_profile, +kafka_version,operator_version,metadata_mode,kraft_roles,broker_count,controller_count, +zookeeper_replicas,broker_cpu,broker_memory,jvm_xms,jvm_xmx,storage_class,pvc_size, +topic,partitions,replication_factor,min_insync_replicas, +record_count_target,producer_records_sent,consumer_records_consumed, +record_size,compression,compression_ratio,key_distribution, +producer_processes,consumer_processes,acks,idempotence,batch_size,linger_ms, +throughput_target,fetch_min_bytes,fetch_max_wait_ms,max_partition_fetch_bytes, +producer_records_s,producer_mib_s,producer_avg_ms,producer_p50_ms, +producer_p95_ms,producer_p99_ms,producer_p999_ms, +consumer_records_s,consumer_mib_s,max_lag,final_lag,error_count,timeout_count, +broker_cpu_cores_max,broker_memory_bytes_max,broker_network_bps_max,pvc_used_bytes_max, +network_idle_min,request_idle_min,under_replicated_max,under_min_isr_max, +active_controller_count,zookeeper_quorum_size,zookeeper_request_latency_max, +zookeeper_outstanding_requests_max,zookeeper_pvc_used_bytes_max, +client_cpu_cores_max,client_cpu_throttling_ratio_max,result_valid,invalid_reason +``` + +Kafka script output uses the column label `MB/sec`. Preserve the original label in the report. To normalize the value to MiB/s, calculate actual payload bytes divided by seconds and by `2^20`, record the formula, and keep the original value. Do not only rename the column. + +Resource peak fields record the maximum for one Pod or PVC by default. If a sum, average, or other aggregation is used, document the aggregation function, unit, interval, and sampling step in the field definition. + +### 15.2 Requirements for a valid result + +Mark a result `valid` only when all of the following are true: + +- command exit codes are zero and producer and consumer record counts reach their targets; +- warmup and steady-state windows complete as planned and their clock intervals are explicit; +- Kafka, operator, metadata mode, configuration, topic, client, and infrastructure snapshots are complete; +- no broker, controller, ZooKeeper, or client Pod restart, OOM event, node pressure, volume anomaly, or unplanned maintenance occurs; +- `UnderReplicatedPartitions=0`, `UnderMinIsrPartitionCount=0`, and `OfflineLogDirectoryCount=0`; +- the production reliability baseline preserves `acks=all`, idempotence, and minimum ISR requirements; +- the client is not a confirmed bottleneck, or the result is explicitly labeled as a client limit; +- monitoring covers the complete steady-state window without collection gaps; +- in ZooKeeper mode, ensemble size is stable, the sum of `ActiveControllerCount` is 1, broker sessions remain connected, and ZooKeeper request latency and outstanding requests stay within the predefined test criteria; +- variation between repetitions is explained or uncertainty is retained in the conclusion. + +Mark a result `invalid` and repeat the test if any of the following occur: + +- the topic contains unknown data, a consumer group is reused, or the actual record count is insufficient; +- a rolling restart, reassignment, frequent partition-leader change, controller election, ZooKeeper ensemble reconfiguration, or replica catch-up occurs during the test; +- a PVC approaches capacity or storage or node pressure occurs; +- a client is CPU-throttled, network-saturated, or exits abnormally; +- more than one key variable changes between test points; +- only averages are available, without raw output, percentile latency, or time-series monitoring. + +### 15.3 Writing conclusions + +Separate conclusions into: + +- **Facts:** Direct observations from raw output, configuration snapshots, and monitoring. +- **Inferences:** Bottleneck diagnoses supported by multiple metrics, including evidence against the diagnosis. +- **Recommendations:** The next tuning, scaling, or retest action. +- **Items to verify:** Conclusions that cannot yet be made because storage, network, production message model, or long steady-state evidence is missing. + +Also report: + +1. the last usable-capacity point that satisfies all SLOs; +2. the first unstable saturation point and its triggering conditions; +3. the median, range, and coefficient of variation from at least three repetitions; +4. the relative change from the baseline while holding reliability and environment constant; +5. applicability boundaries, especially the message model, storage, network, authentication, retention policy, and test duration. + +## 16. Cleanup + +Archive and verify `$OUT` before deleting resources dedicated to the run. Do not delete shared resources with ambiguous labels or wildcards. + +```bash +# Delete only the dedicated topic created by this guide. Recheck its name first. +kubectl -n "$NS" exec kafka-producer-perf -- \ + /opt/kafka/bin/kafka-topics.sh \ + --bootstrap-server "$BOOTSTRAP" \ + --command-config /opt/perf-config/admin.properties \ + --delete --topic "$TOPIC" + +kubectl -n "$NS" delete pod \ + kafka-producer-perf kafka-consumer-perf --ignore-not-found +kubectl -n "$NS" delete configmap \ + kafka-perf-client-config --ignore-not-found +kubectl -n "$NS" delete secret \ + kafka-perf-client-config --ignore-not-found +``` + +If the entire namespace was created for this run and contains no shared resources, delete it through the change process. Before deleting an instance or namespace, check PVC `deleteClaim` and the StorageClass reclaim policy and confirm that data retention or destruction meets requirements. + +## 17. References + +- [Apache Kafka 4.2 Broker Configuration](https://kafka.apache.org/42/configuration/broker-configs/) +- [Apache Kafka 4.2 Producer Configuration](https://kafka.apache.org/42/configuration/producer-configs/) +- [Apache Kafka 4.2 Consumer Configuration](https://kafka.apache.org/42/configuration/consumer-configs/) +- [Apache Kafka 4.2 Monitoring](https://kafka.apache.org/42/operations/monitoring/) +- [Apache Kafka 4.2 Hardware and Operating System](https://kafka.apache.org/42/operations/hardware-and-os/) +- [Apache Kafka 2.8 ZooKeeper Operations](https://kafka.apache.org/28/operations/zookeeper/) +- [Apache Kafka 2.8 Monitoring](https://kafka.apache.org/28/operations/monitoring/) +- [Strimzi 0.25 Deploying and Upgrading Guide](https://strimzi.io/docs/operators/0.25.0/deploying) +- [Strimzi 0.48 Deploying and Managing Guide](https://strimzi.io/docs/operators/0.48.0/deploying.html) +- [Prometheus Operator API Reference](https://prometheus-operator.dev/docs/api-reference/api/) diff --git a/docs/zh/solutions/ecosystem/kafka/aKafka_Performance_Testing_Guide.md b/docs/zh/solutions/ecosystem/kafka/aKafka_Performance_Testing_Guide.md deleted file mode 100644 index cf64f24a2..000000000 --- a/docs/zh/solutions/ecosystem/kafka/aKafka_Performance_Testing_Guide.md +++ /dev/null @@ -1,1035 +0,0 @@ ---- -products: - - Alauda Application Services -kind: - - Solution ---- - -# aKafka 性能测试指南 - -## 1. 目的和适用范围 - -本文用于在 Alauda Container Platform（ACP）上对 aKafka 实例执行可重复、可审计的性能测试，并回答以下问题： - -本文沿用 Apache Kafka 官方术语：服务端代理节点（Broker）、主题（Topic）、控制器（Controller）和同步副本集合（in-sync replicas，ISR）。 - -- 在给定副本、确认和压缩策略下，实例可持续处理的生产、消费吞吐量是多少； -- 在延迟、错误率、副本同步和资源水位满足约束时，实例的可用容量是多少； -- 瓶颈位于客户端、Broker 的 CPU/内存/网络、持久卷，还是分区与副本布局； -- 调整实例或客户端参数后，结果是否可比，收益是否可复现。 - -本文的命令按 Apache Kafka 4.2.0 的脚本接口编写。其他版本必须先保存 `--version` 和 `--help` 输出，再按该版本支持的参数调整。特别是，Kafka 4.2 的 `kafka-consumer-perf-test.sh` 没有旧版本的 `--threads` 参数；需要并发时，应增加独立进程或 Pod。 - -本文不提供跨环境通用的“最佳吞吐量”或“最佳参数”。硬件、存储、网络、消息大小、压缩率、可靠性策略和业务负载任一项变化，结果都可能失去可比性。 - -> **风险提示**：压测会大量写入数据，并可能耗尽磁盘、网络或节点资源。只在获得授权的隔离实例、命名空间和时间窗口内执行。不要在承载生产流量的实例上寻找饱和点。 - -## 2. 结果口径 - -测试前必须选择一种负载模式和一种可靠性配置。不同配置的结果不得合并比较。 - -### 2.1 负载模式 - -| 模式 | 目的 | 执行方式 | 主要结果 | -| --- | --- | --- | --- | -| 生产者单测 | 测量写入能力 | 只运行生产者 | records/s、MiB/s、生产确认延迟 | -| 消费者单测 | 测量读取能力 | 先准备足量存量数据，再运行消费者 | records/s、MiB/s、消费组积压 | -| 生产与消费并发 | 模拟持续流转 | 先启动消费者，再启动生产者 | 两端吞吐量、积压增长、Broker 资源和稳定性 | - -生产者脚本报告的延迟是生产请求确认延迟，不是业务端到端延迟。消费者脚本报告吞吐量，但不提供消息从生产到消费的端到端延迟。端到端延迟必须在消息中写入发送时间戳，并由业务压测程序在消费端计算；不要用两段脚本输出相减代替。 - -### 2.2 可靠性配置 - -| 配置 | Topic | 生产者 | 用途 | -| --- | --- | --- | --- | -| 生产可靠性基线 | `replication.factor=3`、`min.insync.replicas=2` | `acks=all`、`enable.idempotence=true` | 容量规划和生产验收的默认配置 | -| Leader 确认上限 | `replication.factor=3`，其余参数与被测基线一致 | `acks=1`、`enable.idempotence=false` | 估算降低确认强度后的吞吐上限 | - -`acks=1` 只等待 Leader 确认；Leader 随后立即故障时，尚未复制的数据可能丢失。Kafka 的幂等生产要求 `acks=all`，因此执行 `acks=1` 测试时必须显式设置 `enable.idempotence=false`，避免客户端配置冲突。容量结论必须标注可靠性配置，不能把 Leader 确认上限作为生产可靠性容量。 - -### 2.3 访问路径 - -外部连接可能使用传输层安全性（Transport Layer Security，TLS）和简单认证与安全层（Simple Authentication and Security Layer，SASL）；测试必须使用与业务计划相同的安全配置。 - -| 访问路径 | 客户端位置 | 结果包含的约束 | 使用场景 | -| --- | --- | --- | --- | -| 集群内容量 | 业务集群内的专用压测节点，通过内部 Bootstrap Service 连接 | Kafka、集群网络和存储 | 实例容量和调参，本文命令默认使用该路径 | -| 外部端到端 | 客户指定的源集群或主机，通过实际外部监听器连接 | 以上约束，加上负载均衡器、广域网、防火墙、TLS/SASL 和客户端出口 | 业务接入验收 | - -两种路径的结果不能合并。外部端到端测试必须保存客户端位置、监听器、往返时延、丢包和网络路径，并确认客户端不仅能访问 Bootstrap 地址，还能访问 Kafka 元数据中公布的每个 Broker 地址。完成 4.3 节的变量初始化后，从当前资源状态读取监听器地址，不应按命名规则推测： - -```bash -kubectl -n "$NS" get kafka "$CLUSTER" \ - -o jsonpath='{range .status.listeners[*]}{.name}{"\t"}{.bootstrapServers}{"\n"}{end}' -``` - -## 3. 标准测试矩阵 - -先执行标准矩阵以获得横向可比的数据，再补充与真实业务一致的消息模型、连接安全和并发模式。 - -### 3.1 实例和消息矩阵 - -| Broker 资源规格 | Broker 数量 | Topic 分区 | 副本数 | 消息大小 | -| --- | ---: | ---: | ---: | --- | -| 1 vCPU / 2 GiB | 3 | 30 | 3 | 100 B、500 B、1000 B | -| 2 vCPU / 4 GiB | 3 | 30 | 3 | 100 B、500 B、1000 B | -| 4 vCPU / 8 GiB | 3 | 30 | 3 | 100 B、500 B、1000 B | -| 8 vCPU / 16 GiB | 3 | 30 | 3 | 100 B、500 B、1000 B | - -标准点位使用以下记录数： - -- 1、2、4 vCPU：每个消息大小发送 50,000,000 条； -- 8 vCPU：100 B 和 500 B 各发送 100,000,000 条，1000 B 发送 50,000,000 条。 - -标准生产者点先使用 1 个生产者进程；如果第 12 节证明客户端先达到上限，则增加进程并把聚合结果单独标记。标准消费者点的目标并发为 20 个消费者。Kafka 4.2 不再提供旧版脚本的 `--threads 20`，因此应运行 20 个独立进程，并让它们使用同一消费组；这些进程可分布到多个资源充足的 Pod，但必须记录每个 Pod 的进程数和资源。 - -如果测试窗口或磁盘容量不足，可以减少记录数，但每个稳态点应持续至少 10 分钟，并在报告中记录实际记录数和持续时间。短时突发结果不能替代稳态容量。 - -### 3.2 业务补充矩阵 - -标准矩阵之外，至少增加一组与生产计划一致的配置： - -- 消息大小分布，而不是只使用平均值； -- 消息键（record key）分布和分区策略； -- `compression.type`； -- TLS、SASL 等连接安全； -- 生产者和消费者实例数； -- Topic 数量、分区数、副本数和保留时间； -- 实际的突发与空闲周期。 - -如果没有业务分布数据，应将结论标记为“合成负载结果”，不能推断为业务容量。 - -## 4. 测试前置条件 - -### 4.1 环境隔离 - -1. 创建专用命名空间、aKafka 实例、Topic、消费组和压测客户端。 -2. Broker 应分散到不同 Kubernetes 节点；压测客户端不要与 Broker 共用节点，否则客户端争抢资源会污染结果。 -3. 测试期间不得运行扩缩容、版本升级、节点维护、存储迁移或其他高负载任务。 -4. 节点和持久卷类型必须与目标环境一致。生产容量测试优先使用本地块存储或经验证的块存储；不要把未经验证的共享文件存储结果外推到块存储环境。 -5. 每组实例参数测试使用新实例，或在清理 Topic 后等待实例恢复稳定。修改需要重启的参数后，必须等待所有 Pod Ready、副本同步完成，再开始下一轮。 - -### 4.2 必需工具和权限 - -- 可访问业务集群的 `kubectl`； -- 对测试命名空间内 aKafka、Pod、ConfigMap、Secret、Service 和持久卷声明（PersistentVolumeClaim，PVC）的必要权限； -- 可读取平台监控或 Prometheus 时序数据； -- 可从已部署 Broker Pod 读取其 Kafka 镜像名称。本文复用 Operator 管理的 Kafka 镜像，不要求任何私有镜像名称； -- 用于 SASL/TLS 的客户端配置 Secret。报告不得包含密码、令牌、私钥或完整 Secret。 - -### 4.3 建立证据目录 - -在同一个 Bash 会话中执行以下初始化；后续代码块沿用这些变量和错误处理选项： - -```bash -set -euo pipefail - -export KUBECONFIG=/path/to/business-cluster.kubeconfig -export NS=akafka-perf -export CLUSTER=perf-kafka -export TOPIC=perf-rf3-p30 -export GROUP="perf-$(date -u +%Y%m%dT%H%M%SZ)" -export RUN_ID="$(date -u +%Y%m%dT%H%M%SZ)-${CLUSTER}" -export OUT="$(pwd)/akafka-perf-${RUN_ID}" -mkdir -p "$OUT" - -kubectl version -o yaml >"$OUT/kubectl-version.yaml" -kubectl cluster-info >"$OUT/cluster-info.txt" -kubectl get nodes -o wide >"$OUT/nodes.txt" -kubectl get storageclass -o yaml >"$OUT/storageclasses.yaml" -``` - -只导出非敏感元数据。不要把 kubeconfig、Secret 数据、仓库登录信息或客户端密码写入证据目录。 - -## 5. 导出并冻结实例参数 - -aKafka 的通用参数模板名称通常为 `general-kafka`。模板属于管理集群的产品配置，业务集群中的 `RdsKafka` 只保存最终生效的实例配置。不同产品版本的模板可能变化，因此每次测试都应导出当前安装版本，而不是依赖本文表格。 - -如果具备管理集群只读权限，执行： - -```bash -export MGMT_KUBECONFIG=/path/to/management-cluster.kubeconfig - -kubectl --kubeconfig "$MGMT_KUBECONFIG" get paramtemplate -A \ - -l component=kafka - -export TEMPLATE_NS="$(kubectl --kubeconfig "$MGMT_KUBECONFIG" \ - get paramtemplate -A -l component=kafka \ - -o jsonpath='{range .items[?(@.metadata.name=="general-kafka")]}{.metadata.namespace}{"\n"}{end}')" -test -n "$TEMPLATE_NS" -kubectl --kubeconfig "$MGMT_KUBECONFIG" -n "$TEMPLATE_NS" \ - get paramtemplate general-kafka -o yaml \ - >"$OUT/general-kafka-paramtemplate.yaml" -``` - -没有管理集群权限时，在产品控制台选择通用 Kafka 参数模板，并从实例导出最终 `spec.config`。不要根据模板名称推测值。 - -当前标准通用模板应重点核对以下参数；“示例基线”仅用于识别参数，不替代已安装模板： - -| Broker 参数 | 示例基线 | 作用与测试解释 | -| --- | ---: | --- | -| `auto.create.topics.enable` | `false` | 禁止因拼写错误自动建 Topic；测试 Topic 必须显式创建 | -| `default.replication.factor` | `3` | 未显式指定副本数时的默认值；测试仍显式传入 `3` | -| `min.insync.replicas` | `1` | 模板基线；生产可靠性测试应在 Topic 或实例层显式设为 `2` | -| `num.network.threads` | `3` | 处理网络请求的线程数 | -| `num.io.threads` | `8` | 处理请求和磁盘相关工作的线程数 | -| `num.replica.fetchers` | `1` | 每个源 Broker 的副本拉取线程数 | -| `num.recovery.threads.per.data.dir` | `1` | 每个数据目录用于启动恢复和关闭刷新的线程数 | -| `background.threads` | `10` | 后台任务线程池大小 | -| `message.max.bytes` | `1048588` | Broker 接受的最大记录批次大小，不等同于单条消息大小 | -| `compression.type` | `producer` | Broker 保留生产者选择的压缩类型 | -| `log.segment.bytes` | `1073741824` | 日志段大小 | -| `log.retention.hours` | `168` | 基于时间的保留期 | -| `log.roll.hours` | `168` | 日志段最长滚动时间 | -| `delete.topic.enable` | `true` | 允许删除测试 Topic | -| `unclean.leader.election.enable` | `false` | 禁止非 ISR 副本被选为 Leader | - -参数模板的变更应用策略由当前产品版本决定。测试前查询当前版本的参数定义，记录其 `applyStrategy`；如果是 `RestartApply`，把滚动重启时间排除在测量窗口之外。 - -```bash -kubectl --kubeconfig "$MGMT_KUBECONFIG" get paramdefinition kafka-general -o yaml \ - >"$OUT/kafka-general-paramdefinition.yaml" -``` - -## 6. 持久卷容量和存储验证 - -### 6.1 有界合成测试的容量公式 - -对单个 Topic，可先用下式估算每个 Broker 的消息载荷占用： - -```text -每 Broker 载荷字节 ≈ 记录数 × 单条记录字节 × 副本数 ÷ Broker 数 ÷ 实测压缩比 -建议 PVC 下限 = 每 Broker 载荷字节 × 安全系数 + 其他 Topic、索引和运维预留 -``` - -压缩比定义为“未压缩字节 / 落盘字节”。预估阶段没有实测值时取 `1`，不要假设压缩收益。本文对全新、单 Topic、有界合成测试采用 `1.5` 的工程安全系数；该系数不是 Kafka 产品默认值，也不适用于生产容量规划。存在数据倾斜、重分配、多个 Topic 或保留期内持续写入时，应根据实测峰值提高系数，并单独预留迁移空间。 - -在 3 Broker、3 副本、压缩比为 1 的标准矩阵中，每条记录的一个副本平均落在每个 Broker，因此载荷和建议下限如下： - -| 记录数 | 消息大小 | 每 Broker 载荷 | 乘 1.5 后的最低取整值 | -| ---: | ---: | ---: | ---: | -| 50,000,000 | 100 B | 4.66 GiB | 7 GiB | -| 50,000,000 | 500 B | 23.28 GiB | 35 GiB | -| 50,000,000 | 1000 B | 46.57 GiB | 70 GiB | -| 100,000,000 | 100 B | 9.31 GiB | 14 GiB | -| 100,000,000 | 500 B | 46.57 GiB | 70 GiB | - -表中未单独计算 Key、Record Header、批次开销、索引、控制记录、其他 Topic 和重分配临时空间。因此 50 GiB PVC 不足以安全执行 50,000,000 条 1000 B 或 100,000,000 条 500 B 的完整标准点位。无限时长或未知写入量的测试不能用该有界公式，应按写入速率、保留时间和目标磁盘水位计算。 - -独立 KRaft Controller 不保存 Topic 消息载荷，不能套用 Broker 载荷公式。其 PVC 应满足当前产品版本的最小值，并根据 Topic/分区数量变化、元数据日志增长和保留策略实测校准。Controller/Broker 混合角色共用数据卷时，PVC 必须同时满足 Broker 载荷下限和元数据、索引及运维预留。 - -### 6.2 存储检查 - -```bash -kubectl -n "$NS" get pvc -o wide >"$OUT/pvc-before.txt" -kubectl -n "$NS" get pod -o wide >"$OUT/pods-before.txt" - -# 用当前实例实际标签筛选；不要假设 Pod 名称。 -kubectl -n "$NS" get pod \ - -l "strimzi.io/cluster=${CLUSTER},strimzi.io/broker-role=true" \ - -o wide -``` - -确认： - -- PVC 全部为 `Bound`，容量不小于计算值； -- 每个 Broker 的持久卷和节点分布符合测试设计； -- StorageClass、卷类型、文件系统和节点磁盘与目标环境一致； -- 节点不存在持续磁盘压力，且压测前 Kafka 没有待恢复副本。 - -存储延迟是 Kafka 性能的主要约束之一。只报告 StorageClass 名称不足以证明存储等价，应同时保存供应器、介质类型、拓扑和测试期间的 I/O 延迟/利用率证据。 - -## 7. 创建或调整 aKafka 实例 - -通过产品控制台创建专用实例，或修改已有的专用测试实例。控制台字段随产品版本可能变化，最终以业务集群中的 `RdsKafka` 对象和生成的 Strimzi 资源为准。 - -### 7.1 拓扑 - -- 使用 Kafka Raft 元数据模式（KRaft）； -- 至少 3 个 Broker； -- 生产形态使用独立 Controller 时，性能测试也应使用相同拓扑；生产形态使用 Controller/Broker 混合角色时，必须在报告中明确； -- Broker 通过 Pod 反亲和性分散到不同节点和故障域； -- Broker CPU/内存按第 3 节矩阵逐档测试；Controller 资源固定，避免与 Broker 规格同时变化； -- Java 虚拟机（Java Virtual Machine，JVM）堆上限不能占满容器内存，必须为页缓存和容器内的非堆内存留出空间。 - -> Kafka 依赖操作系统页缓存。增大 JVM 堆不等于增大 Kafka 可用缓存；如果堆设置与容器限制过近，可能导致容器内存不足或压缩页缓存空间。 - -控制台提交后，应按当前 `RdsKafka` 自定义资源定义（CustomResourceDefinition，CRD）核对以下字段，不根据控制台显示值推测最终配置： - -| `RdsKafka` 字段 | 标准测试要求 | -| --- | --- | -| `spec.mode` | `KRaft` | -| `spec.replicas` | 3 个 Broker | -| `spec.resources.requests/limits` | 当前矩阵档位；requests 和 limits 均需记录 | -| `spec.storage.class` | 客户选定并已验证的 StorageClass | -| `spec.storage.size` | 不小于第 6 节计算值 | -| `spec.storage.deleteClaim` | 按数据保留和销毁要求选择；不得仅为方便清理而改为 `true` | -| `spec.controller` | 与生产计划一致的角色、数量、资源和存储；独立 Controller 使用不少于 3 个投票节点 | -| `spec.config` | 已导出的 `general-kafka` 基线，加上本轮唯一的参数变更 | -| `spec.kafka.listeners` | 与本轮集群内或外部访问路径一致 | -| `spec.kafkaExporter` | 需要消费组积压时启用，并限制 Topic/消费组正则范围 | - -```bash -kubectl explain rdskafka.spec --recursive \ - >"$OUT/rdskafka-spec-schema.txt" -``` - -### 7.2 固定变量 - -每轮测试只改变一个待评估变量。以下配置必须固定并写入报告： - -- Kafka 与 Operator 版本； -- KRaft 角色布局、Broker/Controller 数量； -- Broker/Controller 的 CPU、内存、JVM 堆、节点选择和反亲和性； -- StorageClass、PVC 容量和存储介质； -- Broker 配置完整快照； -- Topic 分区、副本和 `min.insync.replicas`； -- 客户端镜像版本、资源、节点位置和连接安全； -- 消息大小、记录数、压缩、消息键分布、确认和消费参数。 - -保存实例最终状态： - -```bash -kubectl -n "$NS" get rdskafka "$CLUSTER" -o yaml \ - >"$OUT/rdskafka.yaml" -kubectl -n "$NS" get kafka "$CLUSTER" -o yaml \ - >"$OUT/kafka.yaml" -kubectl -n "$NS" get kafkanodepool -l "strimzi.io/cluster=${CLUSTER}" -o yaml \ - >"$OUT/kafkanodepools.yaml" -kubectl -n "$NS" get pod,pvc,service -o wide \ - >"$OUT/workloads-before.txt" -``` - -任一采集命令失败都应停止并排查权限、资源名称或实例状态，不能用 `|| true` 掩盖错误。 - -### 7.3 监控开关 - -创建实例时启用平台提供的 Kafka 指标采集；需要消费组积压时同时启用 Kafka Exporter，并把 Topic 和消费组正则限制在本次测试范围。开始压测前核对最终资源和 Exporter Pod： - -```bash -kubectl -n "$NS" get rdskafka "$CLUSTER" \ - -o jsonpath='{.spec.kafkaExporter}' \ - >"$OUT/rdskafka-kafka-exporter.json" -kubectl -n "$NS" get kafka "$CLUSTER" \ - -o jsonpath='{.spec.kafka.metricsConfig}' \ - >"$OUT/kafka-metrics-config.json" -kubectl -n "$NS" get pod \ - -l "strimzi.io/cluster=${CLUSTER},strimzi.io/name=${CLUSTER}-kafka-exporter" \ - -o wide -``` - -如果最后一条命令没有找到 Pod，说明 Kafka Exporter 未启用或标签与当前版本不同。先检查 `RdsKafka`、生成的 `Kafka` 资源和实际 Pod 标签，不要把空结果解释为“积压为 0”。 - -## 8. 部署压测客户端 - -### 8.1 复用实例的 Kafka 镜像 - -从 Broker Pod 读取 Operator 实际使用的 Kafka 镜像。这样可保证脚本版本与服务端版本一致，也不需要在外部文档中固化镜像仓库地址。 - -```bash -export KAFKA_IMAGE="$(kubectl -n "$NS" get pod \ - -l "strimzi.io/cluster=${CLUSTER},strimzi.io/broker-role=true" \ - -o jsonpath='{.items[0].spec.containers[?(@.name=="kafka")].image}')" - -test -n "$KAFKA_IMAGE" -export BOOTSTRAP="${CLUSTER}-kafka-bootstrap.${NS}.svc:9092" -``` - -上面的 Service 名称符合 Strimzi 内部 Bootstrap Service 规则，但仍应查询当前实例验证监听器、端口和认证方式： - -```bash -kubectl -n "$NS" get service -l "strimzi.io/cluster=${CLUSTER}" -o wide -kubectl -n "$NS" get kafka "$CLUSTER" \ - -o jsonpath='{.spec.kafka.listeners}' >"$OUT/listeners.json" -``` - -以下示例使用隔离命名空间内的明文内部监听器。启用 TLS/SASL 时，应把完整客户端属性放入受控 Secret，并以只读卷挂载；不要通过命令行参数传递密码。 - -### 8.2 创建客户端属性 - -```bash -kubectl -n "$NS" create configmap akafka-perf-client-config \ - --from-literal=admin.properties='' \ - --from-literal=producer.properties='acks=all -enable.idempotence=true -batch.size=50000 -linger.ms=5 -compression.type=none -buffer.memory=134217728' \ - --from-literal=consumer.properties='auto.offset.reset=earliest -fetch.min.bytes=1 -fetch.max.wait.ms=500 -max.partition.fetch.bytes=1048576 -fetch.max.bytes=52428800' \ - --dry-run=client -o yaml | kubectl apply -f - -``` - -如果使用认证，在本地受控目录中准备包含性能参数和连接参数的完整 `producer.properties`、`consumer.properties`，以 Secret 替代上面的 ConfigMap： - -```bash -kubectl -n "$NS" create secret generic akafka-perf-client-config \ - --from-file=admin.properties=/secure/path/admin.properties \ - --from-file=producer.properties=/secure/path/producer.properties \ - --from-file=consumer.properties=/secure/path/consumer.properties \ - --dry-run=client -o yaml | kubectl apply -f - -``` - -同时把 8.3 节两个 Pod 的卷源从 `configMap` 改为： - -```yaml -secret: - secretName: akafka-perf-client-config -``` - -`admin.properties` 只保存管理脚本所需的连接安全属性；生产者和消费者文件另外保存各自性能参数。属性中的 `security.protocol`、SASL 或 SSL 设置必须与实例监听器一致。报告和证据目录只保存脱敏副本，不得导出 Secret、密码、令牌、私钥或未脱敏的 Java 身份验证和授权服务（Java Authentication and Authorization Service，JAAS）配置。 - -### 8.3 创建客户端 Pod - -```bash -kubectl -n "$NS" apply -f - <"$OUT/producer-perf-help.txt" 2>&1 - -kubectl -n "$NS" exec akafka-consumer-perf -- \ - /opt/kafka/bin/kafka-consumer-perf-test.sh --help \ - >"$OUT/consumer-perf-help.txt" 2>&1 -``` - -如果镜像中的 Kafka 安装目录不是 `/opt/kafka`，先在镜像说明或 Broker 容器环境中确认实际路径，再替换命令。不要下载另一个未知版本的脚本混用。 - -## 9. 创建和验证测试 Topic - -```bash -kubectl -n "$NS" exec akafka-producer-perf -- \ - /opt/kafka/bin/kafka-topics.sh \ - --bootstrap-server "$BOOTSTRAP" \ - --command-config /opt/perf-config/admin.properties \ - --create --if-not-exists \ - --topic "$TOPIC" \ - --partitions 30 \ - --replication-factor 3 \ - --config min.insync.replicas=2 - -kubectl -n "$NS" exec akafka-producer-perf -- \ - /opt/kafka/bin/kafka-topics.sh \ - --bootstrap-server "$BOOTSTRAP" \ - --command-config /opt/perf-config/admin.properties \ - --describe --topic "$TOPIC" \ - | tee "$OUT/topic-before.txt" - -kubectl -n "$NS" exec akafka-producer-perf -- \ - /opt/kafka/bin/kafka-configs.sh \ - --bootstrap-server "$BOOTSTRAP" \ - --command-config /opt/perf-config/admin.properties \ - --entity-type topics --entity-name "$TOPIC" --describe \ - | tee "$OUT/topic-config.txt" -``` - -开始前逐项确认： - -- `PartitionCount=30`、`ReplicationFactor=3`； -- 每个分区有 3 个副本，ISR 数量为 3； -- Leader 在 Broker 间没有明显失衡； -- 动态 Topic 配置中 `min.insync.replicas=2`； -- Topic 是本轮新建或已按测试设计清理，消费组名称没有复用。 - -## 10. 执行单个测试点 - -### 10.1 生产可靠性基线 - -先记录参数，再执行。示例为 1,000,000 条、每条 500 B 的功能验证；正式标准点位替换 `--num-records`，并确保 PVC 容量足够。 - -```bash -export NUM_RECORDS=1000000 -export RECORD_SIZE=500 - -kubectl -n "$NS" exec akafka-producer-perf -- sh -c \ - "cat /opt/perf-config/producer.properties" \ - >"$OUT/producer.properties" - -kubectl -n "$NS" exec akafka-producer-perf -- \ - /opt/kafka/bin/kafka-producer-perf-test.sh \ - --bootstrap-server "$BOOTSTRAP" \ - --topic "$TOPIC" \ - --num-records "$NUM_RECORDS" \ - --record-size "$RECORD_SIZE" \ - --throughput -1 \ - --warmup-records 100000 \ - --reporting-interval 5000 \ - --command-config /opt/perf-config/producer.properties \ - --print-metrics \ - | tee "$OUT/producer-${RECORD_SIZE}B.txt" -``` - -`--throughput -1` 表示客户端不主动限速，用于寻找该客户端并发下的上限。它不保证 Broker 已饱和；必须结合客户端和 Broker 指标判断。正式对比中应固定 `--warmup-records`，并只使用脚本的稳态汇总结果，不把预热阶段混入结果。 - -### 10.2 Leader 确认上限 - -保持其他属性不变，只覆盖确认和幂等配置： - -```properties -acks=1 -enable.idempotence=false -batch.size=50000 -linger.ms=5 -compression.type=none -buffer.memory=134217728 -``` - -Kafka 4.2 可通过 `--command-property` 覆盖配置文件中的这两个值，避免无意改变其他参数： - -```bash -kubectl -n "$NS" exec akafka-producer-perf -- \ - /opt/kafka/bin/kafka-producer-perf-test.sh \ - --bootstrap-server "$BOOTSTRAP" \ - --topic "$TOPIC" \ - --num-records "$NUM_RECORDS" \ - --record-size "$RECORD_SIZE" \ - --throughput -1 \ - --warmup-records 100000 \ - --reporting-interval 5000 \ - --command-config /opt/perf-config/producer.properties \ - --command-property acks=1 \ - --command-property enable.idempotence=false \ - --print-metrics \ - | tee "$OUT/producer-leader-ack-${RECORD_SIZE}B.txt" -``` - -结果必须标记为 `leader-ack`，且不得与 `acks=all` 的容量直接比较。 - -### 10.3 消费者单测 - -消费者单测前先准备不小于 `NUM_RECORDS` 的存量数据，然后使用新的消费组： - -```bash -export NUM_RECORDS=1000000 -export RECORD_SIZE=500 -export GROUP="perf-consumer-$(date -u +%Y%m%dT%H%M%SZ)" - -kubectl -n "$NS" exec akafka-consumer-perf -- sh -c \ - "cat /opt/perf-config/consumer.properties" \ - >"$OUT/consumer.properties" - -kubectl -n "$NS" exec akafka-consumer-perf -- \ - /opt/kafka/bin/kafka-consumer-perf-test.sh \ - --bootstrap-server "$BOOTSTRAP" \ - --topic "$TOPIC" \ - --group "$GROUP" \ - --num-records "$NUM_RECORDS" \ - --fetch-size 200000 \ - --timeout 60000 \ - --show-detailed-stats \ - --reporting-interval 5000 \ - --command-config /opt/perf-config/consumer.properties \ - --print-metrics \ - | tee "$OUT/consumer-${RECORD_SIZE}B.txt" -``` - -Kafka 4.2 中 `--messages` 已弃用，应使用 `--num-records`。`--fetch-size` 限制每次从单个分区获取的数据量；客户端属性中的 `max.partition.fetch.bytes` 也参与限制。必须保存 `--print-metrics` 输出中的 `records-consumed-total`，确认实际消费量达到目标。超时、Topic 数据不足或权限错误都可能使脚本提前结束，不能只看最后一行吞吐量。 - -### 10.4 生产和消费并发 - -1. 使用新消费组启动消费者，并确认其已加入组； -2. 启动生产者； -3. 在整个稳态窗口采集指标； -4. 生产者停止后继续运行消费者，直到积压归零或达到预定义超时； -5. 保存最终消费位点和日志末端偏移量（log end offset，LEO）。 - -并发测试不能直接把单次消费者命令放在前台后再启动生产者。应在两个终端运行，或通过 Job/测试编排器同时启动并分别保存标准输出。任何后台执行方式都必须保存进程退出码；仅有日志文件不足以证明命令成功。 - -### 10.5 Kafka 4.2 的 20 消费者执行示例 - -以下示例在一个客户端 Pod 中启动 20 个独立消费者进程，并检查每个 `kubectl exec` 的退出码。正式测试前必须根据 12.1 节确认该 Pod 的 CPU、内存和网络不会成为瓶颈；资源不足时，复制 8.3 节的消费者 Pod 并把进程分散到多个压测节点。 - -```bash -export CONSUMER_COUNT=20 -export CONSUMER_POD=akafka-consumer-perf -export GROUP="perf-consumer-20-$(date -u +%Y%m%dT%H%M%SZ)" -test "$NUM_RECORDS" -ge "$CONSUMER_COUNT" - -q=$((NUM_RECORDS / CONSUMER_COUNT)) -r=$((NUM_RECORDS % CONSUMER_COUNT)) -pids=() - -for ((i = 1; i <= CONSUMER_COUNT; i++)); do - process_records="$q" - if ((i <= r)); then - process_records=$((q + 1)) - fi - - kubectl -n "$NS" exec "$CONSUMER_POD" -- \ - /opt/kafka/bin/kafka-consumer-perf-test.sh \ - --bootstrap-server "$BOOTSTRAP" \ - --topic "$TOPIC" \ - --group "$GROUP" \ - --num-records "$process_records" \ - --fetch-size 200000 \ - --timeout 60000 \ - --reporting-interval 5000 \ - --command-config /opt/perf-config/consumer.properties \ - --command-property "client.id=perf-consumer-${i}" \ - --print-metrics \ - >"$OUT/consumer-${i}-${RECORD_SIZE}B.txt" 2>&1 & - pids+=("$!") -done - -failed=0 -for pid in "${pids[@]}"; do - wait "$pid" || failed=1 -done -test "$failed" -eq 0 -``` - -把 20 个文件中的实际消费记录数相加，并核对总数等于 `NUM_RECORDS`。聚合吞吐量按公共测量窗口中的总记录数/总字节数计算；各进程起止时间不一致时，不能直接相加各自打印的平均速率。任一进程超时、退出非零或记录数不符，都应使该测试点无效。 - -## 11. 客户端参数说明 - -### 11.1 生产者 - -| 参数 | 标准值 | 机制和边界 | -| --- | ---: | --- | -| `--num-records` | 见矩阵 | 实际发送条数；总载荷约为该值乘 `--record-size` | -| `--record-size` | 100/500/1000 | 合成记录 Value 的字节数；只可与消息载荷（payload）文件模式二选一 | -| `--throughput` | `-1` | 不限速上探；受控负载测试时填写目标 records/s | -| `--warmup-records` | 固定值 | 排除连接建立、元数据和 JVM 预热影响；不计入稳态结果 | -| `acks` | `all` 或 `1` | 确认强度；改变后可靠性和性能口径同时变化 | -| `enable.idempotence` | `true` | 防止重试导致重复；要求兼容的 `acks`、重试和在途请求设置 | -| `batch.size` | `50000` | 每分区批次的目标上限；过大可能增加内存和等待时间 | -| `linger.ms` | `5` | 等待合批的最长时间；Kafka 4.x 默认值与早期版本不同，测试应显式设置 | -| `compression.type` | `none` | 标准矩阵关闭压缩；业务矩阵使用真实算法并记录 CPU 与实测压缩比 | -| `buffer.memory` | `134217728` | 生产者可用于缓冲待发送记录的总内存；不是容器内存需求的全部 | - -每个分区有独立批次，增加生产者数量、分区数、批次或 `linger.ms` 可能提高合批效率，但也可能增加内存和延迟。只有在报告中保持其余变量不变时，参数前后结果才可归因。 - -### 11.2 消费者 - -| 参数 | 标准值 | 机制和边界 | -| --- | ---: | --- | -| `--group` | 每轮唯一 | 同组消费者共同分配分区；复用组可能从旧位点开始 | -| `--num-records` | 与存量数据匹配 | 计划读取的总记录数；必须用客户端指标核对实际值 | -| `--fetch-size` | `200000` | 脚本传入的单分区抓取大小 | -| `--timeout` | `60000` | 两次返回记录之间允许的最长时间，不是整个测试的总时长；超时会使工具提前退出并打印警告 | -| `fetch.min.bytes` | `1` | Broker 返回 Fetch 的最小数据量；增大可提高批量效率，但可能增加等待延迟 | -| `fetch.max.wait.ms` | `500` | 未达到 `fetch.min.bytes` 时的最长等待时间 | -| `max.partition.fetch.bytes` | `1048576` | 每分区每次 Fetch 的返回上限；要能容纳 Broker 允许的最大记录批次 | -| `fetch.max.bytes` | `52428800` | 单次 Fetch 的总体返回上限，仍受服务端和分区级限制 | - -同一消费组的有效并行度不超过可分配分区数。Kafka 4.2 的脚本每次调用运行一个消费者；需要 2、4、8 个消费者时，创建相同配置的独立 Pod 或进程，并使用同一个消费组。消费者数超过分区数时，多出的消费者不会获得分区。 - -矩阵中的 `NUM_RECORDS` 是全部进程的聚合目标，不是每个进程的目标。并发数为 `C` 时，令 `q=NUM_RECORDS/C`、`r=NUM_RECORDS%C`：前 `r` 个进程使用 `q+1`，其余进程使用 `q`，保证各进程目标之和等于 `NUM_RECORDS`。多生产者测试也使用相同分配方法。若每个进程都发送或消费完整的 `NUM_RECORDS`，总数据量已经改变，该结果不能与单进程点直接比较。 - -## 12. 寻找饱和点和可用容量 - -### 12.1 先排除客户端上限 - -对生产者和消费者分别使用 1、2、4、8 个独立客户端进程上探；标准消费者点继续增加到 12、16、20，或在此之前已达到分区并行度/资源上限时停止。客户端应尽量分布在不同压测节点，每个进程使用相同属性，并聚合所有进程的吞吐量。每增加一次并发，至少核对： - -- 客户端 Pod CPU、内存、CPU 节流、网络和重启次数； -- Broker CPU、内存、网络、磁盘延迟/利用率； -- 请求错误、超时、重试和生产确认延迟； -- `UnderReplicatedPartitions`、`UnderMinIsrPartitionCount` 和消费组积压； -- 各 Broker 的分区 Leader、日志字节和流量是否失衡。 - -如果增加客户端资源或进程后吞吐量仍显著增长，前一个结果是客户端上限，不能报告为 Kafka 上限。如果单个压测节点网卡已饱和，应增加客户端节点，而不是继续增加同节点进程。 - -### 12.2 两阶段负载阶梯 - -1. **无节流上探**：使用 `--throughput -1`，逐步增加客户端进程，得到可达吞吐量范围。 -2. **受控负载验证**：以无节流结果为基准，依次施加约 25%、50%、75%、90%、100%、110% 的目标吞吐量。所有生产者的 `--throughput` 之和等于该点目标值。 -3. 每个点先预热，再保持至少 10 分钟稳态；结束后等待积压归零并确认副本健康。 -4. 90% 附近加密测试点；对最后一个稳定点和第一个不稳定点各重复至少 3 次。 -5. 打乱重复测试顺序或重新运行基线点，以识别硬件温度、缓存、邻居负载和时间漂移。 - -百分比只是寻找区间的起始策略，不是产品阈值。如果第一轮已违反健康条件，应降低负载重新建立区间。 - -### 12.3 预先定义判定规则 - -测试开始前由客户根据业务服务等级目标（Service Level Objective，SLO）确定以下阈值： - -- 生产确认 p95/p99 延迟上限； -- 错误、超时和重试率上限； -- 允许的最大消费积压及清空时间； -- Broker、客户端、节点和 PVC 的资源水位； -- 允许的 `UnderReplicatedPartitions` 和 `UnderMinIsrPartitionCount`，通常应始终为 0； -- 稳态窗口和重复次数。 - -如果客户尚无阈值，可把以下规则作为本次测试的工程判定方法，但必须在报告中标记为“测试规则”，不能称为 Kafka 官方阈值： - -- 增加一档负载或并发后，聚合吞吐量增幅连续两档小于 5%；且 -- p99 延迟、错误/超时、积压或资源水位持续恶化；或 -- 出现 `UnderReplicatedPartitions>0`、`UnderMinIsrPartitionCount>0`、离线日志目录、Pod 重启或磁盘压力。 - -满足上述组合条件的第一个点记为**饱和点**。饱和前最后一个满足全部 SLO、重复结果稳定且无健康异常的点记为**可用容量**。不要用瞬时最大值代替可用容量。 - -重复结果的离散程度也要报告。默认可用中位数作为中心值，并同时给出最小值、最大值和变异系数；如果同一测试点吞吐量变异系数超过 5%，应调查环境漂移并增加重复次数。5% 同样是本文的测试规则，不是 Kafka 产品保证。 - -## 13. 实例参数调优 - -先用模板基线完成一轮，再根据监控证据一次只调整一个参数。以下建议是诊断方向，不是固定最优值。 - -| 参数/资源 | 何时考虑调整 | 验证方法 | 风险和停止条件 | -| --- | --- | --- | --- | -| Broker CPU | 请求处理线程忙、CPU 持续接近约束、客户端未饱和 | 增加 CPU 后重复相同负载 | 若吞吐不增而磁盘/网络已满，继续加 CPU 无益 | -| Broker 内存/JVM 堆 | 容器内存压力、频繁垃圾回收（garbage collection，GC），或页缓存不足 | 同时观察堆、GC、常驻内存集（resident set size，RSS）、工作集和磁盘读取 | 堆过大挤压页缓存；堆与容器限制过近可能发生内存不足（out of memory，OOM） | -| `num.network.threads` | `NetworkProcessorAvgIdlePercent` 持续偏低且 CPU 尚有余量 | 小步增加并重复同一点 | 增加线程会增加调度和内存开销；不能修复网卡饱和 | -| `num.io.threads` | `RequestHandlerAvgIdlePercent` 持续偏低，且磁盘/CPU 尚有余量 | 小步增加并比较请求等待和吞吐 | 至少覆盖数据卷数量；过多线程可能增加上下文切换和 I/O 竞争 | -| `num.replica.fetchers` | 副本追赶慢、复制流量未用满资源 | 在故障恢复或高复制负载下验证 | 增加后会提高网络、磁盘和 CPU 消耗 | -| `num.recovery.threads.per.data.dir` | 启动或恢复过慢 | 在受控恢复测试中验证 | 恢复速度提高会加重前台 I/O 干扰 | -| Topic 分区数 | 单分区吞吐成为限制，且有足够消费者并发 | 新建不同分区数的 Topic 对比 | 增加分区改变 Key 顺序范围、元数据和副本开销；不能随意减少 | -| Broker 数量 | 单机资源已接近约束，需要横向扩展 | 扩容并等待副本重分配完成后重测 | 只增加 Broker 不会自动均衡已有分区数据；重分配期间结果无效 | -| `message.max.bytes` | 业务最大记录批次超过默认限制 | 同步核对 Topic、生产者和消费者抓取上限 | 增大后提高内存和网络突发，配置不一致会产生发送或消费失败 | -| 保留与日志段参数 | 磁盘占用、删除粒度或恢复时间不满足需求 | 长时间测试保留和删除行为 | 不应仅为短时跑分改变生产保留策略 | - -Apache Kafka 的线程空闲指标越接近 0 表示越忙。`NetworkProcessorAvgIdlePercent` 或 `RequestHandlerAvgIdlePercent` 长时间低于约 0.3 时，应结合 CPU、网络和存储判断是否出现处理能力不足；该值是诊断信号，不是单独的扩容触发器。 - -每次变更后： - -1. 保存变更前后完整配置差异； -2. 按当前产品版本的应用策略完成滚动重启或动态更新； -3. 等待 Kafka 和 NodePool Ready、ISR 完整、KRaft 控制器仲裁状态稳定； -4. 以相同 Topic 数据状态、客户端并发和负载重新执行至少 3 次； -5. 收益不稳定或健康指标恶化时回到前一个已验证配置。 - -## 14. 监控和证据采集 - -### 14.1 Kafka 健康检查 - -在测前、稳态窗口和测后执行。Broker Pod 名称通过标签发现： - -```bash -export BROKER_POD="$(kubectl -n "$NS" get pod \ - -l "strimzi.io/cluster=${CLUSTER},strimzi.io/broker-role=true" \ - -o jsonpath='{.items[0].metadata.name}')" - -kubectl -n "$NS" exec akafka-producer-perf -- \ - /opt/kafka/bin/kafka-metadata-quorum.sh \ - --bootstrap-server "$BOOTSTRAP" \ - --command-config /opt/perf-config/admin.properties \ - describe --status \ - | tee "$OUT/metadata-quorum-after.txt" - -kubectl -n "$NS" exec akafka-producer-perf -- \ - /opt/kafka/bin/kafka-topics.sh \ - --bootstrap-server "$BOOTSTRAP" \ - --command-config /opt/perf-config/admin.properties \ - --describe --under-replicated-partitions \ - | tee "$OUT/under-replicated-after.txt" - -kubectl -n "$NS" exec akafka-producer-perf -- \ - /opt/kafka/bin/kafka-consumer-groups.sh \ - --bootstrap-server "$BOOTSTRAP" \ - --command-config /opt/perf-config/admin.properties \ - --describe --group "$GROUP" \ - | tee "$OUT/consumer-group-after.txt" - -kubectl -n "$NS" exec akafka-producer-perf -- \ - /opt/kafka/bin/kafka-log-dirs.sh \ - --bootstrap-server "$BOOTSTRAP" \ - --command-config /opt/perf-config/admin.properties \ - --describe --topic-list "$TOPIC" \ - >"$OUT/log-dirs-after.txt" -``` - -`--under-replicated-partitions` 在健康状态下可以没有输出。应同时保存命令退出码或由编排器记录成功状态，避免把连接失败误判为“没有异常分区”。 - -`kafka-log-dirs.sh` 会在 JSON 数据前输出状态文本，因此上例按原始文本保存；需要程序化解析时，应先验证并移除非 JSON 前缀，不要直接把整个文件交给 JSON 解析器。 - -### 14.2 必采 Kafka 指标 - -以下 Java 管理扩展（Java Management Extensions，JMX）管理型 Bean（MBean）或其 Prometheus 映射必须覆盖完整稳态窗口： - -| 类别 | 指标或 JMX MBean | 用途 | -| --- | --- | --- | -| 写入/读取 | `BytesInPerSec`、`BytesOutPerSec`、`MessagesInPerSec` | 与客户端吞吐交叉验证 | -| 副本健康 | `UnderReplicatedPartitions`、`UnderMinIsrPartitionCount` | 前者表示 ISR 数量小于配置副本数，后者表示 ISR 数量小于 `min.insync.replicas` | -| 日志目录 | `OfflineLogDirectoryCount` | 识别不可用存储目录 | -| 网络线程 | `NetworkProcessorAvgIdlePercent` | 判断网络处理线程繁忙程度 | -| 请求线程 | `RequestHandlerAvgIdlePercent` | 判断请求处理线程繁忙程度 | -| 请求延迟 | Produce/Fetch 请求的 `TotalTimeMs`、`RequestQueueTimeMs`、`LocalTimeMs`、`RemoteTimeMs`、`ResponseQueueTimeMs` 和 `ResponseSendTimeMs` | 定位等待发生在哪个阶段 | -| 消费组 | 当前位点、LEO、每分区和总积压（lag） | 判断消费者能否跟上生产流量 | - -JMX Exporter 的 Prometheus 指标名称由实例的指标映射规则决定。不要假设所有环境使用同一名称。先从 Broker 暴露的 `/metrics` 端点保存原始指标并确认名称： - -```bash -kubectl -n "$NS" exec "$BROKER_POD" -- \ - curl -fsS http://127.0.0.1:9404/metrics \ - >"$OUT/broker-jmx-metrics.txt" - -grep -m 100 -E 'bytesin|bytesout|messagesin|underreplicated|underminisr|avgidle' \ - "$OUT/broker-jmx-metrics.txt" -``` - -如果实例启用了 Kafka Exporter，先发现 Pod，再保存其原始指标： - -```bash -export EXPORTER_PODS="$(kubectl -n "$NS" get pod \ - -l "strimzi.io/cluster=${CLUSTER},strimzi.io/name=${CLUSTER}-kafka-exporter" \ - -o jsonpath='{.items[*].metadata.name}')" - -if test -n "$EXPORTER_PODS"; then - export EXPORTER_POD="${EXPORTER_PODS%% *}" - kubectl -n "$NS" exec "$EXPORTER_POD" -- \ - curl -fsS http://127.0.0.1:9404/metrics \ - >"$OUT/kafka-exporter-metrics.txt" -else - printf '%s\n' 'Kafka Exporter Pod not found' \ - >"$OUT/kafka-exporter-not-found.txt" -fi -``` - -原始端点有数据只证明 Exporter 正常暴露指标，不证明平台已经抓取时序数据。正式测试前，应在平台监控或 Prometheus 中查询最近 15 分钟的 Broker 指标，确认每个 Broker 都有连续样本。如果查询为空或存在采集空洞，应停止测试，按当前平台监控配置补齐 ServiceMonitor/PodMonitor 和指标映射；其选择标签由当前 Prometheus 配置决定，不能照抄其他集群的标签。 - -### 14.3 Kubernetes 和基础设施指标 - -至少采集以下时序数据，而不是只保存测后瞬时值： - -- Broker、Controller 和客户端 Pod：CPU 使用、CPU 节流、内存工作集、网络收发、重启和 OOM； -- 节点：CPU、内存、网络带宽/丢包、磁盘吞吐、每秒输入/输出操作数（input/output operations per second，IOPS）、平均和高分位延迟、利用率、队列深度； -- PVC：已用字节、容量、使用率和增长速率； -- Kubernetes 事件、Pod 调度节点和资源 requests/limits。 - -`kubectl top` 只提供近实时快照，可用于快速检查，但不能替代 Prometheus 范围查询： - -```bash -kubectl -n "$NS" top pod --containers \ - | tee "$OUT/top-pods-after.txt" -kubectl -n "$NS" get events --sort-by=.lastTimestamp \ - >"$OUT/events.txt" -kubectl -n "$NS" get pod -o wide \ - >"$OUT/pods-after.txt" -kubectl -n "$NS" get pvc -o wide \ - >"$OUT/pvc-after.txt" -``` - -以下 PromQL 仅表示计算意图。先在当前 Prometheus 中检查实际指标名和标签，再替换 `$NS`、`$TOPIC`、`$GROUP` 和集群标签；保存查询表达式、时间范围、步长和原始响应。 - -```text -sum(rate(kafka_server_brokertopicmetrics_messagesin_total{namespace="$NS",topic="$TOPIC"}[5m])) - -sum(rate(kafka_server_brokertopicmetrics_bytesin_total{namespace="$NS",topic="$TOPIC"}[5m])) - -sum(rate(kafka_server_brokertopicmetrics_bytesout_total{namespace="$NS",topic="$TOPIC"}[5m])) - -sum(kafka_server_replicamanager_underreplicatedpartitions{namespace="$NS"}) - -sum(kafka_server_replicamanager_underminisrpartitioncount{namespace="$NS"}) - -min(kafka_network_socketserver_networkprocessoravgidle_percent{namespace="$NS"}) - -min(kafka_server_kafkarequesthandlerpool_requesthandleravgidle_percent{namespace="$NS"}) - -sum(kafka_consumergroup_lag{namespace="$NS",consumergroup="$GROUP",topic="$TOPIC"}) - -sum by (pod) ( - rate(container_cpu_usage_seconds_total{namespace="$NS",container!="",image!=""}[5m]) -) - -max by (pod) ( - container_memory_working_set_bytes{namespace="$NS",container!="",image!=""} -) - -sum by (pod) ( - rate(container_network_receive_bytes_total{namespace="$NS"}[5m]) -) - -sum by (pod) ( - rate(container_network_transmit_bytes_total{namespace="$NS"}[5m]) -) - -kubelet_volume_stats_used_bytes{namespace="$NS"} -/ -kubelet_volume_stats_capacity_bytes{namespace="$NS"} -``` - -CPU 节流和节点磁盘指标在不同容器运行时、cgroup 版本和监控栈中的名称可能不同。如果平台没有对应时序指标，应在测试前补齐采集或从节点运行时、cgroup 和存储系统导出等价证据；不能把“查询为空”记为 0。 - -### 14.4 Redis 或其他关联组件 - -如果业务链路还包含 Redis，应为 Redis 使用相同的 `RUN_ID`、UTC 时间窗口和节点事件记录，并附独立的 Redis 性能报告。Kafka 和 Redis 的吞吐量、延迟及饱和点属于不同指标，不能合并，也不能仅凭时间相关性推断一个组件导致另一个组件变慢。Kafka 报告只保留关联报告链接、版本、测试窗口和经验证的依赖关系。 - -## 15. 结果汇总和有效性检查 - -### 15.1 每个测试点的记录字段 - -至少记录： - -```text -run_id,start_utc,end_utc,measurement_seconds,test_mode,reliability_profile, -kafka_version,operator_version,kraft_roles,broker_count,controller_count, -broker_cpu,broker_memory,jvm_xms,jvm_xmx,storage_class,pvc_size, -topic,partitions,replication_factor,min_insync_replicas, -record_count_target,producer_records_sent,consumer_records_consumed, -record_size,compression,compression_ratio,key_distribution, -producer_processes,consumer_processes,acks,idempotence,batch_size,linger_ms, -throughput_target,fetch_min_bytes,fetch_max_wait_ms,max_partition_fetch_bytes, -producer_records_s,producer_mib_s,producer_avg_ms,producer_p50_ms, -producer_p95_ms,producer_p99_ms,producer_p999_ms, -consumer_records_s,consumer_mib_s,max_lag,final_lag,error_count,timeout_count, -broker_cpu_cores_max,broker_memory_bytes_max,broker_network_bps_max,pvc_used_bytes_max, -network_idle_min,request_idle_min,under_replicated_max,under_min_isr_max, -client_cpu_cores_max,client_cpu_throttling_ratio_max,result_valid,invalid_reason -``` - -Kafka 脚本输出使用 `MB/sec` 列名。报告应保留原始列名；需要统一为 MiB/s 时，另行使用 `实际载荷字节 ÷ 秒数 ÷ 2^20` 计算并记录公式，不要只修改列名。 - -资源峰值字段默认记录“单个 Pod 或 PVC 的最大值”；如果使用求和、平均或其他聚合方式，应在字段说明中记录聚合函数、单位、时间窗口和采样步长。 - -### 15.2 有效结果的必要条件 - -仅当以下条件全部满足时，结果才标记为 `valid`： - -- 命令退出码为 0，生产和消费记录数达到目标； -- 预热和稳态窗口按计划完成，时钟范围明确； -- Kafka、Operator、配置、Topic、客户端和基础设施快照完整； -- 未发生 Pod 重启、OOM、节点压力、卷异常或计划外维护； -- `UnderReplicatedPartitions=0`、`UnderMinIsrPartitionCount=0`、`OfflineLogDirectoryCount=0`； -- 生产可靠性基线没有违反 `acks=all`、幂等和最小 ISR 要求； -- 客户端不是已确认的瓶颈，或结果已明确标记为“客户端上限”； -- 监控覆盖完整稳态窗口，并且没有采集空洞； -- 重复测试的差异已经解释，或结论保留不确定性。 - -以下任一情况应标记为 `invalid` 并重测： - -- Topic 已有未知数据、消费组复用或实际记录数不足； -- 测试期间发生滚动重启、重分配、Leader 频繁切换或副本追赶； -- PVC 接近满、存储或节点出现压力； -- 客户端被 CPU 限流、网络饱和或异常退出； -- 不同测试点改变了两个以上关键变量； -- 仅有平均值，没有原始输出、分位延迟或时序监控。 - -### 15.3 报告结论写法 - -结论必须区分： - -- **事实**：原始输出、配置快照和监控直接显示的结果； -- **推断**：由多个指标共同支持的瓶颈判断，并列出反证； -- **建议**：下一步调参、扩容或重测方案； -- **待验证事项**：缺少存储、网络、业务消息模型或长稳测试时尚不能确认的内容。 - -推荐同时报告： - -1. 最后一个满足全部 SLO 的可用容量； -2. 第一个不稳定的饱和点及触发条件； -3. 3 次以上重复结果的中位数、范围和变异系数； -4. 与基线相比的相对变化，并保持可靠性和环境配置相同； -5. 适用边界，特别是消息模型、存储、网络、认证、保留策略和测试持续时间。 - -## 16. 清理 - -先归档并校验 `$OUT`，再删除本轮独立资源。不要使用模糊标签或通配符删除共享资源。 - -```bash -# 仅删除本文创建的专用 Topic；命令执行前再次核对名称。 -kubectl -n "$NS" exec akafka-producer-perf -- \ - /opt/kafka/bin/kafka-topics.sh \ - --bootstrap-server "$BOOTSTRAP" \ - --command-config /opt/perf-config/admin.properties \ - --delete --topic "$TOPIC" - -kubectl -n "$NS" delete pod \ - akafka-producer-perf akafka-consumer-perf --ignore-not-found -kubectl -n "$NS" delete configmap \ - akafka-perf-client-config --ignore-not-found -kubectl -n "$NS" delete secret \ - akafka-perf-client-config --ignore-not-found -``` - -如果整个命名空间都是本轮创建且已确认无共享资源，可以通过变更流程删除命名空间。删除实例或命名空间前，必须核对 PVC 的 `deleteClaim` 和 StorageClass 回收策略，确认数据保留或销毁符合要求。 - -## 17. 参考资料 - -- [Apache Kafka 4.2 Broker 配置](https://kafka.apache.org/42/configuration/broker-configs/) -- [Apache Kafka 4.2 Producer 配置](https://kafka.apache.org/42/configuration/producer-configs/) -- [Apache Kafka 4.2 Consumer 配置](https://kafka.apache.org/42/configuration/consumer-configs/) -- [Apache Kafka 4.2 监控](https://kafka.apache.org/42/operations/monitoring/) -- [Apache Kafka 4.2 硬件与操作系统](https://kafka.apache.org/42/operations/hardware-and-os/) -- [Strimzi 0.48 部署和运行指南](https://strimzi.io/docs/operators/0.48.0/deploying.html) -- [Prometheus Operator API 参考](https://prometheus-operator.dev/docs/api-reference/api/) From a4843d7834bc99776d0d8a78052a38637a163d44 Mon Sep 17 00:00:00 2001 From: Richard Feng Date: Wed, 29 Jul 2026 23:53:46 -0400 Subject: [PATCH 7/8] docs: align ZooKeeper guide with RdsKafka API --- .../kafka/Kafka_Performance_Testing_Guide.md | 51 ++++++++++++++++--- 1 file changed, 43 insertions(+), 8 deletions(-) diff --git a/docs/en/solutions/ecosystem/kafka/Kafka_Performance_Testing_Guide.md b/docs/en/solutions/ecosystem/kafka/Kafka_Performance_Testing_Guide.md index c2f0bd27c..adf348153 100644 --- a/docs/en/solutions/ecosystem/kafka/Kafka_Performance_Testing_Guide.md +++ b/docs/en/solutions/ecosystem/kafka/Kafka_Performance_Testing_Guide.md @@ -213,9 +213,12 @@ kubectl --kubeconfig "$MGMT_KUBECONFIG" get paramdefinition kafka-general -o yam ### 5.1 Additional snapshot for a ZooKeeper-based instance -The legacy generated `Kafka` resource stores the broker and ZooKeeper specifications separately. Save both before testing: +The product-facing `RdsKafka` API and the generated Strimzi `Kafka` resource are different layers. Save both before testing. The first snapshot records the requested mode and ZooKeeper sizing; the second records what the operator generated: ```bash +kubectl -n "$NS" get rdskafka "$CLUSTER" \ + -o jsonpath='{.spec.mode}{"\n"}{.spec.version}{"\n"}{.spec.zookeeper}{"\n"}' \ + >"$OUT/rdskafka-zookeeper-requested.txt" kubectl -n "$NS" get kafka "$CLUSTER" \ -o jsonpath='{.spec.kafka.version}{"\n"}' \ >"$OUT/kafka-version.txt" @@ -229,7 +232,7 @@ kubectl -n "$NS" get kafka "$CLUSTER" \ Record `log.message.format.version` and `inter.broker.protocol.version` when present. A value that does not match the broker major and minor version can indicate an incomplete upgrade. Do not change either value during a performance comparison. -ZooKeeper settings under `spec.zookeeper.config` are version-specific, and the operator prevents changes to settings that it owns, including server addresses, data directories, client ports, quorum authentication, and several TLS properties. Query the installed custom resource definition before changing a ZooKeeper setting. Keep the complete ZooKeeper configuration, resources, JVM options, and metrics mapping fixed unless one of them is the single variable under test. +The `RdsKafka` API exposes ZooKeeper replicas, resources, storage, JVM options, Pod template, and logging. It does not expose arbitrary ZooKeeper configuration or `metricsConfig`. The product operator injects the generated Strimzi `Kafka.spec.zookeeper.metricsConfig` and the corresponding metrics ConfigMap. Do not patch the generated `Kafka` resource to bypass the `RdsKafka` API; reconciliation can overwrite that change. Keep the exposed ZooKeeper fields and the generated metrics mapping fixed unless one of them is the single variable under test. ## 6. Persistent-Volume Capacity and Storage Qualification @@ -327,7 +330,40 @@ kubectl explain rdskafka.spec --recursive \ >"$OUT/rdskafka-spec-schema.txt" ``` -For a ZooKeeper-based instance, also verify these fields in the generated `Kafka` resource. Field availability is determined by the installed CRD: +For a ZooKeeper-based instance, verify the product-facing fields below against the installed `RdsKafka` CRD: + +```bash +kubectl explain rdskafka.spec.zookeeper --recursive \ + >"$OUT/rdskafka-zookeeper-schema.txt" +kubectl get imageversion \ + -l 'middleware.instance/type=kafka,middleware.instance/current=true' \ + -o jsonpath='{range .items[*].spec.components.kafka.versions[*]}{.displayVersion}{"\t"}{.version}{"\n"}{end}' \ + >"$OUT/kafka-supported-versions.txt" +test -s "$OUT/kafka-supported-versions.txt" +``` + +| `RdsKafka` field | Standard test requirement | +| --- | --- | +| `spec.mode` | Explicitly set to the API enum value `Zookeeper` | +| `spec.version` | A ZooKeeper-capable version advertised by the installed product; for the deployed operator 2.x path, select a supported Kafka 2.x version | +| `spec.replicas` | 3 brokers | +| `spec.resources.requests/limits` | Current broker matrix size; record both requests and limits | +| `spec.storage` | Qualified broker persistent storage sized by section 6 | +| `spec.config` | Frozen broker baseline; record `inter.broker.protocol.version` and `log.message.format.version` when present | +| `spec.kafka` | Listener, authorization, Pod template, JVM, and logging settings for brokers | +| `spec.zookeeper.replicas` | Odd count, with at least 3 for a production-representative test | +| `spec.zookeeper.resources` | Fixed requests and limits; required when `spec.zookeeper` is present | +| `spec.zookeeper.storage` | Persistent, low-latency block storage with fixed class, size, and `deleteClaim`; required when `spec.zookeeper` is present | +| `spec.zookeeper.jvmOptions` | Fixed heap settings, when explicitly configured | +| `spec.zookeeper.template` | Fixed Pod security, affinity, and toleration settings, when explicitly configured | +| `spec.zookeeper.logging` | Fixed inline logging settings, when explicitly configured | +| `spec.kafkaExporter` | Enabled when consumer group lag is required, with topic and group regular expressions limited to the test scope | + +The operator 2.x implementation is selected from `spec.version`, not from `spec.mode`. For the legacy Kafka 2.x versions supported by the installed release, the `RdsKafka` reconciler always builds a ZooKeeper-based Strimzi resource. Still set `spec.mode: Zookeeper` explicitly so the declared API state matches the generated topology and remains unambiguous to automation. + +If `spec.zookeeper` is omitted, the current implementation copies broker replica, resource, and storage settings into ZooKeeper defaults. Do not rely on that fallback for a performance baseline: specify `spec.zookeeper` so metadata-service sizing is explicit and can be held constant while broker sizes change. + +Also verify these read-only results in the generated Strimzi `Kafka` resource. Field availability is determined by the installed Strimzi CRD: | Generated `Kafka` field | Standard test requirement | | --- | --- | @@ -340,11 +376,10 @@ For a ZooKeeper-based instance, also verify these fields in the generated `Kafka | `spec.zookeeper.resources` | Fixed requests and limits | | `spec.zookeeper.storage` | Persistent, low-latency block storage with a fixed type, class, and size | | `spec.zookeeper.jvmOptions` | Fixed heap and JVM settings | -| `spec.zookeeper.config` | Frozen ZooKeeper configuration | -| `spec.zookeeper.metricsConfig` | Java Management Extensions (JMX) Prometheus Exporter mapping enabled for the test | +| `spec.zookeeper.metricsConfig` | Automatically generated Java Management Extensions (JMX) Prometheus Exporter mapping is present | | `spec.kafkaExporter` | Enabled when consumer group lag is required, with topic and group regular expressions limited to the test scope | -Do not apply KRaft-only fields such as `spec.mode`, `spec.controller`, or `KafkaNodePool` to a legacy ZooKeeper-based resource. +Do not set the KRaft-only `RdsKafka.spec.controller` field or create `KafkaNodePool` resources for a ZooKeeper-based instance. ### 7.2 Fixed variables @@ -417,7 +452,7 @@ kubectl -n "$NS" get pod \ -l "strimzi.io/name=${CLUSTER}-zookeeper" -o wide ``` -If the last command finds no Pod, Kafka Exporter is disabled or the current version uses different labels. Check `RdsKafka`, the generated `Kafka` resource, and actual Pod labels. Do not interpret an empty result as zero lag. +If the Kafka Exporter query finds no Pod, Kafka Exporter is disabled or the current version uses different labels. Do not interpret that empty result as zero lag. If the ZooKeeper query finds no Pod, stop the ZooKeeper-mode test and inspect `RdsKafka` status, the generated `Kafka` resource, and actual Pod labels. ## 8. Deploy Load-Test Clients @@ -1060,7 +1095,7 @@ fi Raw endpoint output proves only that the exporter exposes metrics; it does not prove that the platform is collecting time-series data. Before formal testing, query broker metrics for the last 15 minutes in platform monitoring or Prometheus and confirm continuous samples for every broker. If the query is empty or has collection gaps, stop the test and correct the ServiceMonitor or PodMonitor and metric mapping for the current platform. Selector labels depend on the current Prometheus configuration; do not copy labels from another cluster. -For ZooKeeper mode, enable the `spec.zookeeper.metricsConfig` mapping and collect at least: +For ZooKeeper mode, verify that the product operator populated the generated `Kafka.spec.zookeeper.metricsConfig` mapping. This is generated output, not a configurable field in `RdsKafka.spec.zookeeper`. Collect at least: | ZooKeeper evidence | Purpose | | --- | --- | From 43b9102665b8d44ab734c358102ef5bcc4ecced8 Mon Sep 17 00:00:00 2001 From: hfxia Date: Thu, 30 Jul 2026 00:52:53 -0400 Subject: [PATCH 8/8] docs: apply cross-review fixes to Kafka and Redis performance guides - Cover Kafka 4.0/4.1 clients (Strimzi 0.48 line) alongside the 4.2 and legacy 2.x script interfaces instead of a 4.2-vs-2.x dichotomy - Replace ParamTemplate/ParamDefinition kubectl access with the product console workflow; parameter templates live in the global cluster and are not reachable from the business cluster - Add ZooKeeper-mode RdsKafka field checks and supported-version query - Require adequate client resources for the 20-consumer example and document the 30-partition/20-consumer assignment skew - Remove internal references (Ares, e2e matrix), fix frontmatter kind and ProductsVersion, correct a capacity rounding artifact --- .../kafka/Kafka_Performance_Testing_Guide.md | 67 +++++++------------ .../redis/Redis_Performance_Testing_Guide.md | 44 +++++------- 2 files changed, 40 insertions(+), 71 deletions(-) diff --git a/docs/en/solutions/ecosystem/kafka/Kafka_Performance_Testing_Guide.md b/docs/en/solutions/ecosystem/kafka/Kafka_Performance_Testing_Guide.md index adf348153..56b18107e 100644 --- a/docs/en/solutions/ecosystem/kafka/Kafka_Performance_Testing_Guide.md +++ b/docs/en/solutions/ecosystem/kafka/Kafka_Performance_Testing_Guide.md @@ -3,6 +3,9 @@ products: - Alauda Application Services kind: - Solution +ProductsVersion: + - 3.x + - 4.x --- # Kafka Performance Testing Guide @@ -22,7 +25,7 @@ In-sync replica (ISR) terminology follows the Apache Kafka documentation. This guide covers both Kafka Raft metadata mode (KRaft) and legacy ZooKeeper-based instances. Treat the metadata mode as part of the test profile. Do not compare or combine results across metadata modes. -The primary commands use the Apache Kafka 4.2.0 script interfaces. Section 8.5 lists the required command changes for the legacy Kafka 2.x clients shipped with ZooKeeper-based operator releases. For any installed version, first save the `--version` and `--help` output and use only the options shown by that version. +The primary commands use the Apache Kafka 4.2.0 script interfaces. Several of these options were standardized only in Kafka 4.2 and do not exist in earlier clients: operator releases based on Strimzi 0.48 ship Kafka 4.0 or 4.1 images, and the ZooKeeper-based operator line ships legacy Kafka 2.x images. Section 8.5 lists the equivalent commands for clients older than Kafka 4.2. For any installed version, first save the `--version` and `--help` output and use only the options shown by that version. This guide does not provide a universally applicable "best throughput" or "best configuration." A change in hardware, storage, network, message size, compression ratio, reliability policy, or workload can make results incomparable. @@ -163,26 +166,9 @@ Export only non-sensitive metadata. Do not write kubeconfig files, Secret data, ## 5. Export and Freeze Instance Parameters -The general Kafka parameter template is usually named `general-kafka`. The template is product configuration in the management cluster. The `RdsKafka` resource in the business cluster stores the final effective instance configuration. Templates can change between product versions, so export the installed version for every test instead of relying on the table in this guide. +The general Kafka parameter template is usually named `general-kafka`. Parameter templates are product configuration stored in the global cluster, not in the business cluster, so do not read them with `kubectl` from the test environment. Select and review the template in the product console; the `RdsKafka` resource in the business cluster stores the final effective instance configuration. Templates can change between product versions, so freeze the baseline from the created instance for every test instead of relying on the table in this guide. -If you have read access to the management cluster, run: - -```bash -export MGMT_KUBECONFIG=/path/to/management-cluster.kubeconfig - -kubectl --kubeconfig "$MGMT_KUBECONFIG" get paramtemplate -A \ - -l component=kafka - -export TEMPLATE_NS="$(kubectl --kubeconfig "$MGMT_KUBECONFIG" \ - get paramtemplate -A -l component=kafka \ - -o jsonpath='{range .items[?(@.metadata.name=="general-kafka")]}{.metadata.namespace}{"\n"}{end}')" -test -n "$TEMPLATE_NS" -kubectl --kubeconfig "$MGMT_KUBECONFIG" -n "$TEMPLATE_NS" \ - get paramtemplate general-kafka -o yaml \ - >"$OUT/general-kafka-paramtemplate.yaml" -``` - -If management-cluster access is unavailable, select the general Kafka parameter template in the product console and export the final `spec.config` from the instance. Do not infer values from the template name. +When creating the instance, select the general Kafka parameter template in the product console and record the template name, description, and displayed parameter values. After the instance is created, treat the final `spec.config` exported in section 7.2 as the frozen baseline. Do not infer values from the template name. Review at least the following parameters in the current standard general template. The example baseline only identifies parameters; it does not replace the installed template. @@ -204,12 +190,7 @@ Review at least the following parameters in the current standard general templat | `delete.topic.enable` | `true` | Allows test topics to be deleted | | `unclean.leader.election.enable` | `false` | Prevents a replica outside the ISR from becoming leader | -The current product version determines how parameter-template changes are applied. Before testing, query the parameter definition and record its `applyStrategy`. If it is `RestartApply`, exclude the rolling-restart interval from the measurement window. - -```bash -kubectl --kubeconfig "$MGMT_KUBECONFIG" get paramdefinition kafka-general -o yaml \ - >"$OUT/kafka-general-paramdefinition.yaml" -``` +The current product version determines how parameter changes are applied. Before changing a parameter, check in the console parameter editor whether the change is marked as requiring a restart, and record that classification in the report. If applying the change performs a rolling restart, exclude the rolling-restart interval from the measurement window. ### 5.1 Additional snapshot for a ZooKeeper-based instance @@ -321,7 +302,7 @@ After submitting the console form, verify the following KRaft fields against the | `spec.storage.size` | At least the value calculated in section 6 | | `spec.storage.deleteClaim` | Select according to data-retention and destruction requirements; do not set it to `true` only to simplify cleanup | | `spec.controller` | Roles, count, resources, and storage match the production plan; use at least 3 voting nodes for dedicated controllers | -| `spec.config` | Exported `general-kafka` baseline plus the one parameter changed in this run | +| `spec.config` | Baseline recorded from the selected general template plus the one parameter changed in this run | | `spec.kafka.listeners` | Matches the in-cluster or external access path for this run | | `spec.kafkaExporter` | Enabled when consumer group lag is required, with topic and group regular expressions limited to the test scope | @@ -458,7 +439,7 @@ If the Kafka Exporter query finds no Pod, Kafka Exporter is disabled or the curr ### 8.1 Reuse the instance Kafka image -Read the Kafka image used by the operator from a broker Pod. This keeps the script version aligned with the server version without exposing an image-registry address in an external document. +Read the Kafka image used by the operator from a broker Pod. This keeps the script version aligned with the server version without exposing an image-registry address in an external document. The Kafka version of this image also determines which script options exist; record it in section 8.4 and use the section 8.5 forms when the image is older than Kafka 4.2. ```bash export KAFKA_IMAGE="$(kubectl -n "$NS" get pod \ @@ -619,15 +600,15 @@ kubectl -n "$NS" exec kafka-consumer-perf -- \ If Kafka is installed at a path other than `/opt/kafka` in the image, confirm the path from the image documentation or broker container environment and update the commands. Do not mix in scripts downloaded from an unknown version. -### 8.5 Legacy Kafka 2.x script compatibility +### 8.5 Script compatibility for clients older than Kafka 4.2 -The ZooKeeper-based operator line uses Kafka 2.x client scripts whose interfaces differ from the Kafka 4.2 examples. The reviewed legacy scripts have these differences: +The standardized options used in the primary examples were introduced in Apache Kafka 4.2 and are absent from earlier clients. This affects two shipped client generations: operator releases based on Strimzi 0.48 provide Kafka 4.0 or 4.1 images, and the ZooKeeper-based operator line provides legacy Kafka 2.x images. Replace the affected options for both generations as follows: -| Operation | Kafka 4.2 example in this guide | Legacy Kafka 2.x form | +| Operation | Kafka 4.2 example in this guide | Form for Kafka 2.x, 4.0, and 4.1 | | --- | --- | --- | | Producer bootstrap and property file | `--bootstrap-server` and `--command-config` | Put `bootstrap.servers=$BOOTSTRAP` in the property file and use `--producer.config` | -| Producer warmup | `--warmup-records` | No equivalent option in the reviewed scripts; run a separate, predefined warmup cycle | -| Producer interval option | `--reporting-interval` | Not exposed by the reviewed producer script; preserve its raw interval output | +| Producer warmup | `--warmup-records` | No equivalent option (`--warmup-records` was added in Kafka 4.2); run a separate, predefined warmup cycle | +| Producer interval option | `--reporting-interval` | Not exposed before Kafka 4.2; preserve the script's raw interval output | | Producer property override | `--command-property` | Use a separate complete `--producer.config` file | | Consumer record target | `--num-records` | `--messages` | | Consumer property file | `--command-config` | `--consumer.config` | @@ -635,7 +616,7 @@ The ZooKeeper-based operator line uses Kafka 2.x client scripts whose interfaces Some earlier Kafka 2.x scripts implement `--threads`, while later Kafka 2.x scripts accept it but ignore it. Never use `--threads` to establish the standard concurrency point. Use independent processes or Pods and verify the actual consumer group membership. -For a legacy test, add the resolved bootstrap address to the producer property file before creating the ConfigMap or Secret: +For a pre-4.2 producer test, add the resolved bootstrap address to the producer property file before creating the ConfigMap or Secret: ```properties bootstrap.servers=perf-kafka-kafka-bootstrap.kafka-perf.svc:9092 @@ -709,7 +690,7 @@ kubectl -n "$NS" exec kafka-producer-perf -- \ `--throughput -1` disables client-side throttling and searches for the limit at the current client concurrency. It does not prove that the brokers are saturated; evaluate client and broker metrics together. Keep `--warmup-records` fixed in formal comparisons and use only the script's steady-state summary. Do not include warmup in the result. -For a legacy ZooKeeper-based instance, use the same payload and producer properties with the legacy interface: +For a pre-4.2 client (a Kafka 4.0/4.1 image or a legacy ZooKeeper-based instance), use the same payload and producer properties with the earlier interface: ```bash kubectl -n "$NS" exec kafka-producer-perf -- \ @@ -723,7 +704,7 @@ kubectl -n "$NS" exec kafka-producer-perf -- \ | tee "$OUT/producer-${RECORD_SIZE}B.txt" ``` -Run a separate, fixed warmup cycle before the measured legacy command. Do not count its records in the test target, and use a fresh topic or account explicitly for warmup data before a consumer-only test. +Run a separate, fixed warmup cycle before the measured pre-4.2 command. Do not count its records in the test target, and use a fresh topic or account explicitly for warmup data before a consumer-only test. ### 10.2 Leader acknowledgment ceiling @@ -759,7 +740,7 @@ kubectl -n "$NS" exec kafka-producer-perf -- \ Label the result `leader-ack`. Do not compare it directly with `acks=all` capacity. -For a legacy client, create a separate `producer-leader-ack.properties` file containing the same properties as the baseline except for `acks=1` and `enable.idempotence=false`, then pass it with `--producer.config`. Save a redacted diff between the two files. Do not use an unsupported `--command-property` option. +For a pre-4.2 client, create a separate `producer-leader-ack.properties` file containing the same properties as the baseline except for `acks=1` and `enable.idempotence=false`, then pass it with `--producer.config`. Save a redacted diff between the two files. Do not use an unsupported `--command-property` option. ### 10.3 Consumer-only test @@ -791,7 +772,7 @@ kubectl -n "$NS" exec kafka-consumer-perf -- \ In Kafka 4.2, `--messages` is deprecated; use `--num-records`. `--fetch-size` limits the data fetched from one partition in one request, and the `max.partition.fetch.bytes` client property also applies. Save `records-consumed-total` from the `--print-metrics` output and confirm that the consumer read the target count. A timeout, insufficient topic data, or authorization error can make the script exit early, so do not rely only on the final throughput line. -For a legacy ZooKeeper-based instance, use `--messages` and `--consumer.config`: +For a pre-4.2 client, use `--messages` and `--consumer.config`: ```bash kubectl -n "$NS" exec kafka-consumer-perf -- \ @@ -809,7 +790,7 @@ kubectl -n "$NS" exec kafka-consumer-perf -- \ | tee "$OUT/consumer-${RECORD_SIZE}B.txt" ``` -The legacy script reports messages rather than records in some headings. Preserve its original output labels and normalize only in separate report fields. +The legacy Kafka 2.x script reports messages rather than records in some headings. Preserve its original output labels and normalize only in separate report fields. ### 10.4 Concurrent produce and consume @@ -823,7 +804,7 @@ Do not run one consumer command in the foreground and then attempt to start the ### 10.5 Example with 20 consumers on Kafka 4.2 -This example starts 20 independent consumer processes in one client Pod and checks every `kubectl exec` exit code. Before a formal test, use section 12.1 to confirm that Pod CPU, memory, and network are not bottlenecks. If resources are insufficient, replicate the consumer Pod from section 8.3 and distribute the processes across load-test nodes. +This example starts 20 independent consumer processes in one client Pod and checks every `kubectl exec` exit code. The consumer Pod from section 8.3 cannot run it unchanged: each script process starts a JVM whose default maximum heap is 512 MiB, so 20 processes can demand several times the Pod's 2Gi limit, and 20 consumers sharing 1 vCPU make the client the bottleneck before Kafka. Increase the consumer Pod's resources, or replicate the Pod from section 8.3 and spread the processes across load-test nodes, for example four Pods with five processes each; then use section 12.1 to confirm that client CPU, memory, and network are not bottlenecks. Also account for the partition arithmetic: with 30 partitions and 20 consumers, ten consumers receive two partitions and ten receive one, so the equal per-process record targets complete only as finished consumers leave the group and rebalances reassign their partitions. Expect rebalances near the end of the point and verify the summed record counts as described below. ```bash export CONSUMER_COUNT=20 @@ -866,7 +847,7 @@ test "$failed" -eq 0 Sum the actual records consumed across all 20 files and confirm the total equals `NUM_RECORDS`. Calculate aggregate throughput as total records or bytes divided by the common measurement window. If process start and end times differ, do not add the individual average rates. A timeout, nonzero exit, or record-count mismatch in any process invalidates the test point. -For legacy Kafka 2.x, use the same 20-process allocation but replace `--num-records` with `--messages` and `--command-config` with `--consumer.config`. Do not add `--threads`; on later Kafka 2.x scripts it is deprecated and ignored. +For pre-4.2 clients, use the same 20-process allocation but replace `--num-records` with `--messages` and `--command-config` with `--consumer.config`, and omit `--command-property`; if a per-process `client.id` is required, use a separate properties file for each process. Do not add `--threads`; on later Kafka 2.x scripts it is deprecated and ignored. ## 11. Client Parameter Reference @@ -877,7 +858,7 @@ For legacy Kafka 2.x, use the same 20-process allocation but replace `--num-reco | `--num-records` | See the matrix | Records sent; total payload is approximately this value multiplied by `--record-size` | | `--record-size` | 100/500/1000 | Bytes in the synthetic record value; mutually exclusive with a message payload file | | `--throughput` | `-1` | Unthrottled limit search; for controlled load, set the target records/s | -| `--warmup-records` | Fixed value | Kafka 4.2 option that excludes connection setup, metadata, and JVM warmup; use a separate fixed warmup cycle on the reviewed legacy scripts | +| `--warmup-records` | Fixed value | Added in Kafka 4.2; excludes connection setup, metadata, and JVM warmup from the summary; use a separate fixed warmup cycle on pre-4.2 clients | | `acks` | `all` or `1` | Acknowledgment strength; changing it changes reliability and performance semantics | | `enable.idempotence` | `true` | Prevents duplicates caused by retries; requires compatible acknowledgments, retries, and in-flight request settings | | `batch.size` | `50000` | Target maximum batch size per partition; a large value can increase memory use and wait time | @@ -892,7 +873,7 @@ Each partition has an independent batch. More producers, partitions, batch capac | Parameter | Standard value | Mechanism and boundary | | --- | ---: | --- | | `--group` | Unique per run | Consumers in the same group share partitions; a reused group can start at old offsets | -| `--num-records` or legacy `--messages` | Matches preloaded data | Planned total records; select the version-supported option and verify the actual value with client metrics | +| `--num-records` or pre-4.2 `--messages` | Matches preloaded data | Planned total records; select the version-supported option and verify the actual value with client metrics | | `--fetch-size` | `200000` | Per-partition fetch size passed by the script | | `--timeout` | `60000` | Maximum time between returned records, not total test duration; a timeout makes the tool exit early with a warning | | `fetch.min.bytes` | `1` | Minimum data before the broker returns a Fetch response; increasing it can improve batching but can add wait latency | diff --git a/docs/en/solutions/ecosystem/redis/Redis_Performance_Testing_Guide.md b/docs/en/solutions/ecosystem/redis/Redis_Performance_Testing_Guide.md index 7c496a34a..4932a580c 100644 --- a/docs/en/solutions/ecosystem/redis/Redis_Performance_Testing_Guide.md +++ b/docs/en/solutions/ecosystem/redis/Redis_Performance_Testing_Guide.md @@ -2,7 +2,9 @@ products: - Alauda Application Services kind: - - Solutions + - Solution +ProductsVersion: + - 4.x --- # Redis Cluster and Sentinel Performance Testing Guide @@ -32,7 +34,7 @@ Redis Database (RDB) persistence, Append Only File (AOF) persistence, and a non- | Axis | Standard values | Purpose | |---|---|---| | Parameter template | RDB persistence; AOF persistence; no persistence (cache) | Quantifies durability and replication cost separately using shipped product baselines | -| Value size (`-d`) | `512` bytes; `4096` bytes (4 KiB) | Matches the latest performance e2e matrix for small and medium values | +| Value size (`-d`) | `512` bytes; `4096` bytes (4 KiB) | Standard small and medium value sizes for comparable baseline results | | Pipeline (`-P`) | `1`, `16` | `1` measures non-pipelined behavior; `16` measures a throughput-oriented workload | | Commands (`-t`) | `set,get` | Produces separate write and read results | | Random keyspace (`-r`) | `100000` | Avoids benchmarking a single hot key | @@ -40,7 +42,7 @@ Redis Database (RDB) persistence, Append Only File (AOF) persistence, and a non- ### Select a product parameter template -This procedure uses the shipped Redis 7.2 templates as its only parameter baseline. In the Redis create form, choose Redis 7.2 and the topology first. Under **Parameter Templates** in the English interface or **参数模板** in the Chinese interface, select the exact identifier that matches both. The identifier displayed in the selector is `metadata.name` and is the same in both languages; only the description is localized. The bilingual profile names below describe the intent and are not alternative resource names. +This procedure uses the shipped Redis 7.2 templates as its only parameter baseline. In the Redis create form, choose Redis 7.2 and the topology first. Under **Parameter Templates** in the English interface or **参数模板** in the Chinese interface, select the exact identifier that matches both. The identifier displayed in the selector is the template's unique name and is the same in both languages; only the description is localized. The bilingual profile names below describe the intent and are not alternative resource names. | Profile name (中文 / English) | Redis 7.2 Sentinel | Redis 7.2 Cluster | |---|---|---| @@ -58,34 +60,24 @@ The shipped Redis 7.2 templates establish these test baselines: | AOF persistence | `appendonly=yes`; `appendfsync=everysec`; `save=""` | AOF durability/throughput balance | | No persistence (cache) | `appendonly=no`; `save=""`; `repl-diskless-sync=yes`; `repl-backlog-size=50mb` | Cache workloads that can tolerate data loss | -All listed templates leave `maxmemory` unset and set `maxmemory-policy=noeviction`, `repl-diskless-sync=yes`, `io-threads=4`, and `io-threads-do-reads=no`. With this combination, the Operator generates `maxmemory` as 80% of the Redis data container's memory limit. This is Operator behavior, not the native Redis default. Because installed package revisions can differ, export the installed template and verify the effective configuration rather than reproducing these values manually. +All listed templates leave `maxmemory` unset and set `maxmemory-policy=noeviction`, `repl-diskless-sync=yes`, `io-threads=4`, and `io-threads-do-reads=no`. With this combination, the Operator generates `maxmemory` as 80% of the Redis data container's memory limit. This is Operator behavior, not the native Redis default. Because installed package revisions can differ, review the selected template's parameters in the product console and verify the effective configuration on the running instance rather than reproducing these values manually. For Sentinel topology, template selection also supplies Sentinel-process parameters. Keep them unchanged during a data-plane throughput comparison; Sentinel processes do not serve the benchmark traffic, and failover testing is outside this steady-state procedure. The no-persistence profile is not "diskless persistence": it disables local persistence and uses diskless synchronization only for replication. Use it only when the application can tolerate data loss. Redis documents the durability and performance trade-offs of RDB, AOF, and no persistence in [Redis persistence](https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/). -List and export the installed templates before testing. Discover the namespace instead of assuming one: - -```bash -kubectl get paramtemplate.middleware.alauda.io -A \ - -l component=redis,support_version=7.2 - -TEMPLATE_NS=replace-with-template-namespace -TEMPLATE=system-rdb-redis-7.2-cluster -kubectl -n "${TEMPLATE_NS}" get paramtemplate.middleware.alauda.io \ - "${TEMPLATE}" -o yaml > "paramtemplate-${TEMPLATE}.yaml" -``` +Parameter templates are stored in the global cluster and are not visible from the business cluster, so verify them through the product console instead of `kubectl`. Before testing, open the parameter-template list in the console, confirm that the identifiers above exist for the selected Redis version and topology, and record each selected template's displayed parameters for the report. After instance creation, verify the effective values with `CONFIG GET` as described later in this guide. ### Other Redis versions -Template identifiers, supported parameters, defaults, persistence behavior, and apply strategies can differ by Redis and product package version. For any version other than 7.2, select a template whose labels match that exact version and topology, export the installed `ParamTemplate`, and build a separately named test matrix from its effective values. Do not reuse the Redis 7.2 identifiers or assume that its resource, storage, persistence, or I/O-thread defaults apply. +Template identifiers, supported parameters, defaults, persistence behavior, and apply strategies can differ by Redis and product package version. For any version other than 7.2, select the template that the console offers for that exact version and topology, record its displayed parameters, and build a separately named test matrix from the instance's effective values. Do not reuse the Redis 7.2 identifiers or assume that its resource, storage, persistence, or I/O-thread defaults apply. ## Prerequisites Prepare the following before the test window: - A Kubernetes cluster with the Redis Operator installed and a qualified `StorageClass` for RDB and AOF profiles. -- The Redis 7.2 system `ParamTemplate` resources installed for the topology under test. +- The Redis 7.2 system parameter templates available in the Redis create form for the topology under test. - A namespace dedicated to the test, for example `redis-perf`. - Product-console permission to create Redis instances and select parameter templates, plus `kubectl` access that can create the load Pod and read Redis resources and metrics. Install `jq` for producing allowlisted evidence exports. - A Redis password Secret with key `password`. Reuse the instance Secret or create it from a protected file; never place the password in a manifest or report: @@ -139,13 +131,9 @@ SC_NAME=sc-topolvm # Example; replace with the selected StorageClass. kubectl get storageclass "${SC_NAME}" -o yaml \ > "${OUT}/storageclass-${SC_NAME}.yaml" -TEMPLATE_NS=replace-with-template-namespace -TEMPLATE=replace-with-selected-template-identifier -kubectl -n "${TEMPLATE_NS}" get paramtemplate.middleware.alauda.io \ - "${TEMPLATE}" -o yaml > "${OUT}/paramtemplate-${TEMPLATE}.yaml" ``` -Also record the Redis Operator version, Redis server version, image IDs, node CPU model and frequency policy, network link speed, StorageClass/Container Storage Interface (CSI) implementation, volume performance class, service mesh status, and test time zone. Redis recommends isolated hardware, stable CPU frequency, known client/server network latency, and no unrelated storage I/O for reproducible results. Verify `vm.overcommit_memory=1` and Transparent Huge Pages are disabled on Redis nodes according to [Redis administration guidance](https://redis.io/docs/latest/operate/oss_and_stack/management/admin/); platform administrators must apply host changes through the supported node-management process. +Also record the selected parameter-template identifier and its displayed parameters, the Redis Operator version, Redis server version, image IDs, node CPU model and frequency policy, network link speed, StorageClass/Container Storage Interface (CSI) implementation, volume performance class, service mesh status, and test time zone. Redis recommends isolated hardware, stable CPU frequency, known client/server network latency, and no unrelated storage I/O for reproducible results. Verify `vm.overcommit_memory=1` and Transparent Huge Pages are disabled on Redis nodes according to [Redis administration guidance](https://redis.io/docs/latest/operate/oss_and_stack/management/admin/); platform administrators must apply host changes through the supported node-management process. ## Configure Redis instances @@ -155,10 +143,10 @@ Create each test instance through the product console so the selected system tem 2. Select the exact **Parameter Templates / 参数模板** identifier from the table above. Do this before manually changing Redis parameters. 3. The listed system templates assign limits of `4` CPU and `8Gi` memory to each Redis server, matching the standard capacity profile. Confirm those values after selection and set requests equal to limits where the form exposes both. Configure three primary shards and one replica per primary for Cluster, or one primary, one replica, and three Sentinel processes for Sentinel. Treat any resource or topology change as a separately named profile. 4. For RDB or AOF, choose the qualified `StorageClass` for the declared performance-ceiling or production-representative profile. For both topologies, the shipped templates prefill `24Gi` for RDB and `32Gi` for AOF. Treat these values as minimum starting capacities, calculate the requirement as described below, and use the larger value. The no-persistence templates do not require persistent data storage. -5. Enable the Redis exporter. Expand **Parameters / 参数配置** and verify `appendonly`, `appendfsync`, `save`, `maxmemory`, `maxmemory-policy`, `repl-diskless-sync`, `io-threads`, and `io-threads-do-reads` against the exported template. An empty template value for `maxmemory` is intentional; verify the Operator-derived effective value after creation. +5. Enable the Redis exporter. Expand **Parameters / 参数配置** and verify `appendonly`, `appendfsync`, `save`, `maxmemory`, `maxmemory-policy`, `repl-diskless-sync`, `io-threads`, and `io-threads-do-reads` against the parameters displayed for the selected template. An empty template value for `maxmemory` is intentional; verify the Operator-derived effective value after creation. 6. After correcting resource and storage capacity for the target dataset, create the baseline with the Redis parameters unchanged. For a tuned comparison, start again from the same template and change only the documented target parameter. If the installed platform supports customer-owned templates, save the tuned copy under a new name; never edit the shipped system template in place. -Template selection materializes the template values into the Redis instance configuration; the Redis custom resource does not retain a template-name reference. Therefore, the report must preserve both the exact exported `ParamTemplate` and the resulting Redis custom resource. Do not invent a `spec.paramTemplate` field for automation. +Template selection materializes the template values into the Redis instance configuration; the Redis custom resource does not retain a template-name reference. Therefore, the report must record the exact template identifier selected in the console, its displayed parameter values, and the resulting Redis custom resource. Do not invent a `spec.paramTemplate` field for automation. ### Set persistent volume capacity for the standard matrix @@ -186,7 +174,7 @@ For the standard matrix (`N=1,000,000`, `K=100,000`, `S=1`), the calculation is: | Persistence | Value size | Calculated peak | Peak × `1.5` | Template minimum | Provision per data Pod | |---|---:|---:|---:|---:|---:| | RDB | 512 B | 0.095 GiB | 0.143 GiB | 24Gi | **24Gi** | -| RDB | 4096 B | 0.763 GiB | 1.145 GiB | 24Gi | **24Gi** | +| RDB | 4096 B | 0.763 GiB | 1.144 GiB | 24Gi | **24Gi** | | AOF | 512 B | 0.572 GiB | 0.858 GiB | 32Gi | **32Gi** | | AOF | 4096 B | 4.578 GiB | 6.867 GiB | 32Gi | **32Gi** | @@ -203,7 +191,7 @@ Redis serialization, AOF protocol framing, key names, and other files are not in Use a fresh instance for every parameter-template, value-size, pipeline, and Redis I/O-thread scenario. During a client-concurrency sweep on an AOF instance, either recreate the instance for every measured point or run `BGREWRITEAOF` after warm-up and after each point, wait for `aof_rewrite_in_progress:0`, and require `aof_last_bgrewrite_status:ok` before continuing. Run this housekeeping outside the measured interval. -If `-n`, `-r`, `-d`, or `S` changes, recompute the capacity before creating the instance. Do not reuse the standard values for the earlier 5-30-million-request Ares cases or a customer workload without recalculation. Multiply the per-Pod result by two for the standard Sentinel topology or by six for the standard three-primary, one-replica-per-primary Cluster topology to obtain total Kubernetes-requested capacity. Storage-system replication, thin-provisioning reserve, and filesystem overhead are additional. For local block storage, every eligible node must have enough allocatable local capacity for the Redis data Pods that can be scheduled there. Provision the calculated capacity before testing; do not resize a PVC during a measured run. +If `-n`, `-r`, `-d`, or `S` changes, recompute the capacity before creating the instance. Do not reuse the standard values for larger request counts (for example, 5 to 30 million requests) or a customer workload without recalculation. Multiply the per-Pod result by two for the standard Sentinel topology or by six for the standard three-primary, one-replica-per-primary Cluster topology to obtain total Kubernetes-requested capacity. Storage-system replication, thin-provisioning reserve, and filesystem overhead are additional. For local block storage, every eligible node must have enough allocatable local capacity for the Redis data Pods that can be scheduled there. Provision the calculated capacity before testing; do not resize a PVC during a measured run. Wait for `.status.phase` to become `Ready`, then verify Pod placement: @@ -250,7 +238,7 @@ io-threads 3 io-threads-do-reads no ``` -The product `ParamDefinition` classifies both keys as `RestartApply`. Use a fresh instance for the cleanest comparison, or allow the managed restart to finish, wait for `Ready`, and confirm that all expected primaries and replicas have returned before loading data. Verify the effective values on every Redis data Pod with the collection helper below; a mixed value across nodes invalidates the run. +The product applies changes to both keys through a restart. Use a fresh instance for the cleanest comparison, or allow the managed restart to finish, wait for `Ready`, and confirm that all expected primaries and replicas have returned before loading data. Verify the effective values on every Redis data Pod with the collection helper below; a mixed value across nodes invalidates the run. Calibrate the load generator before comparing server candidates. Redis 7.2 recommends running `redis-benchmark` in threaded mode when evaluating I/O threads. Set client `--threads` high enough that the client retains CPU and network headroom at the largest server candidate, then hold the client setting constant across the `io-threads` comparison. The load-generator thread count and the server I/O thread count are different parameters and must be reported separately. @@ -584,7 +572,7 @@ topology,redis_version,parameter_template,persistence,storage_profile,storage_cl |---|---| | Environment | Date/time, Kubernetes/Alauda Container Platform (ACP)/Operator/Redis versions, node count/type, CPU model, memory, network, storage profile, `StorageClass`, provisioner, parameters, `volumeBindingMode`, `allowVolumeExpansion`, filesystem/backing media, PVC capacity per Redis Pod, qualification results, image names and immutable digests | | Topology | Cluster shard/replica count or Sentinel primary/replica/Sentinel count; Pod-to-node placement; Service access path | -| Redis configuration | Exact `ParamTemplate` identifier, `metadata.resourceVersion`, `cpaas.io/description` (Chinese), `cpaas.io/description-en` (English), exported template YAML, and allowlisted effective keys (`save`, `appendonly`, `appendfsync`, `maxmemory`, `maxmemory-policy`, `repl-diskless-sync`, `io-threads`, `io-threads-do-reads`, `hz`); approved tuning delta; TLS/ACL state without credentials | +| Redis configuration | Exact parameter-template identifier and description as shown in the console, the resulting Redis custom resource, and allowlisted effective keys (`save`, `appendonly`, `appendfsync`, `maxmemory`, `maxmemory-policy`, `repl-diskless-sync`, `io-threads`, `io-threads-do-reads`, `hz`) verified with `CONFIG GET`; approved tuning delta; TLS/ACL state without credentials | | Load | Tool version/image digest, client location, access type and round-trip latency, commands, `-c`, `-n`, `-d`, `-P`, `--threads`, `-r`, warm-up method, start/end/duration | | Results | `SET`/`GET` requests per second; average, min, p50, p95, p99, max latency when emitted; raw CSV | | Resources | Client and server peak/average CPU/memory, throttling, network receive/transmit, storage operation rate/throughput/latency/queue depth/throttling, storage recovery or rebalancing activity, node pressure |