Skip to content

Proposal: add high-concurrency connection tests (e.g., 100k / 500k / 1M connections) #1759

Open

Description

@lesismal

Motivation

The current tests show how fast a framework is at moderate concurrency. Many production systems also hold very large numbers of mostly-idle connections: IM/chat, push notification gateways, IoT, WebSocket services, long-polling and API gateways. At that scale, the deciding factors are usually memory per connection, CPU overhead and tail latency, not peak RPS. Frameworks that look similar at a few thousand connections can differ by an order of magnitude at 1M.

Adding large-connection tests would make HttpArena a much stronger reference when choosing a framework for these workloads.

Proposal

Add a test category that runs each framework at 100k, 500k and 1M concurrent connections, for example:

  1. Idle hold: open N keep-alive connections and hold them idle. Measure server RSS and memory per connection.
  2. Sparse activity: N connections are open, but only a small fraction (e.g. 1–5%) send requests at any moment. Measure throughput, p50/p99/p999 latency, CPU usage and RSS.
  3. Full load (optional): all N connections send requests. Measure throughput, latency and errors/timeouts.
  4. Ramp-up (optional): measure how long it takes to establish N connections and how many fail.

The same idea could cover WebSocket at high connection counts as well.

Suggested metrics

  • Peak and steady-state RSS, and memory per connection
  • CPU utilization
  • Throughput (RPS)
  • Latency percentiles (p50 / p99 / p999)
  • Connection errors and timeouts

Implementation notes

  • Port limits: one client IP can open only about 64k connections to one server IP:port. The load generator needs multiple source IPs (IP aliases or several client machines) or multiple server ports.
  • System tuning: the server and client need raised ulimit -n, fs.file-max, net.core.somaxconn, net.ipv4.ip_local_port_range and TCP memory settings. Documenting these in the test setup would make results reproducible.
  • Fewer frameworks: frameworks could opt in to this category, so it doesn't add much to the total benchmark time.

Activity

  1. MDA2AV commented on Nov 13, 2026

    @MDA2AV
    Owner

    I'll check what can be done, 1 million is likely not possible when we run both load gen and server on the same box, I am inclined to 200k, maybe push a bit more.

    This test is likely to be unscored though, this is mostly a edge type server test case, most frameworks here are not built to handle high connection count

  2. lesismal commented on Nov 14, 2026

    @lesismal
    ContributorAuthor

    I'll check what can be done, 1 million is likely not possible when we run both load gen and server on the same box, I am inclined to 200k, maybe push a bit more.

    We can use multiple ports, for example, each server can listen on 50 ports.

    That's how I do it in my own benchmark library:
    https://github.com/lesismal/go-websocket-benchmark/blob/main/config/config.go#L73

    This test is likely to be unscored though, this is mostly a edge type server test case, most frameworks here are not built to handle high connection count

    Some frameworks would struggle to handle such a large number of connections and should perhaps be categorized separately. In fact, the current engine, infra, and gateway classifications might not be very clear.

  3. Kaliumhexacyanoferrat commented on Nov 14, 2026

    @Kaliumhexacyanoferrat
    Collaborator

    I like the idea - and agree that servers are mostly idle having tons of keep alive connections open. The scoring would be interesting though - just memory?

  4. lesismal commented on Nov 14, 2026

    @lesismal
    ContributorAuthor

    I like the idea - and agree that servers are mostly idle having tons of keep alive connections open. The scoring would be interesting though - just memory?

    @Kaliumhexacyanoferrat My own tests ranked TPS, CPU, and memory efficiency ratios(Energy Efficiency Ratio), as shown in the image: CPU EER and MEM EER items. However, I don't have a good idea for the overall score yet; you guys are more professional in this area.

    image

    github actions runs the benchmarks but only 2 CPU for each server in github action.
    https://github.com/lesismal/go-websocket-benchmark/actions/runs/42801023661

  5. MDA2AV commented on Nov 16, 2026

    @MDA2AV
    Owner

    first draft #1815

  6. Antruly commented on Nov 18, 2026

    @Antruly
    Contributor

    I think this reflects the real state of a server, and I also think we should add some request/response handling for random idle connections. That would better represent the framework’s response speed under real-world conditions, rather than relying on pre-defined expected request processing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    testsAbout test design, issues with tests etc.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions