Skip to content

benchmarking: add memory working set to server telemetry - #2386

Merged
Max Smythe (maxsmythe) merged 1 commit into
agent-substrate:mainfrom
Nishanth29:working-set
Oct 17, 2026
Merged

Max Smythe (maxsmythe) merged 1 commit into
agent-substrate:mainfrom
Nishanth29:working-set

Conversation

@Nishanth29

@Nishanth29 Nishanth Kotla (Nishanth29) commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Part of #1590

Adds working_set to server telemetry:

  • node, worker, atelet: from cAdvisor (container_memory_working_set_bytes).
  • actor, per_actor: reuse the atelet's existing actor stats metrics, ate_actor_stats_memory_working_set_bytes and ate_actor_stats_sampled_actors. actor is the actors' working set per atelet; per_actor divides it by the number of actors that atelet measured.

Both sources are already collected (cAdvisor and the atelet), so nothing changes in Substrate or the monitoring setup; this only adds queries.

Live check (gVisor, 4 users, 3m, through the telemetry-meter):

  • actor 1.13 GiB vs worker 1.22 GiB at the same time

  • per_actor × count = actor (0.5628 × 2 = 1.1257)

  • 74 measurements, none null

  • Tests pass

  • Appropriate changes to documentation are included in the PR

@Nishanth29

Copy link
Copy Markdown
Contributor Author

@Nishanth29

Copy link
Copy Markdown
Contributor Author

Da Huang (@git286) Tim Bai (@baizhenyu), adding you as reviewers. For actors this only reads your existing ate.actor.stats.memory.working_set and ate.actor.stats.sampled_actors (no new metrics); node/ateom/atelet come from cAdvisor.

@Nishanth29
Nishanth Kotla (Nishanth29) force-pushed the working-set branch 2 times, most recently from fbb8a69 to a89e3d9 Compare October 14, 2026 18:57
Comment thread benchmarking/locust/server_telemetry.py
@Nishanth29

Copy link
Copy Markdown
Contributor Author

Haowei Cai (Roy) (@roycaihw) Rebased onto main and updated ateom -> worker across the selectors, keys, tests, and docs. Ready for LGTM / approval.

@maxsmythe Max Smythe (maxsmythe) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@maxsmythe
Max Smythe (maxsmythe) added this pull request to the merge queue Oct 17, 2026
Merged via the queue into agent-substrate:main with commit 322239e Oct 17, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants