feat: testplane profiler by KuznetsovRoman · Pull Request #1314 · gemini-testing/testplane

KuznetsovRoman · 2026-08-18T15:09:58Z

No description provided.

pkg-pr-new · 2026-08-18T15:12:10Z

npm i https://pkg.pr.new/gemini-testing/testplane@1314

commit: 5f66820

github-actions · 2026-08-18T15:12:19Z

✅ Testplane E2E run succeed

Report

github-actions · 2026-08-18T15:12:35Z

✅ Testplane browser-env run succeed

Report

KuznetsovRoman · 2026-08-19T02:06:28Z

+    configuredSessionsPerBrowser?: Record<string, number>;
+}
+
+export interface ProfilerResultV1 {


This object is emitted as event and saved to file

KuznetsovRoman · 2026-08-19T02:08:46Z

+                collector: bucket,
+                seen: state.seen,
+                retained,
+                rule: `top ${RETENTION_POLICY_V1.operationLimits[bucket]} by wall time; keep ancestors of retained operations`,


Rules, reasons, evidence, confidence and such are saved as human-readable values so we could display clear strings in profiler UI with at least as possible logic

KuznetsovRoman · 2026-08-19T02:09:54Z

@@ -0,0 +1,49 @@
+import type { EnabledProfilerLevel } from "../schema";
+
+export const RETENTION_POLICY_V1 = Object.freeze({


Object configures "how much slowest objects of each type we should keep"

KuznetsovRoman · 2026-08-19T02:13:23Z

+    evicted?: T;
+}
+
+export class TopK<T> {


classic min-heap topK implementation on array
The least element is stored at the root

KuznetsovRoman · 2026-08-19T02:21:34Z

+    private _max = -Infinity;
+    private _mean = 0;
+    private _m2 = 0;
+    private readonly _reservoir: number[] = [];


Reservoir Sampling for percentile readings

KuznetsovRoman · 2026-08-19T03:46:04Z


                this._socket.on(BrowserEventNames.runBrowserCommand, this._handleRunBrowserCommand(browser));
+                this._socket.on(BrowserEventNames.profilerFragment, payload => {
+                    this._profiler.recordMeasurement("browser.runnable", payload.wallMs, {


Recording somewhat same values on browser env, as we do on node.js runtime, but it lacks:

detailed hook spans

test body

commands

event loop utilization/delay

CPU
We also have just one browser runnable here, which would just help to decide between "slow test" and "slow transport"

KuznetsovRoman · 2026-08-19T03:49:15Z

After testplane profiler finalization, process might already have error code, but then get to another process.exit, which messes up exit code.
In this module we calculate origin exit code

KuznetsovRoman · 2026-08-19T03:51:06Z

In "profile CLI commands" we wrap other CLI commands to same profiler flow

KuznetsovRoman · 2026-08-19T03:52:19Z

            }),
        }),

+        profiler: section({


Describe profiler root section

KuznetsovRoman · 2026-08-19T03:53:03Z

        input: ".testplane/failed.json",
    },
+    profiler: {
+        level: 0,


Disabled by default in order to not waste extra CPU cycles in CI

KuznetsovRoman · 2026-08-20T04:16:35Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 35bab54a27

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-08-20T04:25:17Z

+
+    private async _deliverResult(result: Readonly<ProfilerResultV1>): Promise<void> {
+        try {
+            printProfilerResult(result, this._console);


Keep profiler summaries off stdout for data commands

When profiler.level is enabled for data-producing CLI commands such as config, list-browsers, or list-tests, those commands still write their JSON/plain result to stdout inside the profiled action, and this unconditional finalization step appends the human-readable profiler report to the same stream. That makes otherwise successful command output invalid for consumers that parse stdout as JSON; route the profiler summary to stderr or suppress it for these commands unless explicitly requested.

Useful? React with 👍 / 👎.

DudaGod

I suggest adding info about new feature to docs

DudaGod · 2026-08-24T08:33:41Z

+                    );
+
+                    if (browsersInstallPerStatus[BrowserInstallStatus.Error].length) {
+                        process.exitCode = resolveExitCode(1);


looks like here callback ends successfully and return undefined in action inside ProfilerManager is this expected behavior? In this case runOutcome would be passed

Good catch, fixed in latest commit "fix: profile install dependency failures"

DudaGod · 2026-08-24T08:45:26Z

-        <script type="module" src="${MODULE_NAMES.mocha}"></script>
-        <script type="module" src="${MODULE_NAMES.browserRunner}"></script>
+        <script type="module" src="${viteFsUrl(modules.globals)}"></script>
+        <script src="${MOCHA_RUNTIME_URL}"></script>


why we move vite module loading with @testplane/mocha to script /__testplane__/mocha.js ?

There was a bug: Relative browser-env runtime URLs were resolved under /run-uuids/, causing Vite’s HTML fallback to return an error page instead of runtime JavaScript, which resulted in "Browser didn't connect to the Vite server in 10000ms"

The bug predates the profiler and was only discovered while browser runtime telemetry increased browser-env coverage.

Runtime ESM files now use rooted /@fs URLs, Mocha is served as a classic UMD script, route parsing is stricter, and the Testplane package root is allowed by Vite, which fixes the issue.

DudaGod · 2026-08-24T08:53:22Z

+    await fs.mkdir(directory, { recursive: true });
+    try {
+        await fs.writeFile(temporary, `${JSON.stringify(result, null, 2)}\n`, { encoding: "utf8", flag: "wx" });
+        await fs.rename(temporary, destination);


using public api we can call readTests() with configured profiler and then run method. Looks like in this case I will get only profiler with the last call. Is this intentionally behavior?

Canonical ProfilerResultV1 DTO and schema exports for internal consumers.

Internal profiler level and configuration types without enabling the public option.

Bounded top-K storage with parent-safe retention and focused tests.

Bounded streaming moments, reservoirs, percentiles, weighted merge, and focused tests.

Runtime interfaces, strict no-op path, clocks, async activity tracking, and startup probe.

Span lifecycle, ALS context, retention interaction, and child interval union with matching tests.

Aggregate and metric updates plus bounded resource sampling with matching tests.

Fragment deltas, runtime-owned cloning, hostile input validation, and matching tests.

Source-scoped ordering, deduplication, gaps, pending caps, and matching tests.

Synthetic root, parent repair, timing formulas, compact references, and their behavioral tests.

Versioned thresholds, robust outlier detection, and deterministic finding order.

Lifecycle, major-phase, discovery, and unattributed-time analyzers with matching tests.

File, module, and listener analyzers with their positive and negative tests.

Slow-test, hook, and browser-command analyzers with matching tests.

Worker, session, CPU, and event-loop resource analyzers with matching tests.

Descriptor-safe redaction and structural sanitization for every public delivery path.

Serialized timeline and whole-result budgets with explicit truncation fallbacks.

Optional JSON delivery through a unique temporary file and atomic rename.

Grouped ANSI-aware presentation emitted through one logger call with renderer tests.

One-result guard, finalization pipeline, immutable payload, and fail-open deliveries with tests.

Node-compatible listener observation while preserving registration, removal, and emission semantics.

Translate event observer callbacks into profiler envelopes and individual listener spans.

Initialize config observation and a dormant no-op-compatible manager.

Composable CommonJS and ESM module observation shared with selectivity, with regression tests.

Profile browser preparation and command execution with bounded safe attributes and focused tests.

Pool queue, limiter state, acquisition, release, and session lifecycle observation with tests.

Master attempts and worker hook, body, cleanup, and retry spans with execution tests.

Discovery, parsing, module scopes, cache waits, and module-graph collection with tests.

Process messages, master ingestion, flush ordering, late fragments, and registry tests.

Worker clock handshake, facade RPC, fragment batching, and worker-side tests.

Propagate the optional runtime through master and worker runner boundaries.

Bounded runnable, resource, and long-task telemetry relayed across the browser environment.

Add top-level operations, teardown, signals, and exit-code preservation.

Early CLI bootstrap and command wrappers with preserved success and failure exit semantics.

Enable public config, result delivery, exports, and end-to-end tests.

KuznetsovRoman commented Aug 19, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Aug 20, 2026

View reviewed changes

DudaGod approved these changes Aug 24, 2026

View reviewed changes

KuznetsovRoman added 23 commits August 25, 2026 03:49

feat(profiler): define versioned result schema

a7f3553

Canonical ProfilerResultV1 DTO and schema exports for internal consumers.

feat(profiler): define internal config contract

6a233d5

Internal profiler level and configuration types without enabling the public option.

feat(profiler): add bounded operation retention

1386440

Bounded top-K storage with parent-safe retention and focused tests.

feat(profiler): add streaming statistics

128bbe4

Bounded streaming moments, reservoirs, percentiles, weighted merge, and focused tests.

feat(profiler): add runtime foundations

698ab8b

Runtime interfaces, strict no-op path, clocks, async activity tracking, and startup probe.

feat(profiler): collect process-local spans

2b40387

Span lifecycle, ALS context, retention interaction, and child interval union with matching tests.

feat(profiler): collect runtime statistics

2c6fa59

Aggregate and metric updates plus bounded resource sampling with matching tests.

feat(profiler): produce validated fragments

685c479

Fragment deltas, runtime-owned cloning, hostile input validation, and matching tests.

feat(profiler): merge sequenced fragments

084035a

Source-scoped ordering, deduplication, gaps, pending caps, and matching tests.

feat(profiler): normalize profile timings

aeb04b3

Synthetic root, parent repair, timing formulas, compact references, and their behavioral tests.

feat(profiler): define analysis policy

a9a75d0

Versioned thresholds, robust outlier detection, and deterministic finding order.

feat(profiler): analyze lifecycle and discovery

524c824

Lifecycle, major-phase, discovery, and unattributed-time analyzers with matching tests.

feat(profiler): analyze files modules and listeners

84b17a1

File, module, and listener analyzers with their positive and negative tests.

feat(profiler): analyze tests hooks and commands

716e70b

Slow-test, hook, and browser-command analyzers with matching tests.

feat(profiler): analyze capacity and resources

83db3e9

Worker, session, CPU, and event-loop resource analyzers with matching tests.

feat(profiler): sanitize public results

cd424b1

Descriptor-safe redaction and structural sanitization for every public delivery path.

feat(profiler): add bounded result budgets

2c5859a

Serialized timeline and whole-result budgets with explicit truncation fallbacks.

feat(profiler): write atomic JSON output

b71bafe

Optional JSON delivery through a unique temporary file and atomic rename.

feat(profiler): render console report

a0a6cbe

Grouped ANSI-aware presentation emitted through one logger call with renderer tests.

feat(profiler): orchestrate profile lifecycle

793329f

One-result guard, finalization pipeline, immutable payload, and fail-open deliveries with tests.

feat(events): add fail-open listener observation

e7c0e4b

Node-compatible listener observation while preserving registration, removal, and emission semantics.

feat(profiler): record event listener spans

5e3a83d

Translate event observer callbacks into profiler envelopes and individual listener spans.

feat(profiler): connect dormant base lifecycle

c631a84

Initialize config observation and a dormant no-op-compatible manager.

KuznetsovRoman added 13 commits August 25, 2026 03:49

feat(modules): share observer registry

52dc2b6

Composable CommonJS and ESM module observation shared with selectivity, with regression tests.

feat(profiler): instrument browser commands

ce243a9

Profile browser preparation and command execution with bounded safe attributes and focused tests.

feat(profiler): observe browser pools and sessions

01a0bec

Pool queue, limiter state, acquisition, release, and session lifecycle observation with tests.

feat(profiler): instrument test execution

08add85

Master attempts and worker hook, body, cleanup, and retry spans with execution tests.

feat(profiler): instrument discovery and modules

76fe4bb

Discovery, parsing, module scopes, cache waits, and module-graph collection with tests.

feat(profiler): define worker transport protocol

327fb57

Process messages, master ingestion, flush ordering, late fragments, and registry tests.

feat(profiler): exchange worker fragments

40b7c64

Worker clock handshake, facade RPC, fragment batching, and worker-side tests.

feat(profiler): wire profiler through runners

25a8342

Propagate the optional runtime through master and worker runner boundaries.

feat(profiler): collect browser runtime telemetry

11f53a0

Bounded runnable, resource, and long-task telemetry relayed across the browser environment.

feat(profiler): finalize Testplane operations

98aee59

Add top-level operations, teardown, signals, and exit-code preservation.

feat(profiler): profile CLI commands

4e369df

Early CLI bootstrap and command wrappers with preserved success and failure exit semantics.

feat(profiler): activate public profiling

9590234

Enable public config, result delivery, exports, and end-to-end tests.

fix: profile install dependency failures

5f66820

KuznetsovRoman force-pushed the TESTPLANE-1087.profiler branch 2 times, most recently from 4fb24c4 to 5f66820 Compare August 25, 2026 01:20

KuznetsovRoman merged commit 875031c into master Aug 25, 2026
9 checks passed

KuznetsovRoman deleted the TESTPLANE-1087.profiler branch August 25, 2026 23:04

		@@ -0,0 +1,49 @@
		import type { EnabledProfilerLevel } from "../schema";

		export const RETENTION_POLICY_V1 = Object.freeze({

Uh oh!

Conversation

KuznetsovRoman commented Aug 18, 2026

Uh oh!

pkg-pr-new Bot commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Testplane E2E run succeedReport

Uh oh!

github-actions Bot commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Testplane browser-env run succeedReport

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

KuznetsovRoman commented Aug 20, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Aug 20, 2026

Choose a reason for hiding this comment

Uh oh!

DudaGod left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

pkg-pr-new Bot commented Aug 18, 2026 •

edited

Loading

github-actions Bot commented Aug 18, 2026 •

edited

Loading

✅ Testplane E2E run succeed

Report

github-actions Bot commented Aug 18, 2026 •

edited

Loading

✅ Testplane browser-env run succeed

Report