Skip to content

test(playground): smoke-test the server plugins and showcase the testing kit - #542

Draft
IamGalymzhan wants to merge 2 commits into
feat/testing-kit-helpersfrom
feat/dev-playground-smoke-tests
Draft

test(playground): smoke-test the server plugins and showcase the testing kit#542
IamGalymzhan wants to merge 2 commits into
feat/testing-kit-helpersfrom
feat/dev-playground-smoke-tests

Conversation

@IamGalymzhan

@IamGalymzhan IamGalymzhan commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Stack

Each PR targets the one above it, so the diff shown here is only the delta on top of #555. Rebased onto the current #555 tip, so the helpers it adds (useTestApp, the kit-faked workspace client) are available here.


Summary

The playground's server plugins had no server-side tests. apps/dev-playground/tests/ holds Playwright specs that intercept /api at the browser boundary (page.route + route.fulfill), so the Express server never runs in them — route handlers, execute() interceptors, and executeStream are unexercised.

Two suites cover that side with the harness from #540 — real HTTP, faked data plane, no workspace, no credentials, no network — and dogfood the kit the way the docs tell customers to test their own plugins.

smoke.test.ts — the plugins boot and answer

Test Proves
all three boot together the plugin set composes and registers under its manifest names
GET /api/reconnect the plain JSON route answers
reconnect stream opens as text/event-stream and emits its first message
POST /api/telemetry-examples/combined the whole nested-span body runs against the real TelemetryProvider and threads userId through
lakebase-examples unconfigured degrades to no routes without PGHOST/LAKEBASE_ENDPOINT, and the app still boots

testing-kit.integration.test.ts — a worked example of each recommended helper

One describe per helper, driven against a real plugin where the seam is clean:

Helper Demonstrated via
getMock (+ responses) files metadata route → files.getMetadata, asserted on app.client
createMockWorkspaceClient declared paths answer; undeclared resolve undefined
createApiError a genuine ApiError, and seeded as a route failure
useTestApp a fresh reconnect app per test
withEnv set/restore env around a block
useTestCache / resetTestCache the real CacheManager — stable & per-user keys, hit-then-miss
createTestPlugin the genie factory → a configured instance
createTestPluginContext (+ createMockRequest) cross-plugin tool dispatch, asUser, token-less reject
expectStream (+ createMockResponse) toEmit / toEmitExactly on a bounded source
useServiceContextMock the spied data-plane singleton

Where no playground plugin fits hermetically, the demo mirrors the docs: expectStream on a bounded generator (reconnect's stream is long-lived, so smoke.test.ts reads it by hand), and tool dispatch straight through the context (agents needs a live model).

Adds a dev-playground vitest project so pnpm test and CI's Unit Tests job pick both suites up. The tests/** exclusion is load-bearing — without it vitest's default glob collects the Playwright specs, which fail on import.


This pull request and its description were written by Isaac.

@IamGalymzhan
IamGalymzhan requested a review from a team as a code owner August 20, 2026 09:51
@IamGalymzhan
IamGalymzhan requested review from pkosiec and removed request for a team August 20, 2026 09:51
@IamGalymzhan
IamGalymzhan marked this pull request as draft August 20, 2026 10:06
@IamGalymzhan
IamGalymzhan force-pushed the feat/dev-playground-smoke-tests branch from f909a49 to 868331a Compare August 27, 2026 12:25
@IamGalymzhan
IamGalymzhan changed the base branch from feat/testing-kit-harness to feat/testing-kit-helpers August 27, 2026 12:25
@IamGalymzhan
IamGalymzhan marked this pull request as ready for review August 27, 2026 12:29
@IamGalymzhan
IamGalymzhan force-pushed the feat/dev-playground-smoke-tests branch 2 times, most recently from 5919db4 to 705c30f Compare August 28, 2026 11:35
The playground's server plugins had no tests at all. `tests/` holds Playwright
specs that intercept `/api` at the browser boundary (`page.route` + `fulfill`),
so the Express server never runs in them — route handlers, `execute()`
interceptors, and `executeStream` are all unexercised. These five tests cover
that side using the harness from #540: real HTTP, faked data plane, no
workspace, no credentials, no network.

Adds a `dev-playground` vitest project so `pnpm test` (and CI's Unit Tests job)
picks them up: 4517 -> 4522. The `tests/**` exclusion is load-bearing — without
it vitest's default `**/*.spec.ts` glob collects the Playwright specs and they
fail on import with "Playwright Test did not expect test.describe() to be
called here".

Two things worth knowing, both found by writing this:

- `telemetry-example-plugin` really calls `fetch("https://example.com")` in its
  external-api span, so a naive test needs the internet. The suite stubs
  non-loopback fetches and passes loopback through, since that is how the
  harness reaches its own server. Verified hermetic under a socket guard that
  throws on any non-loopback connect.
- `expectStream` buffers a source to completion and throws on timeout rather
  than returning partial events, so it cannot assert on a long-lived stream.
  The reconnect stream is five messages three seconds apart, so the SSE test
  reads one payload and hangs up instead of costing ~12s.

Scoped to smoke coverage deliberately. Not included: reconnection replay
(`executeStream`'s ring buffer and `Last-Event-ID` handling is the real prize
and deserves its own tests), and a `typecheck` script for the app — the
existing tsconfig reports 2128 errors from the client's unset `--jsx`, and a
server-scoped one still has 13 pre-existing errors that CI has never seen
because `pnpm -r typecheck` skips this workspace.

Signed-off-by: Galymzhan <zhangazy2004@gmail.com>
@IamGalymzhan
IamGalymzhan force-pushed the feat/dev-playground-smoke-tests branch from 705c30f to 7df788e Compare August 31, 2026 08:20
@IamGalymzhan IamGalymzhan changed the title test(playground): smoke-test the playground's own server plugins test(playground): smoke-test the server plugins and showcase the testing kit Aug 31, 2026
…ugins

smoke.test.ts proves the playground's plugins boot and answer; this adds a
worked example of each recommended `@databricks/appkit/testing` helper, so the
reference app dogfoods the kit the way the docs tell customers to test their
own plugins. One describe per helper: getMock, createMockWorkspaceClient,
createApiError, useTestApp, withEnv, useTestCache/resetTestCache,
createTestPlugin, createTestPluginContext, expectStream, and
useServiceContextMock.

Driven against real plugins where the seam is clean: files.getMetadata for
getMock and the createApiError error path, reconnect for useTestApp, genie for
createTestPlugin. Where no playground plugin fits hermetically the demo mirrors
the docs — expectStream on a bounded generator (the reconnect stream is
long-lived, so smoke.test.ts reads it by hand), and cross-plugin tool dispatch
straight through the context (agents needs a live model).

Signed-off-by: Galymzhan <zhangazy2004@gmail.com>
@IamGalymzhan
IamGalymzhan force-pushed the feat/dev-playground-smoke-tests branch from 7df788e to 5f35ad8 Compare August 31, 2026 08:26
@IamGalymzhan
IamGalymzhan marked this pull request as draft August 31, 2026 09:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant