Skip to content

Timer reach metric: attach audience_size to timer_started events (#107) - #138

Merged
ShuhaoZQGG merged 6 commits into
devfrom
factory/107-timer-reach-audience
Nov 12, 2026
Merged

ShuhaoZQGG merged 6 commits into
devfrom
factory/107-timer-reach-audience

Conversation

@claude

@claude claude Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Closes #107

Why

We only count timer starts, so a class of 20 where the teacher starts the timer 15 times looks the same as 15 solo web starts. The traffic report already has a "Timer reach" line (sum of per-meeting max audience_size per day) waiting for the client to send a head count.

What changes

  • In the Zoom app, timer_started now carries audience_size: the meeting's participant count read at START (so people who joined after the app opened are counted).
  • The property is left out (not zeroed) when Zoom withholds the list (non-host/co-host), there is no SDK, or the read fails. Analytics treats missing as a reach of 1.
  • Only the number leaves the device, never names.
  • The timer starts at once; the event is sent after the participant read, inside the existing idle callback.
  • Web app unchanged (test pins current behaviour). Spec timer-core gains TC-11/TC-12, and this run fixed two out-of-date steps (TC-02, TC-06).

Security

None. One integer added to an existing analytics event; no names or ids.

Production impact

After release, master users' timer_started events carry audience_size for hosts. No flags, no migration. Rollback is a revert. A very quick STOP right after START can now be captured before timer_started is sent (analytics order only).

Testing

  • npm test: 2225 pass (verify station, at 2e965b6); CI test pass on 53585cf
  • npm run test:e2e: 41 pass (verify station)
  • npm run test:specs: valid (7 specs)
  • Oracle: .factory/oracles/107.json — apps/zoom-app/src/components/LiveTab.test.jsx AC-1…AC-5, apps/web/src/components/LiveTab.test.jsx AC-4

Evidence (real Zoom client)

Run 2026-11-11 16:54–17:02 EST, Zoom 7.2.5 on macOS 26.7.2, dev version a4c66603 (this branch @ 2e965b6, committed vars). Spec timer-core. Deployed chunk LiveTab-C3jtJnUn.js contains audience_size.

  • TC-11 ✅ PostHog timer_started from zoom.timer-dev.simple-tech.app, host-only meeting (Participants: 1): 21:55:30, 21:57:27 and 22:00:58 UTC all have audience_size: 1 plus speaker_name, role, timing_rules, zoom_meeting_id. The same meeting's START on the previous build (21:46:04) has no audience_size. Timer read 00:04 at +4 s, so START was not held up. CONTINUE sent no extra timer_started.
  • TC-01 ✅ Refresh app loaded from the zoom. host (GET / 200, POST /api/zoom/session 200).
  • TC-03 ✅ real video green by 0:36, yellow by 0:52, red by 1:06 (ChairToast1, 30/45/60 s).
  • TC-04 ✅ FINISH resets the video. TC-05 ✅ 9 saved backgrounds before and after.
  • TC-06 ✅ ⋯ → Close current app, reopen: reload, toast "Timer resumed from where it was", 01:25 red.
  • TC-07 ✅ paused 01:40 survives close ("Paused timer restored"); CONTINUE carries on.
  • TC-08 ✅ after Reset and after Finish, reopen shows 00:00 and no toast.
  • TC-10 ✅ with Standard Speech hidden, the picker starts on the first role and START runs; timer_started carries that role.
  • TC-02 ⚠️ count-up on video ✅; the spec's "background turns green on start" was out of date (no colour until the green threshold). Spec fixed in 53585cf.

Not verified

  • TC-12 (human): a non-host organizer's START omits audience_size. Needs a second Zoom account; covered by LiveTab.test.jsx (restricted list).
  • A second participant / late joiner: host-only meeting, so audience_size was 1. The AC-2 unit test covers a count read after START.
  • TC-09 (human): hour-old saved session.
  • timer-sounds: not run; sounds are locked on this account (dev gate on, flag off) and this PR only added a pointer in its Not covered list.
  • Debug-log setVirtualBackground counts, card_color_applied, localStorage: Zoom's webview inspector is off on this Mac.
  • The test device hides Table Topics/Standard Speech, so TC-02/03/10 ran on a custom 30/45/60 s role.

Look here first

… events

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@claude claude Bot added the plan:review Factory: plan is up; answer its open questions, then add plan:approved label Nov 11, 2026
@claude

claude Bot commented Nov 11, 2026

Copy link
Copy Markdown
Author

plan — The participant count is read at START, not when the app opens. In LiveTab.jsx handleStart, the idle callback that sends timer_started now calls getZoomParticipants() first. It adds audience_size: participants.length only when the list isn't restricted and isn't empty, so the property is left out for non-hosts and when there's no Zoom SDK (including the web timer). If the fetch fails, the event is still sent, just without the property. startTimer() doesn't change, so START never waits on Zoom. I'm not using useZoomParticipants here because it fetches once at mount and would miss anyone who joins later. Tests: unit tests in apps/zoom-app/src/components/LiveTab.test.jsx (plus a guard in the web LiveTab.test.jsx), and new steps TC-11 (auto, host) and TC-12 (human, non-host account) in e2e/zoom-client/timer-core.md. Risk is low: the change only affects analytics and touches no riskHigh paths.

Open questions (defaults apply if unanswered):

  • Q1: Keep the six ACs. AC-2 (count at START) comes from your comment. AC-5 (a failed fetch still sends the event) is my proposal. AC-6 (the PostHog insight) can only be checked after a production release.
  • Q2: Does a participant count need a privacy-policy or Marketplace data-use update? Default: no change in this PR; I'll flag it in "Not verified".
  • Q3: The dev test meeting usually has only the host, so TC-11 will see audience_size: 1. Default: accept that as proof the property is attached; AC-2 is proven by the unit test.
  • Q4: Each time a non-host presses START, one warning ("Participant list needs host or co-host") goes to the debug log. Default: accept it, with no cache and no role pre-check.

@ShuhaoZQGG

Copy link
Copy Markdown
Contributor

Plan approved with the defaults for Q1–Q4: keep all six acceptance criteria; no privacy-policy change in this PR (list it under "Not verified"); accept audience_size: 1 from a host-only test meeting; accept one debug-log warning per START for non-hosts.

@ShuhaoZQGG ShuhaoZQGG added the plan:approved Factory: plan approved; oracle, implement and verify run label Nov 11, 2026
@claude claude Bot removed the plan:review Factory: plan is up; answer its open questions, then add plan:approved label Nov 11, 2026
Failing unit tests for audience_size on timer_started (Zoom app LiveTab),
a web-timer guard for AC-4, and timer-core TC-11/TC-12 (status: stale).
AC-6 is post-release and not testable here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@claude

claude Bot commented Nov 11, 2026

Copy link
Copy Markdown
Author

oracle — Commit 5666a11 adds failing tests for AC-1 to AC-5 and records them in .factory/oracles/107.json. They are in a new describe('timer_started and the meeting audience (#107)') block in apps/zoom-app/src/components/LiveTab.test.jsx. Each test opens the app, waits for the mount-time participant read from SpeakerInput to finish, then clears it, so every check is about the read made at START.

  • AC-1 (host, 3 participants): expects timer_started to equal {speaker_name, role, timing_rules, audience_size: 3}. Fails now because audience_size is missing.
  • AC-2 (1 participant when the app opened, 4 at START): expects a getZoomParticipants call after the press and audience_size: 4. Fails now because nothing reads participants at START.
  • AC-3 (restricted: true), AC-4 (empty list, no SDK) and AC-5 (getZoomParticipants rejects): each expects a read at START and a timer_started without audience_size. AC-5 also checks that FINISH is showing right after the click, before any idle work runs. All three fail now on the same missing read at START.
  • AC-4, web timer: apps/web/src/components/LiveTab.test.jsx gets a new test, "sends timer_started without audience_size". It passes now, as intended: the plan changes nothing on the web, so this only guards against a regression.

To check the tests themselves, I ran a ~10-line throwaway version of the planned change: all 13 tests in the file passed, and I reverted it. The rest of npm test passes (2220 tests).

The Zoom-client spec e2e/zoom-client/timer-core.md gets new steps TC-11 (auto, host: audience_size matches Zoom's Participants count; 1 is accepted in a host-only meeting) and TC-12 (human, a non-host account: no audience_size). The spec is now status: stale and its INDEX row is updated. npm run test:specs passes. AC-6 can only be checked after a production release, so it has no test.

github-actions Bot and others added 2 commits November 11, 2026 01:34
timer_started counted operator presses, not people served. The idle
callback that sends it now reads getZoomParticipants() at START (not at
mount, so late joiners count) and adds audience_size only when Zoom gave
the full list. Non-hosts, no SDK and a failed read send the event without
it; startTimer() still runs first and never waits on Zoom. Only a count
leaves the device.

Closes #107

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…audience_size

No step changes: timer-core's TC-11/TC-12 (added with the oracle) already
describe audience_size on timer_started, and no timer-sounds step looks at
the event. Note under "Not covered" that a failed participant read is
unit-tested only, and point timer-sounds at timer-core for the event.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@claude

claude Bot commented Nov 11, 2026

Copy link
Copy Markdown
Author

implement — Commit ca851eb changes handleStart in apps/zoom-app/src/components/LiveTab.jsx. startTimer() still runs first and is unchanged. The idle callback that sends timer_started now calls getZoomParticipants() at START, so people who joined after the app opened are counted. It adds audience_size: participants.length only when the list isn't restricted and isn't empty. A failed read is caught, and the event is sent without the property. The other properties are the same as before. Nothing changes on the web timer. The oracle tests pass (13 in the Zoom app's LiveTab.test.jsx, 18 in the web one), npm test passes (2225 tests) and npm run test:specs passes. The pre-push drift check flagged timer-core.md and timer-sounds.md, which both cover LiveTab.jsx. I reviewed both against the change and committed the review as cac0180. No steps changed: TC-11/TC-12 already describe this behaviour, and I added two "Not covered" notes. One says a failed participant read is only unit-tested. The other points timer-sounds to timer-core for this event. Neither spec has been run in the real Zoom client yet: timer-core stays status: stale, and that run is the Zoom station's job.

The spec has never run (verified_at: never); 'stale' is for a blocked
run, not an edit. The Zoom station sets the status after its run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@claude

claude Bot commented Nov 11, 2026

Copy link
Copy Markdown
Author

verify — all checks pass. Handing off to the Zoom station.

Checks (at 2e965b6)

  • npm test: 113 files, 2225 tests passed
  • npm run test:e2e: 41 passed
  • npm run test:specs: 7 specs valid
  • node scripts/factory/check-oracle.mjs: 1 oracle unchanged since its oracle commit
  • npm run build: OK. The build's sitemap.xml lastmod rewrite was discarded because it is not part of this change.
  • CI test: pass (3m32s)

Review (fresh context; inputs were the plan, the diff against origin/dev and the rubric)

  • Every AC has a test that fails without the change. Read against the origin/dev LiveTab.jsx:
    • AC-1 uses an exact toEqual that includes audience_size: 3.
    • AC-2 needs a read after the press with count 4, which rules out a count taken at mount.
    • AC-3, AC-4 (Zoom app) and AC-5 assert getZoomParticipants is called after START.
    • AC-5 also checks that START stays synchronous.
    • The web AC-4 test only pins down current behaviour, which the plan accepts.
  • No test gaming: no skip, only or todo, no weakened assertions, no snapshots, no test-only branches.
  • Scope: only the planned files changed. No debug leftovers, no secrets, no .env reads.
  • Specs: timer-core TC-01…TC-10 are unchanged and TC-11/TC-12 were appended. verified_at is untouched.
  • Fixed (attempt 1/3), 2e965b6: timer-core status had gone from draft to stale with no run. It has never run, and CLAUDE.md uses stale for a blocked run, so it is back to draft in the spec and in INDEX.md.
  • Not changed (non-blocking):
    • timer_started is now sent after the participant read, so a very quick STOP can be captured before it. This is analytics-only and in line with the plan's risk note.
    • The .catch fallback sets restricted: true rather than a neutral value. It has no behavioural effect.

Risk: no riskHigh paths touched, so no risk:high.

Next: needs:zoom-test. Run timer-core TC-11 (auto) and TC-12 (human) in the real Zoom client. timer-sounds covers LiveTab.jsx but has no steps that involve this event.

@claude claude Bot added needs:zoom-test Factory: waiting for the real-Zoom station and removed plan:approved Factory: plan approved; oracle, implement and verify run labels Nov 11, 2026
Real Zoom run 2026-11-11: TC-01, TC-03 to TC-08, TC-10, TC-11 pass;
TC-09 and TC-12 are human and were skipped. Fix two steps the run
showed out of date: TC-02 (no colour before the green threshold) and
TC-06 (close with ⋯ → Close current app; the panel X only hides it).
@ShuhaoZQGG ShuhaoZQGG added ready-for-review Factory: report written; final review and merge and removed needs:zoom-test Factory: waiting for the real-Zoom station labels Nov 12, 2026
@ShuhaoZQGG
ShuhaoZQGG marked this pull request as ready for review November 12, 2026 01:15
@ShuhaoZQGG
ShuhaoZQGG merged commit ba66363 into dev Nov 12, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-review Factory: report written; final review and merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant