Repository navigation
ci(competitive-matrix): land the feature-watch report through a bot-merged PR - #583
Merged
Merged
Conversation
…erged PR The weekly report PR was opened with the workflow token, so no CI ran on it and it waited for a human push. A run whose only change is scripts/competitive-watch-state.json now lands the way the adoption wall does: copilotkit-bot (app 1108748, DEVOPS_BOT_PRIVATE_KEY) opens a bot/competitive-watch-* PR and merges it with --admin --merge. Any docs/ change still goes to a review PR. The report stays visible in the PR body, the job summary and the Slack digest, which now says the bot merged it.
commit: |
This was referenced Oct 26, 2026
jpr5
added a commit
that referenced
this pull request
Oct 26, 2026
) ## What Follow-up to #583 (and the same bug #581 had). When a refresh needs a human, the review PR was still pushed and opened with `GITHUB_TOKEN`. GitHub does not start workflows from `GITHUB_TOKEN` events, so CI never ran on that PR. Before, in `update-competitive-matrix.yml`: ```yaml - name: Configure git credentials git config --local url."https://x-access-token:${{ secrets.GITHUB_TOKEN }}@github.com/".insteadOf "https://github.com/" - name: Create PR env: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} ``` `update-adoption-wall.yml` had the same gap: ```yaml - name: Open review PR env: GH_TOKEN: ${{ github.token }} ``` ## Change - Both review paths push and open their PR with the copilotkit-bot App token. The token comes from the existing "Mint app token" step (same app, same pinned action, same secret, same `contents`/`pull-requests` write scopes). No new secret. - The review PR is **never merged** by the workflow. It waits for a human, as before. - The mint condition is now the union of the two steps that use the token: - competitive matrix: `changed == 'true'` - adoption wall: `exists && changed && (status == 'safe' || status == 'needs-review')` - Least privilege for the workflow token: `pull-requests: write` is gone from both jobs. The competitive-matrix job is now `contents: read`, because its workflow token only reads the API now. - The "Configure git credentials" step is removed. It wrote `GITHUB_TOKEN` into `.git/config`. The review step now passes the App token through git's environment config, like the auto-land step. ## `if:` trace Competitive matrix: | case | mint | auto-land | review PR | |---|---|---|---| | no change | skip | skip | skip | | docs/ changed (confine skipped) | run | skip (`docs_changed`) | run (`confined != 'true'`) | | state only, confined | run | run | skip | | state only, not confined | run | skip | run | Adoption wall: auto-land (`exists && safe && changed && confined`) and review (`exists && changed && (needs-review || (safe && !confined))`) are both subsets of the mint condition. A clean run, a run with no change, or a run with no state branch mints nothing. ## Red-green proof Contract test added to `src/__tests__/competitive-matrix-slack.test.ts` (`review PRs are opened with the App token`), for both workflows. It checks that the review step uses `steps.app-token.outputs.token` for push and PR, that no workflow-token git credential remains, that the review step never merges, and that the mint condition covers the review path. RED (new test against the `origin/main` workflows): ``` ⎯⎯⎯⎯⎯⎯⎯ Failed Tests 4 ⎯⎯⎯⎯⎯⎯⎯ FAIL src/__tests__/competitive-matrix-slack.test.ts > review PRs are opened with the App token > update-competitive-matrix.yml: "Create PR" pushes and opens the PR on the App token FAIL src/__tests__/competitive-matrix-slack.test.ts > review PRs are opened with the App token > update-competitive-matrix.yml: the App token is minted for the review path too FAIL src/__tests__/competitive-matrix-slack.test.ts > review PRs are opened with the App token > update-adoption-wall.yml: "Open review PR" pushes and opens the PR on the App token FAIL src/__tests__/competitive-matrix-slack.test.ts > review PRs are opened with the App token > update-adoption-wall.yml: the App token is minted for the review path too Tests 4 failed | 7 passed (11) ``` GREEN (same test against this branch): ``` Tests 11 passed (11) ``` Full suite: 483 files, 19670 tests passed. `actionlint` exit 0. `zizmor --offline` exit 0 (no findings). typecheck, eslint and prettier clean. ## End-to-end proof Not triggered by hand: mock-server.com currently returns 403, so a run would fail before these steps. The end-to-end proof is the next scheduled Monday run.
Merged
jpr5
added a commit
that referenced
this pull request
Oct 27, 2026
Release v1.47.0. Merging this PR publishes through `publish-release.yml`: npm, the git tag, the GitHub Release and Slack. ## Changelog ### Added - Fixture `citations` serve sources on Responses, Chat Completions, Claude and Gemini (#550, #578) - Fixture `files` serve generated images on Gemini and OpenAI Responses (#550, #578) - Raw passthrough fixtures send provider SSE events or JSON exactly as written (#580) - Opt-in `nonStreamingTimingAllProviders` delays non-streaming replies on every provider (#579) ### Fixed - Client disconnects stop and journal every LLM reply; Docker image on Node 22.23.4 (#579, #576) The detailed notes from `[Unreleased]` are condensed into these bullets. The full detail is in the linked PRs. The drift-harness and CI changes (#583–#587) do not change what users see, so they are left out. ## Docs (separate commit) - The fixtures page gets a new **Citations and generated files** section: the fixture shape, the wire shape for each provider, the error codes, and override-table and provider-matrix rows. The Responses, Chat Completions, Claude and Gemini pages link to it. - What's New has three 1.47.0 cards: citations and generated images, raw passthrough, and stream realism (disconnects and timing on every provider). The two 1.46.0 cards are merged into one. The GPT-Live card is removed because it shipped in 1.43.0, more than a month ago. - The README has new feature bullets for citations and files, raw passthrough and client disconnects. The Streaming Physics bullet now mentions `nonStreamingTimingAllProviders`. - The competitive matrix has three new hand-reviewed rows: provider-native citations, model-generated images, and stopping on client disconnect. Each competitor cell was checked against a shallow clone of that competitor's source on 2026-10-26, and an HTML comment on each row records the evidence. The rows are added to `HAND_ROWS`, so the weekly scan never rewrites them. The homepage ND count in the test is now 56. ## Version carriers (`git grep -F 1.46.0`) `package.json`, `charts/aimock/Chart.yaml` appVersion, `.claude-plugin/plugin.json`, `.claude-plugin/marketplace.json` (`^1.47.0`), the `docs/index.html` banner, the `packages/aimock-pytest/README.md` default, and `_version.py` `AIMOCK_VERSION`. These are not bumped: the CHANGELOG history, the `version-note` labels for 1.46.0 features, and `pyproject.toml`. ## Local verification - format:check, lint, typecheck, build, api:report with no `etc/` diff, test:exports, the docs and matrix tests, and the full `pnpm test`. - The citations and files wire shapes were exercised against the built server for Chat, Responses, Claude and Gemini, including the 500 for `files` on Chat Completions. - Screenshots of What's New, the matrix and the fixtures section at 375, 430 and 1440 px.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The weekly competitor feature-watch report PR is opened with the workflow
GITHUB_TOKEN. GitHub does not start workflows from that token's events, so no CI runs on the PR and it waits for a human push. #581 is today's example.Change
A run whose only change is
scripts/competitive-watch-state.jsonnow lands the way the adoption wall does (60fe76c):Verify the auto-land diff is confined to the state file: auto-land only whengit status --porcelain -- docs/ scripts/competitive-watch-state.jsonis exactlyM scripts/competitive-watch-state.json.Mint app token: same action SHA, app1108748,secrets.DEVOPS_BOT_PRIVATE_KEY,contents/pull-requestswrite only, minted only on the auto-land path.Auto-land via bot PR: per-command auth throughGIT_CONFIG_*env,copilotkit-bot[bot]identity,bot/competitive-watch-*branch, stages that one file,gh pr merge --merge --adminintoGITHUB_REF_NAME. If the merge fails, the PR stays open and the job fails.docs/change (homepage or migration pages) still goes to the existing review PR on the workflow token.SLACK_WEBHOOK) gets a "merged via a bot PR" line.HUSKY=0and--no-verifyon the bot commit, andcancel-in-progress: false.No new secrets. No new actions.
Red / green
RED, workflow:
gh pr checks 581returnsno checks reported on the 'auto/competitive-matrix-20261026-40297803406' branch(exit 1). The check-runs API givestotal_count: 0.RED, Slack contract test (
src/__tests__/competitive-matrix-slack.test.ts, which runs the realNotify Slackstep) against the old workflow:GREEN, same test against this workflow:
Workflow and competitive tests: 24 files, 1410 passed.
Lint:
actionlintexit 0.zizmor1.26.1 with.github/zizmor.yml, offline and online: exit 0, no findings.The end-to-end check is a
workflow_dispatchrun after this merges.