Skip to content

feat(scripts): release.sh builds and verifies the direct release - #529

Draft
tieubao wants to merge 7 commits into
developfrom
feat/release-script
Draft

tieubao wants to merge 7 commits into
developfrom
feat/release-script

Conversation

@tieubao

@tieubao tieubao commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Proof of done: scripts/release.sh

scripts/release.sh turns the hand-run direct release (about 26 checks) into one command that builds and verifies the artifacts and writes checks.md. It never tags, uploads or publishes. Head 5cbadef on origin/develop 05aecea.

Check Command Result
Offline suite bash -n scripts/release.sh && bash scripts/test-release.sh 60 passed, 0 failed, 0 skipped (exit 0)
Read-only guards bash scripts/release.sh --guard-selftest 15 writes refused (exit 2 each), 6 reads allowed; exit 0
Developer ID export (1.12.0) bash scripts/release.sh --check-app "<scratch>/build-1.12.0/export/Hidden Bar.app" 7 rows pass, 11 code parts Developer ID + timestamp + runtime; exit 0
Unzipped notarized zip (1.12.0) --check-app "<scratch>/build-1.12.0/unzip/Hidden Bar.app" 7 rows pass; exit 0
Ad-hoc sandboxed copy --check-app "<scratch>/spec-ground/sbx/Hidden Bar.app" identity FAIL and sandbox FAIL; exit 1
Helper vs the 1.12.0 hand run item-exact-output, appcast-prior-verbatim the item and the merged feed equal the hand-run appcast.xml byte for byte
Pins vs the shipped plist pin-feed-url-matches-plist, pin-pubkey-matches-plist PINNED_FEED_URL and PINNED_PUBKEY equal SUFeedURL and SUPublicEDKey in hidden/Info-Direct.plist (consolelabs/hidden)
Orchestration through main main-check-app-failure-exits-1, main-all-stages-pass-exits-0 build, notarize, zip, sign, appcast and smoke stubbed: a failing check_app exits 1 with a FAILED row and runs no later stage; the same harness exits 0 when it passes
Prior-zip branch of the appcast stage pubkey-differs-from-prior-fails, pubkey-equals-prior-passes a stubbed curl serves a prior zip; a prior app with another key fails the "Prior app key" row, the pinned key passes it
App regressions bash scripts/test.sh --dd <fresh dir> Hidden Bar 674 passed 0 failed; App Store 440 passed 0 failed (head 5cbadef); exit 0
Negative controls six mutations, see below each red under mutation, green after restore; Verdict PASS

Green run

Command: bash -n scripts/release.sh && bash scripts/test-release.sh
Exit: 0
Output:
PASS verify-valid
PASS verify-tampered-append
PASS item-exact-output
PASS appcast-prior-verbatim
PASS appcast-cdata-item-text
PASS readback-wrong-url-fails
PASS cut-cmp-detects-altered-prior
PASS check-app-adhoc-unsandboxed-fails
PASS check-app-devid-sandboxed-fails
PASS pin-feed-url-matches-plist
PASS pin-pubkey-matches-plist
PASS pubkey-differs-from-prior-fails
PASS main-check-app-failure-exits-1
PASS notarize-log-unreadable-fails
PASS report-shape
(excerpt: 15 of the 60 PASS lines; the full list is in scripts/test-release.sh)
test-release: 60 passed, 0 failed, 0 skipped
Verdict: PASS
Command: bash scripts/test.sh --dd <fresh dir>
Exit: 0
Output:
selection: full suite   (parallel testing: YES, schemes: both, parallel)
scheme                    build_s   test_s  total_s  passed  failed  result
Hidden Bar                   17.3      8.9     26.2     674       0  ok
Hidden Bar App Store         15.8      3.2     19.0     440       0  ok
Verdict: PASS
Command: bash scripts/release.sh --check-app "<scratch>/spec-ground/sbx/Hidden Bar.app"
Exit: 1
Output:
release.sh: [1] Signature: identity, timestamp, runtime: FAIL: 11 parts checked; app (not Developer ID W777S7V8TN) Contents/MacOS/Hidden Bar (not Developer ID W777S7V8TN)
release.sh: [2] Unsandboxed: FAIL: app-sandbox entitlement present
check-app: FAIL (spec-ground/sbx/Hidden Bar.app)
Verdict: PASS (expected FAIL)

Negative control

Each block is negctl.sh "$PWD" "bash scripts/test-release.sh" "<mutation>" at head 5cbadef. Every mutation was restored by negctl (git checkout HEAD -- <file>), exit 0 after restore, and git status --porcelain showed no tracked change after each run.

# Mutation Cases red under mutation Counts under mutation Verdict
1 verifier in release-helper.swift made to always say VALID verify-tampered-append, verify-tampered-flip, verify-wrong-key, pubkey-equals-prior-passes 56 passed, 4 failed PASS
2 identity gate removed from check_identity check-app-adhoc-unsandboxed-fails, check-app-adhoc-sandboxed-fails, check-app-adhoc-empty-dict-not-credited, check-app-devid-sandboxed-fails, main-check-app-failure-exits-1 55 passed, 5 failed PASS
3 PINNED_FEED_URL reverted to dwarvesf/hidden (the pre-fix pin) pin-feed-url-matches-plist, check-app-fixture-rows-match-pins 58 passed, 2 failed PASS
4 check_app ... || finish 1 in main turned into || true main-check-app-failure-exits-1 59 passed, 1 failed PASS
5 appcast written straight to appcast.xml instead of appcast.xml.unverified pubkey-differs-from-prior-fails 59 passed, 1 failed PASS
6 prior-zip check_pubkey call replaced by a constant pubkey-differs-from-prior-fails, pubkey-equals-prior-passes 58 passed, 2 failed PASS

Rows 1 and 2 are the two controls recorded before the review fixes (earlier counts 3 and 4 red; the new cases add one red each). Row 3 is the drift control for the feed pin.

Review fixes in this branch

  1. Feed and repo pinned to consolelabs/hidden, matching hidden/Info-Direct.plist; the prior-item allowlist accepts both consolelabs and dwarvesf URLs because older published items still use dwarvesf.
  2. Drift tests read the real plist; the fixture bundles carry the plist values instead of the pins.
  3. One stubbed run through main, and the prior-zip branch of the appcast stage exercised with a stubbed curl.
  4. Hardening: an unreadable notary log fails the row; appcast.xml.unverified is renamed only after every check; the branch lookup runs from the repo root; a prior item's shortVersionString must match [0-9A-Za-z._-]+; the unused sums local is gone.

Deviation: the branch lookup uses (cd "$ROOT" && git rev-parse ...), not git -C "$ROOT", because the read-only git guard refuses -C on purpose (git -C . push is one of its self-test refusals).

Not proven

  • The live run (spec TASK-007, human only): real notarization, the real ASC and EdDSA keys, sign_update, the prior-zip downloads, and the smoke test with host restore. Those stages are covered offline by stubs (notarize-*, secrets-cleanup-on-failure, smoke-assert-*, the stubbed main run) and by reading the code against the 1.12.0 hand run, not by a run. Before the next release, run scripts/release.sh --notes <file> once on the Mini and compare its checks.md with the 1.12.0 hand-run checks.
  • The success path of the appcast stage (appcast-named-only-after-checks) stubs release-helper verify for the prior enclosure, since the fixtures hold no private key to sign a prior zip; real signature checks are covered by the verify cases and by the live run.
  • --check-app was run against the notarized 1.12.0 export; no new archive was built here.
  • The test suites never launch, open, or exec an app bundle; the smoke stage's launch path runs only in a live run.

…facts

One command for the GitHub (Sparkle) release: preflight, tests, archive and
export, app checks, notarize and staple, zips, EdDSA sign, appcast, smoke
test with host restore, and a checks.md with one row per check. It never
tags, uploads or publishes; gh and git are wrapped to read verbs only.

- release-helper.swift: Ed25519 verify (CryptoKit), the appcast item, and
  a merge that keeps every prior item byte-identical.
- --check-app runs every app row; the sandbox row counts only after the
  Developer ID identity row passes, so an ad-hoc build cannot pass vacuously.
- SUPublicEDKey, SUFeedURL and the Sparkle tarball digest are pinned.
- test-release.sh: offline cases with fixtures and PATH stubs.
- RUNBOOK: step 3 points at the script; the manual sandbox check now checks
  the Developer ID signature first, then the entitlement.
- QUALITY.md: Q-REL-1 (no key material) and Q-REL-2 (prior items verbatim).
The smoke positive control ran a re-signed /bin/sleep copy inside a fake
"Hidden Bar.app". macOS killed it with a launch constraint violation and
showed a "Hidden Bar is damaged" dialog. The control now runs /bin/sleep in
place; smoke_assert_process takes the expected executable as an optional
fourth argument, so the check is the same without launching a bundle.
Info-Direct.plist on develop points SUFeedURL at consolelabs/hidden, so release.sh pinned the old org and would fail its own Feed URL row. Prior appcast items that still use dwarvesf URLs stay accepted.
The pinned feed URL and Sparkle key are asserted against the real plist, and the fixture bundles now carry the plist values instead of the pins, so a drifted pin can no longer hide itself.
main() runs with build, notarize, zip, sign, appcast and smoke stages stubbed: a failing check_app must exit 1 with a FAILED row and stop the later stages, and the same harness must exit 0 when it passes. The prior-app key check now runs inside stage_appcast with a stubbed curl serving a prior zip; the unit check on check_pubkey is renamed to what it tests.
An unreadable notarytool log now fails the row instead of reading as 'issues none'. appcast.xml is written as appcast.xml.unverified and renamed only after every appcast check passed. The branch lookup runs from the repo root (via cd, since the read-only git guard refuses -C). A prior item's shortVersionString must match [0-9A-Za-z._-]+ before it names a file. The unused sums local is gone.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant