Skip to content

ci: [임시] main 기준 스크린샷 검증 baseline 확인 - #156

Closed
kez-lab wants to merge 1 commit into
mainfrom
ci/screenshot-linux-baseline
Closed

kez-lab wants to merge 1 commit into
mainfrom
ci/screenshot-linux-baseline

Conversation

@kez-lab

@kez-lab kez-lab commented Aug 23, 2026

Copy link
Copy Markdown
Owner

임시 PR입니다. 머지하지 않고 확인 후 닫습니다.

#155에서 Linux runner의 validateDebugScreenshotTest가 9개 전부 실패했습니다. 로컬(macOS arm64)에서는 clean 후 재검증까지 통과하므로, 원인이 #155의 변경인지 기준 이미지가 머신 종속인지 구분해야 합니다.

이 브랜치는 origin/main + 워크플로 파일 2개뿐입니다. picker 코드는 main 그대로입니다.

  • 여기서도 9개가 실패하면 → 기준 이미지가 macOS에서 기록돼 Linux에서 검증되지 않는 기존 문제이고, #155과 무관합니다.
  • 여기서 통과하면 → #155의 변경이 원인입니다.

🤖 Generated with Claude Code

Throwaway branch. Contains only the workflow files from
feature/api-surface-hardening on top of origin/main, so the Linux screenshot
validation runs against main's picker code. Used to determine whether the
9 screenshot failures on that PR come from its changes or from the reference
images being machine-specific.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Aug 23, 2026

Copy link
Copy Markdown

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: c574ad2f-3718-47cc-8597-143961f6e2d0

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

kez-lab added a commit that referenced this pull request Aug 23, 2026
The gate failed all 9 screenshot comparisons on the Linux runner. A baseline
run of the same workflow against unmodified main (PR #156) failed identically,
so the cause is the reference images, not any change in this branch.

Measured it rather than guessing:

| | differing pixels | per-pixel delta |
| --- | --- | --- |
| Same references, Linux runner vs recording host | 0.164%-0.888% | exactly 1/255 |
| One digit changed (LocalTime(13, 5) -> 13, 6), same host | 0.189% | up to 225/255 |

Host noise is 16-90x the configured 0.01% `imageDifferenceThreshold`, and
raising the threshold is not a fix: the setting counts differing pixels with no
per-pixel tolerance, so any value large enough to pass on another host (>=0.9%)
also masks a one-digit regression at 0.189%. What separates the two is how much
each pixel differs - 1 versus 225 - which the setting cannot express.

So the task stays where it already worked: locally on the recording machine and
in the manual workflow. Making it a hosted gate means picking a canonical
rendering environment and re-recording every reference there, which inverts the
problem for local runs; that trade-off is the maintainer's to make and is
recorded in docs/testing/compose-screenshot-tests.md, replacing the previous
claim that real regressions are "far larger" than host noise.

Also fixed the manual workflow's screenshot artifact, which uploaded the HTML
report without the images it links to, and tidied the 0.8.0 changelog section
into one block per heading.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@kez-lab

kez-lab commented Aug 23, 2026

Copy link
Copy Markdown
Owner Author

확인 완료. main 원본에서도 스크린샷 9개가 동일하게 실패해, #155의 변경과 무관한 기존 문제임이 확인됐습니다.

측정 결과:

다른 픽셀 픽셀당 차이
동일 기준 이미지, Linux runner vs 기록 머신 0.164%~0.888% 정확히 1/255
숫자 한 자리 변경(LocalTime(13, 5) → 13, 6), 동일 머신 0.189% 최대 225/255

호스트 노이즈가 설정된 imageDifferenceThreshold 0.01%의 16~90배입니다. 임계값을 올리는 것은 해결책이 아닙니다 — 이 설정은 픽셀당 허용 오차 없이 다른 픽셀 수만 세므로, 다른 호스트에서 통과할 만큼(≥0.9%) 올리면 0.189%인 실제 회귀도 함께 가려집니다.

#155에서 스크린샷 검증을 호스티드 PR 게이트에서 제외하고 docs/testing/compose-screenshot-tests.md에 근거를 기록했습니다.

@kez-lab kez-lab closed this Aug 23, 2026
@kez-lab
kez-lab deleted the ci/screenshot-linux-baseline branch August 23, 2026 13:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant