Skip to content

Fix analytics-review on workflow_dispatch: pass resolved PR context to the agent - #152

Merged
amjithtitus09 merged 2 commits into
mainfrom
amjithtitus09-analytics-review-bot
Aug 17, 2026
Merged

Fix analytics-review on workflow_dispatch: pass resolved PR context to the agent#152
amjithtitus09 merged 2 commits into
mainfrom
amjithtitus09-analytics-review-bot

Conversation

@amjithtitus09

@amjithtitus09 amjithtitus09 commented Aug 17, 2026

Copy link
Copy Markdown
Member

Follow-up to #148/#151. Dry-run 32015295640 (workflow_dispatch, PR #146) went all-green — the v0.86.2 recompile fixed the ENOENT — but no review was posted. Two bugs, both fixed here.

Bug 1 (blocking): resolved PR context never reached the agent

The "Resolve PR context" pre-step worked (AW_PR_NUMBER: 146, AW_HEAD_REF: ENG-909 in every pre-step's env), but those are per-step env: vars — they don't exist inside the agent sandbox, and on workflow_dispatch the event payload has no PR either. The agent's own words from the run log:

This run was triggered via workflow_dispatch with no associated pull request (no PR/issue number in context, no GITHUB_HEAD_REF, event payload has no PR). There's nothing for the analytics-review lenses to review.

…and it called noop. So the manual/backfill dispatch path could never work.

Fix — use the proven file channel (the agent already reads /tmp/gh-aw/context/):

  • The resolve step now writes /tmp/gh-aw/context/run-context.md (event name, PR number, head ref, head SHA — or an explicit NONE RESOLVED marker), and exposes proper step outputs (id: resolve_pr + $GITHUB_OUTPUT) for any future step-level consumers.
  • The workflow prompt now instructs the agent to determine the PR under review from run-context.md, never from the event payload, treats a dispatch run with a resolved PR as a normal first-review/re-review, and reserves noop for runs where run-context.md genuinely says no PR.

(Interpolating ${{ steps.resolve_pr.outputs.* }} into the prompt was not possible: gh-aw renders the prompt in the activation job, while custom steps run later in the agent job — the file channel is the correct mechanism.)

Bug 2: JIRA 404 for ENG-909 — scoped-token root cause + self-diagnosing markers + auto-recovery

Root cause identified: JIRA_API_TOKEN is an Atlassian API token with scopes (read:issue-details:jira, read:issue-field-values:jira, read:comment:jira). Per Atlassian's docs, scoped tokens must target a different base URL than unscoped ones:

token type required JIRA_BASE_URL form
unscoped https://<site>.atlassian.net
scoped (what we have) https://api.atlassian.com/ex/jira/<cloudId>

A scoped token sent to the *.atlassian.net form gets HTTP 404 — the same status JIRA uses for missing/unviewable issues — which fully explains the dry-run's HTTP 404 for ENG-909 with valid credentials.

Fixes in the JIRA pre-step:

  • Auto-recovery: on a failed issue fetch, probe GET https://api.atlassian.com/oauth/token/accessible-resources (fixed absolute URL, independent of JIRA_BASE_URL; returns each reachable site's url + id=cloudId). If a cloudId can be resolved (configured URL matches a site, cloudId embedded in an ex/jira URL, or single-site token), retry via https://api.atlassian.com/ex/jira/<cloudId> — so either token type now works with either JIRA_BASE_URL form. Subsequent comment fetches use the recovered base.
  • Self-diagnosing markers when it still fails, written into jira-ticket.md. All arms are worded as likely causes, never verdicts (a failed probe is not proof the credentials are bad — the endpoint may not accept basic auth for every token type), and report only HTTP codes, site URLs, cloudIds — never secret values:
    • probe 401/403 → inconclusive on its own; likely causes: JIRA_EMAIL doesn't match the token's account, or the token expired (scoped tokens expire ≤365 days).
    • probe 200, base matches no site → likely wrong base URL for the token type; lists the token's reachable sites + cloudIds and the exact value to set (https://api.atlassian.com/ex/jira/<cloudId>, no trailing slash, no /rest suffix).
    • probe 200, endpoint(s) correct → likely the token's account lacks access to the ENG project, or the ticket doesn't exist (JIRA 404s rather than 403s for unviewable issues).
    • probe unreachable/unexpected → explicitly inconclusive; enumerates the three real possibilities (wrong base URL for token type / no project access / nonexistent ticket).
    • Every inconclusive arm also surfaces the two-URL rule whenever the configured base host isn't api.atlassian.com (host shape only — the URL value is never printed).
  • The earlier /rest/api/3/myself probe is removed: it isn't callable with these granular scopes and would have false-reported valid credentials as broken.
  • Never-fail-the-job behaviour kept; JIRA secrets remain confined to this one pre-step.
  • New helper .github/scripts/jira_sites.py (cloudId resolution + site summary; always exits 0). README updated with the base-URL form guidance.

Admin action (recommended)

Set JIRA_BASE_URL to https://api.atlassian.com/ex/jira/<cloudId> for the scoped token. The auto-recovery makes the current value work anyway (single-site tokens recover unconditionally), and if anything is still off, the marker in the run log / review context now says exactly what to change.

Verification

All pre-steps were already green in the dry-run; the agent demonstrably reads /tmp/gh-aw/context/ (it cat'd jira-ticket.md and lint-report.md in its tool log) — only the PR identity was missing, which run-context.md now supplies. The new JIRA logic was exercised locally against a mock Atlassian server using the step script extracted verbatim from the workflow: recovery path renders the full ticket; all failure branches write their marker and exit 0.

After merge:

gh workflow run analytics-review.lock.yml -f pr_number=146

Expected: the agent reads run-context.md, identifies PR #146, and posts a review; jira-ticket.md contains either the rendered ENG-909 ticket (via auto-recovery) or a precise likely-cause diagnosis instead of a bare 404.

Compiled with gh-aw v0.86.2, 0 warnings.

amjithtitus09 and others added 2 commits August 17, 2026 15:10
…o the agent

Dry-run 32015295640 (PR #146) went green end-to-end after the v0.86.2
recompile, but the agent posted nothing: the resolve pre-step's
AW_PR_NUMBER/AW_HEAD_REF land in step env only, which never reaches the
agent sandbox, and on workflow_dispatch the event payload has no PR
object. The agent concluded "no associated pull request ... nothing to
review" and noop'd.

Fix 1: the resolve step (now id: resolve_pr) also writes step outputs
and /tmp/gh-aw/context/run-context.md — the context dir is the one
channel proven to reach the agent (it read jira-ticket.md in the same
run). The prompt body names run-context.md as the source of truth for
PR identity, treats a dispatch run with a resolved PR as a normal
review, and reserves noop for runs where no PR resolves at all.
(${{ steps.* }} interpolation into the body cannot work: the prompt is
rendered in the activation job, the steps run in the agent job.)

Fix 2: the same run showed "JIRA returned HTTP 404 for ENG-909" —
ambiguous, since JIRA answers 404 for missing ticket, missing project
permission, AND bad credentials. On any non-200 the step now probes
/rest/api/3/myself and writes an auth diagnosis into the marker
(credentials rejected vs authenticated-but-cannot-see vs unreachable),
reporting HTTP codes only, never credential values. Still never fails
the job.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…e + auto-recovery)

Root cause of the ENG-909 404: JIRA_API_TOKEN is a scoped Atlassian API
token, and scoped tokens must target https://api.atlassian.com/ex/jira/<cloudId>
instead of https://<site>.atlassian.net. Atlassian answers 404 (not 403)
when a scoped token hits the site-host form, which mimics a missing issue.

- Replace the /rest/api/3/myself probe (not callable with granular
  read:issue scopes -> would false-report valid credentials as broken)
  with GET https://api.atlassian.com/oauth/token/accessible-resources,
  which works for both token types and returns each site's cloudId.
- Auto-recovery: when the issue fetch fails and the probe can resolve a
  cloudId (configured URL matches a site, cloudId embedded in an ex/jira
  URL, or single-site token), retry via the scoped-token endpoint - so
  either token type works with either JIRA_BASE_URL form.
- Self-diagnosing markers on hard failure: credentials rejected (401/403,
  mentions <=365-day scoped-token expiry), base URL matches none of the
  token's sites (lists reachable site URLs + cloudIds and the exact value
  to set), ticket not visible/nonexistent (tried both endpoints), or
  api.atlassian.com unreachable. HTTP codes, site URLs and cloudIds only;
  never credential values.
- Comments fetch now uses the possibly-recovered base URL.
- README: JIRA_BASE_URL form guidance per token type.
- New helper .github/scripts/jira_sites.py (cloud-id resolution + site
  summary; always exits 0).

All six failure/recovery branches exercised locally against a mock
Atlassian server using the step script extracted verbatim from the
workflow; every path exits 0 and writes the marker.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@amjithtitus09
amjithtitus09 merged commit f0df859 into main Aug 17, 2026
4 of 5 checks passed
amjithtitus09 added a commit that referenced this pull request Aug 17, 2026
…o inconclusive arms

Per review of #152: a failed accessible-resources probe is not proof the
credentials are bad (the endpoint may not accept basic auth for every
token type), so the 401/403 arm no longer asserts "credentials rejected"
- it now enumerates likely causes. Every inconclusive arm (401/403,
unreachable, unexpected code) additionally surfaces the scoped/unscoped
two-URL rule whenever the configured base URL host is not
api.atlassian.com, and enumerates the three real possibilities: wrong
base URL for the token type, no access to the project, or nonexistent
ticket. Project name derived from the ticket key for accuracy.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
amjithtitus09 added a commit that referenced this pull request Aug 17, 2026
…nt_info

Item 1 (highest severity): GitHub Actions invokes run: steps as
`bash -e {0}`, so errexit is inherited — `set -uo pipefail` does NOT
clear it, only an explicit `set +e` does. The KEY extraction pipeline
(grep exits 1 on branches without ENG-nnn) therefore killed the
"never fails" JIRA step before its no_ticket fallback could run,
hard-failing the agent job for every non-ENG-branch PR (observed in
run 32017183079 on PR #152). Fix: explicit `set +e` in the JIRA and
lint steps with a comment citing the run, `|| true` on the grep
pipeline as belt-and-braces, and an audit of all pre-steps (the
resolve and skill-move steps stay deliberately strict). Reproduced
locally under `bash -e`: main's version exits 1, this version exits 0
and writes the marker.

Item 2: the two-URL rule is now empirically confirmed (scoped API
tokens are ignored entirely on *.atlassian.net — anonymous and
authenticated requests return identical status codes). Replace the
accessible-resources probe (OAuth-Bearer-only; returns 401 uniformly
for no-auth/bogus-basic/bogus-bearer, so its 401 says nothing about
credentials) with the public, unauthenticated
GET <site>/_edge/tenant_info for cloudId discovery, and auto-retry
via https://api.atlassian.com/ex/jira/<cloudId>. Remove the now
unused .github/scripts/jira_sites.py.

Item 3: carry forward the hedged diagnostics from orphaned commit
c75b882 (pushed after the #152 squash-merge), now decisive where the
evidence allows: when the configured host is *.atlassian.net, the
marker recommends the ex/jira form outright; arms stay hedged only
where JIRA's 404-for-unviewable behaviour makes causes genuinely
indistinguishable.

README: add the _edge/tenant_info cloudId discovery command.
Recompiled with gh-aw v0.86.2 (0 errors, 0 warnings).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
amjithtitus09 added a commit that referenced this pull request Aug 17, 2026
…nt_info (#153)

Item 1 (highest severity): GitHub Actions invokes run: steps as
`bash -e {0}`, so errexit is inherited — `set -uo pipefail` does NOT
clear it, only an explicit `set +e` does. The KEY extraction pipeline
(grep exits 1 on branches without ENG-nnn) therefore killed the
"never fails" JIRA step before its no_ticket fallback could run,
hard-failing the agent job for every non-ENG-branch PR (observed in
run 32017183079 on PR #152). Fix: explicit `set +e` in the JIRA and
lint steps with a comment citing the run, `|| true` on the grep
pipeline as belt-and-braces, and an audit of all pre-steps (the
resolve and skill-move steps stay deliberately strict). Reproduced
locally under `bash -e`: main's version exits 1, this version exits 0
and writes the marker.

Item 2: the two-URL rule is now empirically confirmed (scoped API
tokens are ignored entirely on *.atlassian.net — anonymous and
authenticated requests return identical status codes). Replace the
accessible-resources probe (OAuth-Bearer-only; returns 401 uniformly
for no-auth/bogus-basic/bogus-bearer, so its 401 says nothing about
credentials) with the public, unauthenticated
GET <site>/_edge/tenant_info for cloudId discovery, and auto-retry
via https://api.atlassian.com/ex/jira/<cloudId>. Remove the now
unused .github/scripts/jira_sites.py.

Item 3: carry forward the hedged diagnostics from orphaned commit
c75b882 (pushed after the #152 squash-merge), now decisive where the
evidence allows: when the configured host is *.atlassian.net, the
marker recommends the ex/jira form outright; arms stay hedged only
where JIRA's 404-for-unviewable behaviour makes causes genuinely
indistinguishable.

README: add the _edge/tenant_info cloudId discovery command.
Recompiled with gh-aw v0.86.2 (0 errors, 0 warnings).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant