From 812b4c318137ebf36a764b29d6ab3edc05cf1201 Mon Sep 17 00:00:00 2001
From: Andy Grove <agrove@apache.org>
Date: Mon, 14 Sep 2026 09:27:50 -0600
Subject: [PATCH] docs: add a regression check against the latest release tag
 to bug triage

Triage currently records how bad a bug is but not whether it is new. A defect
that shipped in the last release and one introduced since it read identically
on the issue tracker, even though the second one hits users who upgrade without
changing anything on their side.

Add a `regression` label and the procedure for deciding it. The comparison
point is resolved from the release tags at triage time rather than hard-coded,
so it moves forward on its own as Comet ships, and it is the tag's commit date
rather than the release's publication date because commits landing between the
two are not in the release.

The procedure is ordered cheapest-evidence-first: the issue's creation date
settles most of the backlog for free, then reading the implicated code at the
tag, then checking whether the path was reachable at all, and only then
building the tag and running the reproducer. That last step is the only way to
settle a case that turns on a dependency bump rather than on Comet's own code.

Two traps are called out explicitly. Issues found during PR review routinely
say "this is pre-existing, not caused by this PR", which is a claim about the
pull request under review and not about the last release. And a query that
failed loudly on the release and now returns silently wrong data is a
regression, even though it never produced the right answer on either version.

Update the bug-triage skill to run the check and to report regression status
per bug in its summary issue.
---
 .ai/skills/bug-triage/SKILL.md              | 189 ++++++++++++++++++--
 docs/source/contributor-guide/bug_triage.md |  71 +++++++-
 2 files changed, 242 insertions(+), 18 deletions(-)

diff --git a/.ai/skills/bug-triage/SKILL.md b/.ai/skills/bug-triage/SKILL.md
index 95d67f7e775..0dcc1aafd16 100644
--- a/.ai/skills/bug-triage/SKILL.md
+++ b/.ai/skills/bug-triage/SKILL.md
@@ -1,6 +1,6 @@
 ---
 name: bug-triage
-description: Triage open Comet issues marked `requires-triage` per the project bug triage guide. Classifies each issue as a bug or an enhancement, applies the recommended type (`bug`/`enhancement`), priority, and area labels, removes `requires-triage`, and files a dated summary issue listing what was done. A human reviews the summary issue and closes it when satisfied.
+description: Triage open Comet issues marked `requires-triage` per the project bug triage guide. Classifies each issue as a bug or an enhancement, checks whether each bug is a regression from the most recent release tag, applies the recommended type (`bug`/`enhancement`), priority, area, and `regression` labels, removes `requires-triage`, and files a dated summary issue listing what was done. A human reviews the summary issue and closes it when satisfied.
 ---
 
 <!--
@@ -32,10 +32,13 @@ each one it:
 1. Decides whether the issue is a bug or an enhancement (feature request).
 2. Decides a priority (bugs only) and area labels using the project's triage
    guide.
-3. Applies those labels via `gh` (`bug` or `enhancement`, plus priority/area),
-   ensuring `bug` and `enhancement` are never both present.
-4. Removes the `requires-triage` label.
-5. Records the decision (with rationale) in a single dated summary issue.
+3. Decides whether a bug is a regression from the most recent release, and
+   applies the `regression` label if it is.
+4. Applies those labels via `gh` (`bug` or `enhancement`, plus priority/area
+   and possibly `regression`), ensuring `bug` and `enhancement` are never both
+   present.
+5. Removes the `requires-triage` label.
+6. Records the decision (with rationale) in a single dated summary issue.
 
 `requires-triage` is auto-applied to **every** new issue, not just bug reports,
 so the first decision for each issue is always bug vs. enhancement.
@@ -91,7 +94,7 @@ For each issue, review the title and body and determine:
      be a genuine defect. Classify from the actual content.
    - A bug and an enhancement are mutually exclusive: an issue must never carry
      both `bug` and `enhancement`. If the issue already has the wrong type label,
-     remove it (see Step 5).
+     remove it (see Step 6).
 2. **Priority label** (exactly one, **bugs only**): apply the decision tree from
    the guide.
    - `priority:critical` for correctness issues (silent wrong results, data
@@ -109,7 +112,122 @@ For each issue, review the title and body and determine:
    guide (e.g., a `priority:high` crash that may also produce wrong results),
    note it in the summary.
 
-## Step 4: Skip Issues You Cannot Confidently Classify
+## Step 4: Check Whether Each Bug Is a Regression
+
+Run this step for every issue you classified as a `bug` in Step 3. Skip it for
+enhancements: a missing feature cannot be a regression.
+
+A bug is a **regression** when a workload that behaved correctly on the most
+recent Comet release behaves incorrectly on `main` — wrong results, a new
+failure, a crash, or a fallback-to-Spark path that has since become a wrong
+native answer. It also covers a loss of safety: a query that failed with a
+clear error on the release and now returns silently wrong data is a regression,
+even though it never produced the right answer on either version. A defect that
+already shipped in that release is **not** a regression, however recently it was
+reported.
+
+### Step 4a: Resolve the comparison point from the release tags
+
+Never hard-code a version. Ask GitHub for the current latest release and
+resolve its tag to a commit, so the comparison point moves forward on its own
+as Comet ships:
+
+```bash
+LATEST_RELEASE=$(gh release view \
+  --repo apache/datafusion-comet \
+  --json tagName --jq .tagName)
+git fetch --tags --quiet
+LATEST_RELEASE_SHA=$(git rev-list -n 1 "$LATEST_RELEASE")
+LATEST_RELEASE_DATE=$(git log -1 --format=%cI "$LATEST_RELEASE_SHA")
+```
+
+`gh release view` with no tag argument returns the release GitHub marks as
+"Latest", which excludes pre-releases. Comet marked every `0.x` release as a
+pre-release, so on a repository state where only pre-releases exist this
+returns nothing; in that case fall back to the newest tag by commit date and
+say which tag you used in the summary.
+
+Compare against the **tag's commit date**, not the release's publication date.
+The two differ — Comet's `1.0.0` tag was cut on 2026-08-04 and published on
+2026-08-07 — and commits landing in that window are not in the release.
+
+### Step 4b: Decide, cheapest evidence first
+
+Stop at the first step that gives a definite answer.
+
+1. **Issue creation date.** If the issue was opened before
+   `LATEST_RELEASE_DATE`, the defect was reported before the tag was cut, so it
+   shipped in that release. **Not a regression.** This is free and decisive for
+   most of the backlog.
+2. **Is the defective code present at the tag?** Read the implicated file at
+   the tag and compare it to `main`:
+
+   ```bash
+   git show "$LATEST_RELEASE:native/spark-expr/src/datetime_funcs/unix_timestamp.rs"
+   git grep -n "some_pattern" "$LATEST_RELEASE" -- path/to/dir
+   git diff "$LATEST_RELEASE"..HEAD -- path/to/file
+   ```
+
+   If the defective logic is there verbatim, **not a regression**. If the file
+   or the serde entry that reaches it does not exist at the tag, continue.
+
+3. **Was the path reachable at the tag?** Being absent from the tag is not the
+   same as being a regression. Split it:
+
+   - The feature is new since the release (a new expression serde, a new
+     operator, a new scan mode). A workload that ran on the release cannot
+     reach it. **Not a regression** — it is a defect in new work.
+   - The path existed and was correct, and a post-release change broke it.
+     **Regression.**
+   - The expression previously fell back to Spark (so it was correct) and a
+     post-release change made it run natively with a wrong answer.
+     **Regression** — the user-visible answer changed for the worse.
+
+   A useful check for the reachability question is whether the Scala serde
+   entry, shim, or native registration existed at the tag, not just the Rust
+   kernel:
+
+   ```bash
+   git grep -n "classOf\[SomeExpression\]" "$LATEST_RELEASE" -- 'spark/src/main'
+   git grep -n '"some_function"' "$LATEST_RELEASE" -- native/spark-expr/src/comet_scalar_funcs.rs
+   ```
+
+4. **Bisect or run the reproducer.** If steps 1–3 are inconclusive and the
+   issue has a reproducer, check the tag out into a scratch worktree, build,
+   and run it:
+
+   ```bash
+   git worktree add /tmp/comet-release-check "$LATEST_RELEASE"
+   cd /tmp/comet-release-check/native && cargo build
+   cd /tmp/comet-release-check && ./mvnw test -Dtest=none -Dsuites="<suite>"
+   ```
+
+   Remove the worktree when done (`git worktree remove /tmp/comet-release-check
+--force`). This is the only way to settle a case where the defect depends on
+   a dependency bump (a DataFusion or Arrow/Parquet major version) rather than
+   on Comet's own code.
+
+### Step 4c: Do not trust "pre-existing" in the issue body
+
+Comet issues found during PR review very often say "this is pre-existing, not
+caused by this PR". That claim is about the **pull request under review**, which
+is a narrower and different claim than "this shipped in the last release". A
+defect can be genuinely pre-existing relative to the PR that surfaced it and
+still have landed after the release tag. Verify against the tag either way.
+
+### Step 4d: Apply the label only on positive evidence
+
+- Add `regression` only when step 4b gives you positive evidence that the
+  release behaved correctly.
+- If the evidence is inconclusive, **do not** apply `regression`. Record the
+  issue under "Regression status unclear" in the summary and let the reviewer
+  decide. Do not label the rest of the issue differently on this account —
+  classification, priority, and area still apply.
+- `regression` is orthogonal to priority: a regression keeps the priority its
+  symptoms earn. Per the guide it is also an escalation trigger, so note in the
+  summary when a regression sits below `priority:high`.
+
+## Step 5: Skip Issues You Cannot Confidently Classify
 
 If an issue is too ambiguous to classify with confidence (you cannot tell
 whether it is a bug or an enhancement, or a bug lacks reproduction steps and the
@@ -125,7 +243,11 @@ priority is unclear):
 
 Guessing is worse than skipping.
 
-## Step 5: Apply Labels
+An unclear _regression_ status on its own is not a reason to skip an issue.
+Classify, prioritise, and label it as usual, and record it under "Regression
+status unclear" per Step 4d.
+
+## Step 6: Apply Labels
 
 For each issue you classified in Step 3, apply the labels and remove
 `requires-triage` in a single `gh` call.
@@ -139,6 +261,15 @@ gh issue edit <NUMBER> \
   --remove-label "requires-triage,enhancement"
 ```
 
+For a bug you determined in Step 4 to be a regression, add `regression` too:
+
+```bash
+gh issue edit <NUMBER> \
+  --repo apache/datafusion-comet \
+  --add-label "bug,priority:critical,area:scan,regression" \
+  --remove-label "requires-triage,enhancement"
+```
+
 For an enhancement, add the `enhancement` type label and no priority label:
 
 ```bash
@@ -157,6 +288,10 @@ Notes:
   so it is safe to remove the opposite type unconditionally.
 - Apply a priority label only to bugs. Do not add a priority label to
   enhancements.
+- Apply `regression` only to bugs, and only on the positive evidence described
+  in Step 4d. Never add it to an enhancement.
+- Do not _remove_ an existing `regression` label. If you believe one is wrong,
+  say so in the summary and leave the correction to the reviewer.
 - Pass the labels as a single comma-separated string (no spaces around commas).
 - Quote labels that contain spaces (e.g., `"spark 4"`).
 - Only add labels that already exist in the repo. If a label from the guide is
@@ -168,7 +303,7 @@ If `gh issue edit` fails for any issue, leave that issue's `requires-triage`
 label intact and record the failure in the summary under a "Failed to label"
 section.
 
-## Step 6: File the Summary Issue
+## Step 7: File the Summary Issue
 
 Compute today's date in `YYYY-MM-DD` form (use the system date, not memory):
 
@@ -183,6 +318,8 @@ Body: a markdown report with these sections, in this order:
 1. **Header**
    - Date, total issues processed, count of bugs vs. enhancements, and counts
      per priority
+   - The release tag the regression check compared against, and its commit date
+     (e.g. "Regressions assessed against `1.0.0` (tagged 2026-08-04)")
    - Link to `docs/source/contributor-guide/bug_triage.md`
    - Note that labels have already been applied; the reviewer should spot-check
      and close this issue when satisfied
@@ -198,23 +335,38 @@ Body: a markdown report with these sections, in this order:
 
    - <issue title> ([#1234](https://github.com/apache/datafusion-comet/issues/1234))
      - Area labels: `area:expressions`, `area:scan`
+     - Regression: no — the same code is present at `1.0.0`
      - Rationale: one sentence tying the call to the guide
    ```
 
    The issue number (not the title) is the link target. The title is plain
    text. If there are no area labels, write `Area labels: none`.
 
+   The `Regression:` sub-bullet is required on every bug. Write `yes`, `no`, or
+   `unclear`, followed by the one-line evidence that settled it — which step of
+   Step 4b answered it, and against which tag.
+
 3. **Enhancements** (omit section if empty) — one top-level bullet per issue in
    the same `<title> ([#N](url))` form, with an `Area labels:` sub-bullet and a
    one-sentence rationale for classifying it as an enhancement. Enhancements
-   have no priority subsections.
-4. **Escalations to consider** (omit section if empty) — bullet per issue with
+   have no priority subsections and no `Regression:` sub-bullet.
+4. **Regressions from `<tag>`** (omit section if empty) — every issue you
+   labelled `regression`, collected in one place so a release manager can read
+   them without scanning the priority sections. Bullet per issue in the same
+   `<title> ([#N](url))` form, plus a sub-bullet naming the change that
+   introduced it (a PR or commit, where you identified one) and a sub-bullet
+   with its priority label.
+5. **Regression status unclear** (omit if empty) — bullet per issue with the
+   same `<title> ([#N](url))` form, plus a sub-bullet saying what you checked
+   and what would settle it. These issues were still labelled and had
+   `requires-triage` removed.
+6. **Escalations to consider** (omit section if empty) — bullet per issue with
    the same `<title> ([#N](url))` form, plus a sub-bullet explaining the
    trigger from the guide.
-5. **Skipped — needs more info** (omit if empty) — bullet per issue with the
+7. **Skipped — needs more info** (omit if empty) — bullet per issue with the
    same `<title> ([#N](url))` form, plus a sub-bullet explaining what is
    missing.
-6. **Failed to label** (omit if empty) — bullet per issue with the same
+8. **Failed to label** (omit if empty) — bullet per issue with the same
    `<title> ([#N](url))` form, plus a sub-bullet quoting the `gh` error.
 
 File the issue with `gh`. Use a temp file for the body to keep quoting sane:
@@ -236,8 +388,10 @@ Report back:
 
 1. Number of `requires-triage` issues processed
 2. Counts per priority that were applied
-3. Number skipped (needs more info) and number failed
-4. URL of the new summary issue
+3. The release tag the regression check compared against, how many issues were
+   labelled `regression`, and how many were left unclear
+4. Number skipped (needs more info) and number failed
+5. URL of the new summary issue
 
 Do not paste the full per-issue listing back into the chat; it is in the
 summary issue.
@@ -246,6 +400,11 @@ summary issue.
 
 - Do not invent priority or area labels that are not in the guide
 - Do not create new labels in the repo
+- Do not hard-code a release version in the regression check; always resolve it
+  from the release tags as in Step 4a
+- Do not apply `regression` on the basis of the issue's own "pre-existing"
+  wording, a recent creation date, or a guess — only on the evidence in Step 4b
+- Do not remove a `regression` label that is already on an issue
 - Do not comment on the triaged issues
 - Do not close any triaged issue
 - Do not file the summary issue if there were zero `requires-triage` issues
diff --git a/docs/source/contributor-guide/bug_triage.md b/docs/source/contributor-guide/bug_triage.md
index b2bacd7567c..6e9dfe7a13b 100644
--- a/docs/source/contributor-guide/bug_triage.md
+++ b/docs/source/contributor-guide/bug_triage.md
@@ -76,6 +76,65 @@ A bug should be escalated to a higher priority if:
 - A `priority:medium` bug is reported by multiple users or affects a common workload → consider
   escalating to `priority:high`
 - A `priority:low` CI flake is blocking PR merges consistently → escalate to `priority:medium`
+- A bug turns out to be a `regression` from the most recent release → consider escalating one
+  level, because users who upgrade are exposed to it without changing anything on their side
+
+## Regression Label
+
+| Label        | Description                                             |
+| ------------ | ------------------------------------------------------- |
+| `regression` | A bug that did not affect the most recent Comet release |
+
+Apply `regression` to a bug when a workload that behaved correctly on the most recent release
+behaves incorrectly on `main`. That covers wrong results, a new failure, a new crash, and the case
+where an expression used to fall back to Spark (and was therefore correct) and now runs natively
+with a wrong answer. It also covers a loss of safety: a query that failed with a clear error on the
+last release and now returns silently wrong data is a regression, even though it never produced the
+right answer on either version.
+
+A defect that already shipped in the most recent release is **not** a regression, no matter how
+recently it was reported. Neither is a defect in a feature added after that release: a workload
+running on the release cannot reach code that did not exist yet.
+
+`regression` is orthogonal to priority. A regression still gets the priority label its symptoms
+earn, and it is an escalation trigger rather than a priority of its own. It applies only to bugs.
+
+### Determining the Comparison Point
+
+Always compare against the most recent release **tag**, resolved at triage time rather than
+hard-coded, so the comparison point moves forward as Comet ships:
+
+```bash
+LATEST_RELEASE=$(gh release view --repo apache/datafusion-comet --json tagName --jq .tagName)
+git fetch --tags
+git log -1 --format=%cI "$LATEST_RELEASE"
+```
+
+Compare against the **tag's commit date**, not the release's publication date — commits that land
+between the two are not in the release.
+
+### Establishing Regression Status
+
+Work through these in order and stop at the first definite answer:
+
+1. **Issue creation date.** An issue opened before the tag was cut describes behavior that shipped
+   in that release. Not a regression.
+2. **Is the defective code present at the tag?** `git show "$LATEST_RELEASE:<path>"`,
+   `git grep <pattern> "$LATEST_RELEASE"`, or `git diff "$LATEST_RELEASE"..HEAD -- <path>`. If the
+   defective logic is there verbatim, not a regression.
+3. **Was the path reachable at the tag?** Check that the Scala serde entry, shim, or native
+   registration existed, not just the kernel. Code absent from the tag means new work, not a
+   regression — unless a post-release change broke a path that used to be correct.
+4. **Run the reproducer against the tag.** Build the tag in a scratch worktree and run it. This is
+   the only way to settle cases that turn on a dependency bump (a DataFusion or Arrow/Parquet major
+   version) rather than on Comet's own code.
+
+Issues found during PR review often say "this is pre-existing, not caused by this PR". That is a
+claim about the pull request under review, not about the last release; a defect can be pre-existing
+relative to the PR that surfaced it and still have landed after the tag. Verify against the tag.
+
+If the evidence is inconclusive, leave `regression` off and say so on the issue rather than
+guessing.
 
 ## Area Labels
 
@@ -111,9 +170,11 @@ When a new issue is filed:
 3. **Assess correctness impact first.** Ask: "Could this produce wrong results silently?" This
    is more important than whether it crashes.
 4. **Apply a priority label** using the decision tree above (bugs only).
-5. **Apply area labels** to indicate the affected subsystem(s).
-6. **Apply `good first issue`** if the fix is likely straightforward and well-scoped.
-7. **Remove the `requires-triage` label** to indicate triage is complete.
+5. **Check whether the bug is a regression** from the most recent release tag and apply
+   `regression` if it is (bugs only).
+6. **Apply area labels** to indicate the affected subsystem(s).
+7. **Apply `good first issue`** if the fix is likely straightforward and well-scoped.
+8. **Remove the `requires-triage` label** to indicate triage is complete.
 
 ### For Existing Bugs
 
@@ -122,6 +183,9 @@ Periodically review open bugs to ensure priorities are still accurate:
 - Has a `priority:medium` bug been open for a long time with user reports? Consider escalating.
 - Has a `priority:high` bug been fixed by a related change? Close it.
 - Are there clusters of related bugs that should be tracked under an EPIC?
+- Does an open bug need its regression status re-checked against a newer release? A bug that was
+  a regression from one release is still a regression once the next release ships with it
+  unfixed, so `regression` stays until the bug is fixed.
 
 ### Prioritization Principles
 
@@ -183,3 +247,4 @@ Triage is a valuable contribution that doesn't require writing code. You can hel
 - Identifying duplicate issues
 - Linking related issues together
 - Testing whether old bugs have been fixed by recent changes
+- Checking whether an open bug is a `regression` from the most recent release tag