Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4,369 changes: 4,369 additions & 0 deletions .config/rustqual/baseline-dry.json

Large diffs are not rendered by default.

4,344 changes: 4,344 additions & 0 deletions .config/rustqual/baseline.json

Large diffs are not rendered by default.

35 changes: 35 additions & 0 deletions .config/rustqual/config.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# Structural quality checks, main pass: every dimension except DRY, across the
# full workspace including test files.
#
# Run via `just qual`; gated on CI by `just qual-ci`.
#
# ## Why DRY is disabled here
#
# rustqual's DRY dimension is all-or-nothing. Turning off `[duplicates]`
# disables DRY-002 (dead code) and DRY-006 (dead types) along with DRY-003
# (repeated fragments) — verified, not assumed: the dead-code count goes 256 to
# 0 on this workspace when the section is disabled.
#
# DRY-003 fires ~2038 times inside `*_tests.rs`, on the repeated setup blocks
# that make each `#[test]` self-contained. That shape is deliberate here (see
# the testing guidance in the contributor docs: prefer duplication over a
# shared fixture helper), so those findings are noise rather than signal.
#
# The resolution is a second pass, `dry.toml`, which enables DRY over
# production code only. Test files cannot simply be excluded from *this* pass
# instead, because doing so would take the Test Quality dimension with them:
# TQ-001/002/003 all need `#[test]` functions in the parse.

[duplicates]
enabled = false

# rustqual judges test code by production thresholds unless told otherwise.
# Several test functions here are deliberately long and self-contained, so
# `LONG_FN` and `SRP_MODULE` fire on files that are working as intended. Left
# unset so the baseline records the real numbers; raise these if the findings
# turn out to obscure genuine regressions rather than merely sit in the
# baseline.
#
# [tests]
# max_function_lines = 150
# file_length = 800
56 changes: 56 additions & 0 deletions .config/rustqual/dry.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# Structural quality checks, DRY pass: duplicates and dead code over production
# code only.
#
# Paired with `config.toml`, which runs every other dimension across the full
# workspace. See that file for why the two passes are split.
#
# Excluding test files here is safe because nothing in this pass depends on
# them: `[test_quality]` is off, so TQ-003 cannot mistake an untested-looking
# function for a real finding, and DRY-002 already ignores calls from
# `#[cfg(test)]` code when deciding deadness — the dead-code count is identical
# (256) with and without these globs. What the exclusion removes is ~2038
# DRY-003 fragments and ~210 DRY-001 duplicate test functions, both of which
# are the deliberate self-contained-test shape rather than copy-paste.
#
# Two dimensions cannot be disabled and will appear in this pass as well as in
# the main one: IOSP has no `enabled` key, and MAGIC_NUMBER survives
# `[complexity] enabled = false` (every other complexity rule is correctly
# silenced). Both are absorbed by this pass's own baseline and do not affect the
# regression gate.
#
# The same inconsistency produces three `ORPHAN_SUPPRESSION` findings here, on
# the `qual:allow(complexity, magic_numbers)` markers in the provider model
# catalogues. With `[complexity]` off, rustqual sees no complexity finding to
# match them against and calls them stale — while MAGIC_NUMBER fires anyway.
# They suppress correctly in the main pass (498 findings down to 193). Baselined
# here; not worth excluding the provider files from DRY detection to silence.

exclude_files = [
"**/*_tests.rs",
"**/tests.rs",
"**/tests/**",
]

[duplicates]
enabled = true

[boilerplate]
enabled = false

[complexity]
enabled = false

[srp]
enabled = false

[coupling]
enabled = false

[structural]
enabled = false

[test_quality]
enabled = false

[architecture]
enabled = false
42 changes: 42 additions & 0 deletions .config/rustqual/regressions.jq
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# Report every count in a fresh rustqual report that sits above the committed
# baseline, one `key: old -> new (+delta)` line per category. Empty output means
# no category moved up.
#
# Invoked as:
#
# jq -nr -f regressions.jq --slurpfile old <baseline> --slurpfile new <fresh>
#
# Per category rather than on `total_findings` alone: a change that fixes three
# magic numbers and adds three dead functions leaves the total flat while making
# the codebase worse along an axis we care about.
#
# A baseline that cannot be compared is an error, not a pass. Invalid JSON fails
# in jq before this program runs; a file that parses but carries no numeric
# `total_findings` (`{}`, or a schema from an incompatible rustqual) is rejected
# below. Both would otherwise disable the gate silently.
#
# Keys deliberately not compared:
#
# version schema marker, not a count
# total function count, which grows with the codebase
# quality_score, iosp_score ratios, which are what we are not gating on

{version: 1, total: 1, quality_score: 1, iosp_score: 1} as $skip
| $old[0] as $o
| $new[0] as $n
| if ($o | type) != "object" then
error("baseline is not a JSON object")
elif ($o.total_findings | type) != "number" then
error("baseline has no numeric total_findings: corrupt, or written by an incompatible rustqual")
elif ($n.total_findings | type) != "number" then
error("fresh report has no numeric total_findings: rustqual's baseline format changed")
else
$n
| to_entries[]
| select((.value | type) == "number")
| select($skip[.key] == null)
| . as $entry
| ($o[$entry.key] | if type == "number" then . else 0 end) as $was
| select($entry.value > $was)
| " \($entry.key): \($was) -> \($entry.value) (+\($entry.value - $was))"
end
6 changes: 5 additions & 1 deletion .github/workflows/rust.yml
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ jobs:
insta: ${{ steps.filter.outputs.insta }}
shear: ${{ steps.filter.outputs.shear }}
vet: ${{ steps.filter.outputs.vet }}
qual: ${{ steps.filter.outputs.qual }}
steps:
- uses: actions/checkout@0c366fd6a839edf440554fa01a7085ccba70ac98 # v6
with:
Expand Down Expand Up @@ -93,13 +94,14 @@ jobs:
set_task insta "$source|$snapshots"
set_task shear "$source"
set_task vet "$cargo|^\.config/supply-chain/|$workflow|$justfile"
set_task qual "$source|^\.config/rustqual/"
rust:
needs: changes
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-latest]
task: [lint, fmt, fmt-comments, fmt-markdown, typecheck-js, test, docs, coverage, deny, insta, shear, vet]
task: [lint, fmt, fmt-comments, fmt-markdown, typecheck-js, test, docs, coverage, deny, insta, shear, vet, qual]
include:
- task: test
os: windows-latest
Expand Down Expand Up @@ -136,6 +138,7 @@ jobs:
INSTA: ${{ needs.changes.outputs.insta }}
SHEAR: ${{ needs.changes.outputs.shear }}
VET: ${{ needs.changes.outputs.vet }}
QUAL: ${{ needs.changes.outputs.qual }}
run: |
set -euo pipefail

Expand All @@ -152,6 +155,7 @@ jobs:
insta) run="$INSTA" ;;
shear) run="$SHEAR" ;;
vet) run="$VET" ;;
qual) run="$QUAL" ;;
*) echo "unknown task: $TASK" >&2; exit 1 ;;
esac

Expand Down
2 changes: 2 additions & 0 deletions crates/jp_llm/src/provider/anthropic.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2447,6 +2447,8 @@ fn model_overrides(id: &str) -> Option<&'static ModelOverrides> {
}

/// The facts the capabilities API cannot supply, per model.
// qual:allow(complexity, magic_numbers) reason: "model catalogue: cutoff dates
// are data maintained by hand against Anthropic's published documentation"
static MODEL_OVERRIDES: LazyLock<Catalog<ModelOverrides>> = LazyLock::new(|| {
let cutoff = |year, month| NaiveDate::from_ymd_opt(year, month, 1);

Expand Down
3 changes: 3 additions & 0 deletions crates/jp_llm/src/provider/google.rs
Original file line number Diff line number Diff line change
Expand Up @@ -532,6 +532,9 @@ fn model_overrides(id: &str) -> Option<&'static ModelDetails> {
/// See: <https://ai.google.dev/gemini-api/docs/models> See:
/// <https://ai.google.dev/gemini-api/docs/deprecations> See:
/// <https://ai.google.dev/gemini-api/docs/thinking#levels-budgets>
// qual:allow(complexity, magic_numbers) reason: "model catalogue: thinking
// budgets and cutoff dates are data maintained against Google's published
// documentation"
static MODEL_OVERRIDES: LazyLock<Catalog<ModelDetails>> = LazyLock::new(|| {
let date = |year, month, day| NaiveDate::from_ymd_opt(year, month, day).unwrap();
let id = |name: &str| ModelIdConfig::try_from((PROVIDER, name)).unwrap();
Expand Down
2 changes: 2 additions & 0 deletions crates/jp_llm/src/provider/openai.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1717,6 +1717,8 @@ fn model_overrides(id: &str) -> Option<&'static ModelDetails> {
/// Unlike the Anthropic, OpenRouter, and Cerebras providers, there is nothing
/// to derive from, so every value here is maintained by hand against OpenAI's
/// published model documentation.
// qual:allow(complexity, magic_numbers) reason: "model catalogue: context
// windows, output limits and cutoff dates are data, not constants"
static MODEL_OVERRIDES: LazyLock<Catalog<ModelDetails>> = LazyLock::new(|| {
let date = |year, month, day| NaiveDate::from_ymd_opt(year, month, day).unwrap();
let id = |name: &str| ModelIdConfig::try_from((PROVIDER, name)).unwrap();
Expand Down
116 changes: 116 additions & 0 deletions docs/ticket/0yb2pgw-rustqual-baselines-go-stale-when-main-moves.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
# Rustqual baselines go stale when main moves

- **Status**: Todo
- **Kind**: Chore
- **Authors**: jp
- **Date**: 2026-10-06
- **Label**: domain=tooling
- **Label**: type=follow-up

`just qual-ci` compares the workspace against counts committed in
`.config/rustqual/baseline.json` and `baseline-dry.json`.
Those counts describe the tree at the moment they were recorded, so they go
stale as soon as `main` moves.

The first real run of the gate demonstrated it.
The baselines were recorded on 22 Sep; rebasing the branch onto 6 Oct `main` put
382 extra findings in the main pass and 116 in the DRY pass, across twelve
categories, none of them from the branch — its whole Rust diff was seven
comment lines.
The gate was right that the numbers had moved and had no way to say whose
numbers they were.

## Why it matters

The author who hits the failure is whoever rebases next, and the documented
response (`just qual-baseline`) makes them re-record a baseline covering two
weeks of someone else's code.
Refreshing is the only action that turns the job green, it always works, and it
takes one command — so the gate teaches contributors to reach for the thing
that silences it.
That is the failure mode the gate exists to prevent, arrived at from the other
direction.

The refresh is at least visible: the baselines are pretty-printed, so
`"total_findings": 2404 → 2786` shows up in the pull request diff where a
reviewer can see it.
That is a review-time catch, not a tooling one.

## Options

1. **Accept it.** Refresh on rebase and treat the failure as noise.
Free, and normalizes the reflex.
2. **Re-record from `main` on a schedule.** A workflow runs `qual-baseline` on
`main` and commits the result, so a branch compares against a baseline that
is at most a day old.
One extra workflow; the drift window shrinks but does not close, and the
commits are noise in the log.
3. **Drop the committed baseline.** Analyse `HEAD` and the merge-base in the
same job and compare the two reports.
Staleness becomes impossible, because both numbers come from the run.
Costs four analyses per job instead of two, and needs the merge-base tree
checked out — probably a second worktree.

Option 3 removes the whole class of problem and is the one to aim for.
Worth measuring how long one `rustqual` pass takes over the workspace before
committing to it: if a pass is seconds, the extra cost is irrelevant and option
3 wins outright.

## Related

Introduced with the gate itself in `#1190`.
The same pull request already replaced rustqual's own `--compare
--fail-on-regression`, which scored a regression as the aggregate quality ratio
dropping and so missed new findings entirely; `.config/rustqual/regressions.jq`
now does a per-category comparison.
Whatever replaces the committed baseline should keep that filter — the
per-category check is orthogonal to where the numbers come from.

## Comments

-----

- **From**: jp
- **Date**: 2026-10-06T15:03:44Z

## Observed: the gate fires on commits the branch does not contain

First real CI run of `qual-ci` on `#1190` failed with `srp_module_warnings: 207
-> 208 (+1)` while the same recipe passed locally with identical numbers on both
sides.

Cause is narrower than "baselines go stale", and worse:

`.github/workflows/rust.yml:167` checks out with no `ref:`, so a `pull_request`
run gets `refs/pull/N/merge` — the head merged into *current* `main`.
The branch was rebased onto `150ba7868` and its baselines re-recorded there
(`e7e3b424c`).
`main` then took `dbafe5485` (#1242, 16:30) and `dfae02a91` (#1207, 16:33) the
same afternoon.
CI measured `branch + dfae02a91`; the baseline describes `branch + 150ba7868`.
The `+1` is a file crossing a threshold in that delta — #1242 adds net +21
lines to `crates/jp_cli/src/cmd/query/tool/prompter.rs`, which is otherwise
unflagged.

Two consequences beyond staleness:

1. **Not reproducible locally.** No sequence of local commands shows the
failure, because the tree CI measured does not exist in any worktree.
The author's only lever is to rebase and re-record — which is the reflex
this ticket exists to prevent, now with no diagnostic alternative.
2. **The comparison is invalid in both directions.** A committed baseline
describes exactly one tree.
Compared against a merge tree, it invents regressions *and* can mask real
ones: if the merge happens to delete findings elsewhere, a genuine new
finding nets to zero and passes.

So a committed baseline and the merge ref cannot both be right.
The measured tree and the baseline have to share a base.
That makes option 3 (analyse `HEAD` and the merge-base in the same job, no
committed baseline) the correct target rather than merely the tidiest.

A cheaper interim exists: a per-task `checkout_ref` through `matrix.include`,
set to `github.event.pull_request.head.sha` for `qual` only, leaving `test` and
the other tasks on the merge ref where testing the merge result is the point.
Needs one thing verified first — whether `actions/checkout` treats an empty
`ref:` as unset, which is what the other matrix entries would pass.
Loading
Loading