Repository navigation
Conversation
JeanMertz
force-pushed
the
rustqual
branch
2 times, most recently
from
October 6, 2026 14:42
62711e0 to
cb3bdb2
Compare
`just qual` runs rustqual across the workspace and `just qual-ci` gates CI on it, failing when a change adds findings beyond the committed baselines. The bar for new code is "no worse than what is already there" rather than an absolute score: the workspace carries known findings, and a score target would push contributors toward suppressing rather than fixing. The check runs as two passes because rustqual's DRY dimension is all-or-nothing. Disabling `[duplicates]` also disables DRY-002 dead code and DRY-006 dead types, while leaving it on fires DRY-003 roughly 2038 times inside `*_tests.rs`, on the repeated setup blocks that keep each `#[test]` self-contained. `.config/rustqual/config.toml` therefore covers every dimension except DRY across the full workspace, and `dry.toml` covers DRY over production code only. Excluding test files from that second pass costs nothing: DRY-002 already ignores calls from `#[cfg(test)]` code, so the dead-code count is identical with and without them. Three `qual:allow(complexity, magic_numbers)` markers cover the provider model catalogues in `jp_llm`, where context windows, output limits and cutoff dates are data maintained against each provider's published documentation rather than constants worth naming. They cut the main pass from 498 magic-number findings to 193. Refresh the baselines with `just qual-baseline` after a change that deliberately moves the count, and commit the result alongside it so the diff shows what moved. Signed-off-by: Jean Mertz <git@jeanmertz.com>
The `qual` matrix entry failed immediately with `unknown task: qual`. Adding a task to the `rust` job takes four edits, not two: the `changes` job output, the path filter, the matrix list, and the per-task gate that maps `$TASK` to the filter result. The first three landed with the rustqual gate; this adds the fourth. The gate's `case` statement has no default arm that lets an unmapped task through, so the omission failed the job outright instead of quietly running `qual-ci` on every push. Worth keeping in mind when adding the next task. Signed-off-by: Jean Mertz <git@jeanmertz.com>
`cargo binstall --only-signed` has nothing to resolve for rustqual: the crate declares no `[package.metadata.binstall]`, every GitHub release from v0.4.2 through v1.8.2 ships zero assets, and it is absent from cargo-quickinstall. The `qual` job would have failed at the install step on its first run. `_install-rustqual` builds from crates.io instead. `cargo install` no-ops in well under a second once the pinned version is present, so every recipe can depend on it unconditionally, and a version bump reinstalls on its own rather than needing `JP_INSTALL=1` the way the path-installed tools do. No cache step yet. The `qual` job already runs under sccache, which covers rustqual's dependency tree from the second run onward; whether that leaves the install cheap enough is a question for the CI timings rather than for a guess. Signed-off-by: Jean Mertz <git@jeanmertz.com>
`qual-ci` went red on its first run with `Findings: 2404 → 2404 (+0)`. `--fail-on-regression` adds a failure condition rather than replacing the default one: rustqual returns early when a comparison regresses, and otherwise falls through to a gate that fails whenever any finding exists. On a baselined workspace that is every run. `--no-fail` disables only that second gate. It cannot mask a regression, because the comparison returns before the gates are reached: `handle_baseline_and_compare(cli, &analysis)?` sits above `apply_exit_gates` in rustqual's `run_full_analysis`. Output is now captured rather than streamed. Each pass printed all of its baselined findings on every run, burying the comparison block that carries the verdict; a clean run shows only that block, and a regression shows the whole log. `--format github` is dropped because it annotates every baselined finding rather than the new ones, and GitHub displays at most ten, so the annotations showed pre-existing findings and hid the regression. Signed-off-by: Jean Mertz <git@jeanmertz.com>
`just qual-ci` now fails whenever any finding category sits above its committed baseline, so a PR that adds a dead function or an 8-parameter function is red even when it also adds enough compliant code to hold the overall score level. The previous gate asked rustqual for the verdict via `--compare --fail-on-regression`, which prints a finding delta but defines a regression as `quality_score` dropping. That score is a ratio over every function in the workspace, so new findings arriving alongside new compliant code left it flat and the job passed. A baseline rustqual could not parse was worse: it printed the error, reported "not regressed", and exited 0 under `--no-fail`, leaving a green job with no comparison at all and no gate for that pass. rustqual now runs purely as a report generator, writing a fresh baseline to a temp file, and `.config/rustqual/regressions.jq` decides the verdict. It walks every count in the report, skipping `version`, `total` and the two ratios, and reports each one above its baseline. A baseline that is invalid JSON fails in jq; one that parses but carries no numeric `total_findings` is rejected outright, so neither can silence the gate. Because a gate that has broken open is invisible, both directions are checked on every run before the real passes: a baseline one finding below the current state has to read as a regression, and a malformed baseline has to be an error. Either assertion failing fails the job. A regression now names the categories that moved rather than dumping the whole report, and says what to do about it. Signed-off-by: Jean Mertz <git@jeanmertz.com>
The committed baselines measured the workspace as it stood on 22 Sep, when this branch was written. Rebasing onto `150ba7868` moved the base to 6 Oct, and the two weeks of main development in between add 382 findings to the main pass and 116 to the DRY pass, spread across twelve categories. None of that is this branch's doing: its entire Rust diff is seven comment lines adding `qual:allow` markers to the three provider model catalogues, and a comment cannot introduce an IOSP violation or an unsafe block. The findings arrived through pull requests that merged before this gate existed, so the current tree is the state the gate is meant to accept. Recorded with `just qual-baseline`, which is the documented response to a deliberate move in the count. Both passes report no category above baseline afterwards. Signed-off-by: Jean Mertz <git@jeanmertz.com>
Records the operational weakness the gate's first run exposed: the committed baselines describe a tree snapshot, so `main` moving shows up as a regression in whichever branch rebases next, and the one command that clears it re-records someone else's findings as the new floor. Lists the three ways out with their costs, and names computing both sides of the comparison in the job as the one to aim for, since it is the only option under which a baseline cannot be stale. Signed-off-by: Jean Mertz <git@jeanmertz.com>
Signed-off-by: Jean Mertz <git@jeanmertz.com>
Signed-off-by: Jean Mertz <git@jeanmertz.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🚧 this still needs assistant-facing tools 🚧
just qualruns rustqual across the workspace andjust qual-cigates CI on it, failing when a change adds findings beyond the committed baselines. The bar for new code is "no worse than what is already there" rather than an absolute score: the workspace carries known findings, and a score target would push contributors toward suppressing rather than fixing.The check runs as two passes because rustqual's DRY dimension is all-or-nothing. Disabling
[duplicates]also disables DRY-002 dead code and DRY-006 dead types, while leaving it on fires DRY-003 roughly 2038 times inside*_tests.rs, on the repeated setup blocks that keep each#[test]self-contained..config/rustqual/config.tomltherefore covers every dimension except DRY across the full workspace, anddry.tomlcovers DRY over production code only. Excluding test files from that second pass costs nothing: DRY-002 already ignores calls from#[cfg(test)]code, so the dead-code count is identical with and without them.Three
qual:allow(complexity, magic_numbers)markers cover the provider model catalogues injp_llm, where context windows, output limits and cutoff dates are data maintained against each provider's published documentation rather than constants worth naming. They cut the main pass from 498 magic-number findings to 193.Refresh the baselines with
just qual-baselineafter a change that deliberately moves the count, and commit the result alongside it so the diff shows what moved.