Repository navigation
scripts: color verdicts and show a status line on a terminal - #117
Merged
Merged
Conversation
The qualification runs print long plain logs on the host's terminal, and during a measurement interval of up to 30 minutes the harness printed nothing, so a verdict was hard to pick out and a quiet run looked stuck. On a terminal, the harness, validate-upstream.py and the checks now print each verdict in color: green passed or qualified, red failed, yellow inconclusive, incomplete or not run, and magenta interrupted. A status line under the output shows what runs and for how long: the scenario, or in a measurement interval the time into its planned length, the gateway that serves and the load's runs. `--color auto|always|never` chooses, and auto means a terminal unless NO_COLOR is set or TERM is dumb. The harness's `--console-log <new file>` keeps a plain copy of everything the run prints, without color or the status line, so a run started from a terminal no longer needs a pipe to tee, which turned both off. The file is created new and never written over. Output to a file or a pipe is unchanged, byte for byte, and without a terminal or a console log nothing replaces stdout or stderr. The evidence, verify's JSON and every output a program reads stay plain. scripts/console.py is standard library only and runs on Python 3.9. Since code_tree() covers scripts/, later commits of main get another code tree than the qualified candidate 1c0eadd; the recovery protocol's status says so. The next qualification runs on this harness. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On a terminal, the harness,
validate-upstream.pyand the checks print each verdict in color and show a status line with what runs and for how long. The harness's--console-logkeeps a plain copy of the output. Output to a file or a pipe is unchanged, byte for byte.No product change. No crate, pin, release or policy change: only
scripts/and documentation.Issue
No issue. A follow-up to P1R-C06 (#101): the qualification runs print long plain logs on the qualification host's terminal, where a verdict was hard to pick out.
Summary
Before: every line was plain. During a measurement interval of up to 30 minutes the harness printed nothing, and a run started from a terminal kept its console log only through a pipe to
tee.After: on a terminal, each verdict is in color: green passed or qualified, red failed or the harness failed, yellow inconclusive, incomplete or not run, magenta interrupted. A status line under the output shows what runs and for how long: the scenario, or in a measurement interval the time into its planned length, the gateway that serves and the load's runs, as in
qualify: independent-off-1 idle: 0:27 of 2:00, gateway 2.--color auto|always|neverchooses; auto means a terminal, unlessNO_COLORis set orTERMisdumb. The harness's--console-log <new file>keeps everything the run prints, without color or the status line, so a run started from a terminal no longer needstee, which turned both off. Output to a file or a pipe is byte-identical to before, and without a terminal or a console log nothing replaces stdout or stderr.How.
scripts/console.py, new, standard library only, Python 3.9: the color choice, the styles, the status line and the console log. The status line is redrawn every second on a terminal and taken away before every write to stdout or stderr, so it never lands inside a line. The console log is a new file (O_EXCL, mode 0600) with the ANSI codes removed; an exception orSystemExitmessage that ends the program is copied to it as the interpreter prints it.scripts/qualify-devguard.py: everyqualify:line goes through it, with status lines for each scenario, each control and recovery interval and the suite;--colorand--console-logon every suite butverify;console.pyjoins the harness files thatreport.jsonrecords.scripts/validate-upstream.py: the running stage and its time on the status line, the+commands dim and failures in red.scripts/check_docs.py,scripts/codex_patch.py,scripts/check-no-model-deps.shandscripts/check-upstream-pin.sh: their outcome in color on a terminal.docs/cs-rg-qualification.md,validate-upstream.pyon a terminal indocs/upstream-update.md, and in the status ofdocs/recovery-qualification.md, that this change gives later commits another code tree.The code tree.
code_tree()coversscripts/, so after this PR main has another code tree than the qualified candidate1c0eadd, and the qualified build stays1c0eadditself. The next qualification runs on this harness.Contract changes
None. No tool schema, error code, status value, ID or evidence format changes, and
verify's output and every other JSON output are unchanged.Tests
Verified in the cloud (Linux) on this head:
python3 -B -m unittest discover -s scripts/tests: 144 tests pass on Python 3.13 and on Python 3.9.25.scripts/tests/test_console.py(the color choice, the styles, the status line on a fake terminal, the console log, an existing log refused, an ending exception kept), and intest_qualify_devguard.py, lines with and without color, a run whose console log equals its output without color, and a recovery interval's status line.script, 120 columns) in the minimal environment (env -i), on this head with a clean tree and a gateway and Runner built from it: the recovery suite's rehearsal with--participation off. Its 34 recovery scenarios passed (the two that need a host reboot or a Runner restart are not run), and it endedinconclusivewith exit status 2, as a rehearsal without a DevGuard service does. Verdicts showed in color and the status line was redrawn 493 times; replayed as a terminal shows it, the screen's 40 lines are the console log's 40 lines, which hold no escape code.verifyverified the run with the same harness.bash scripts/check-no-model-deps.shandPIN_ONLY=1 bash scripts/check-upstream-pin.sh: plain when redirected, in color on a pseudo-terminal.python3 scripts/check_docs.py: "Documentation registry verified: 22 paired documents". The documentation site with the pinned Node 24.21.0:npm test,npm run build,site.py checkandsite.py check --commiton this head passed.Not verified here: the qualification host (macOS, its Python 3.9.6, Terminal). The next qualification run uses it there.
Security scenarios
--color always; no environment variable turns it on, so a child whose output is captured, such ascodex_patch.py checkunder the harness, prints plain.O_EXCL, mode 0600), outside the sealed evidence, and holds what the terminal shows. The harness passes credential files by their path and never prints a credential.report.json's record of the harness files gainsconsole.py.Out of scope