Skip to content

scripts: color verdicts and show a status line on a terminal - #117

Merged
novelKR merged 1 commit into
mainfrom
claude/p1r-c06-qualification-report-k7gur1
Oct 19, 2026
Merged

novelKR merged 1 commit into
mainfrom
claude/p1r-c06-qualification-report-k7gur1

Conversation

@novelKR

@novelKR novelKR commented Oct 19, 2026 •

Copy link
Copy Markdown
Owner

On a terminal, the harness, validate-upstream.py and the checks print each verdict in color and show a status line with what runs and for how long. The harness's --console-log keeps a plain copy of the output. Output to a file or a pipe is unchanged, byte for byte.

No product change. No crate, pin, release or policy change: only scripts/ and documentation.

Issue

No issue. A follow-up to P1R-C06 (#101): the qualification runs print long plain logs on the qualification host's terminal, where a verdict was hard to pick out.

Summary

Before: every line was plain. During a measurement interval of up to 30 minutes the harness printed nothing, and a run started from a terminal kept its console log only through a pipe to tee.

After: on a terminal, each verdict is in color: green passed or qualified, red failed or the harness failed, yellow inconclusive, incomplete or not run, magenta interrupted. A status line under the output shows what runs and for how long: the scenario, or in a measurement interval the time into its planned length, the gateway that serves and the load's runs, as in qualify: independent-off-1 idle: 0:27 of 2:00, gateway 2. --color auto|always|never chooses; auto means a terminal, unless NO_COLOR is set or TERM is dumb. The harness's --console-log <new file> keeps everything the run prints, without color or the status line, so a run started from a terminal no longer needs tee, which turned both off. Output to a file or a pipe is byte-identical to before, and without a terminal or a console log nothing replaces stdout or stderr.

How.

  • scripts/console.py, new, standard library only, Python 3.9: the color choice, the styles, the status line and the console log. The status line is redrawn every second on a terminal and taken away before every write to stdout or stderr, so it never lands inside a line. The console log is a new file (O_EXCL, mode 0600) with the ANSI codes removed; an exception or SystemExit message that ends the program is copied to it as the interpreter prints it.
  • scripts/qualify-devguard.py: every qualify: line goes through it, with status lines for each scenario, each control and recovery interval and the suite; --color and --console-log on every suite but verify; console.py joins the harness files that report.json records.
  • scripts/validate-upstream.py: the running stage and its time on the status line, the + commands dim and failures in red.
  • scripts/check_docs.py, scripts/codex_patch.py, scripts/check-no-model-deps.sh and scripts/check-upstream-pin.sh: their outcome in color on a terminal.
  • The documentation, with its Korean translation: the harness's terminal output under "Running the harness" in docs/cs-rg-qualification.md, validate-upstream.py on a terminal in docs/upstream-update.md, and in the status of docs/recovery-qualification.md, that this change gives later commits another code tree.

The code tree. code_tree() covers scripts/, so after this PR main has another code tree than the qualified candidate 1c0eadd, and the qualified build stays 1c0eadd itself. The next qualification runs on this harness.

Contract changes

None. No tool schema, error code, status value, ID or evidence format changes, and verify's output and every other JSON output are unchanged.

Tests

Verified in the cloud (Linux) on this head:

  • python3 -B -m unittest discover -s scripts/tests: 144 tests pass on Python 3.13 and on Python 3.9.25.
  • New tests: scripts/tests/test_console.py (the color choice, the styles, the status line on a fake terminal, the console log, an existing log refused, an ending exception kept), and in test_qualify_devguard.py, lines with and without color, a run whose console log equals its output without color, and a recovery interval's status line.
  • The harness on a pseudo-terminal (script, 120 columns) in the minimal environment (env -i), on this head with a clean tree and a gateway and Runner built from it: the recovery suite's rehearsal with --participation off. Its 34 recovery scenarios passed (the two that need a host reboot or a Runner restart are not run), and it ended inconclusive with exit status 2, as a rehearsal without a DevGuard service does. Verdicts showed in color and the status line was redrawn 493 times; replayed as a terminal shows it, the screen's 40 lines are the console log's 40 lines, which hold no escape code. verify verified the run with the same harness.
  • bash scripts/check-no-model-deps.sh and PIN_ONLY=1 bash scripts/check-upstream-pin.sh: plain when redirected, in color on a pseudo-terminal.
  • python3 scripts/check_docs.py: "Documentation registry verified: 22 paired documents". The documentation site with the pinned Node 24.21.0: npm test, npm run build, site.py check and site.py check --commit on this head passed.

Not verified here: the qualification host (macOS, its Python 3.9.6, Terminal). The next qualification run uses it there.

Security scenarios

  • Nothing a program reads is styled. No JSON, evidence file, MCP message, gateway or Runner log, GitHub output or step summary goes through the console. Color comes only from a terminal or --color always; no environment variable turns it on, so a child whose output is captured, such as codex_patch.py check under the harness, prints plain.
  • The console log is a new file that is never written over (O_EXCL, mode 0600), outside the sealed evidence, and holds what the terminal shows. The harness passes credential files by their path and never prints a credential.
  • Evidence unchanged. Verdicts, evidence files and sealing are the same; only report.json's record of the harness files gains console.py.

Out of scope

  • Color in CI logs: CI runs without a terminal, so its logs stay plain.
  • The gateway and the Runner: their output is JSON, or logs that the harness parses and keeps as evidence.
  • DevGuard's colors and its release path, in a DevGuard pull request.

The qualification runs print long plain logs on the host's terminal, and
during a measurement interval of up to 30 minutes the harness printed
nothing, so a verdict was hard to pick out and a quiet run looked stuck.

On a terminal, the harness, validate-upstream.py and the checks now print
each verdict in color: green passed or qualified, red failed, yellow
inconclusive, incomplete or not run, and magenta interrupted. A status
line under the output shows what runs and for how long: the scenario, or
in a measurement interval the time into its planned length, the gateway
that serves and the load's runs. `--color auto|always|never` chooses, and
auto means a terminal unless NO_COLOR is set or TERM is dumb.

The harness's `--console-log <new file>` keeps a plain copy of everything
the run prints, without color or the status line, so a run started from a
terminal no longer needs a pipe to tee, which turned both off. The file is
created new and never written over.

Output to a file or a pipe is unchanged, byte for byte, and without a
terminal or a console log nothing replaces stdout or stderr. The
evidence, verify's JSON and every output a program reads stay plain.
scripts/console.py is standard library only and runs on Python 3.9.

Since code_tree() covers scripts/, later commits of main get another code
tree than the qualified candidate 1c0eadd; the recovery protocol's status
says so. The next qualification runs on this harness.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@novelKR novelKR self-assigned this Oct 19, 2026
@novelKR
novelKR merged commit 0ac3191 into main Oct 19, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants