Repository navigation
feat(runs): runs list and runs show over the runs root (#94 Q6) - #307
Conversation
… first bytes New package dokimi/runs, in the tracks layer. runs_root() is DOKIMI_RUNS_DIR when set, else results/ in the working directory, where most commands write by default. A missing default root means no runs yet; a DOKIMI_RUNS_DIR that does not exist, or a root that is not a directory, is an InputError. scan() walks the root once. save_artifact writes schema_version as the first key, so a file whose first 256 bytes open with the schema_version of a registered kind (current or superseded) is an artifact and is parsed; any other file is counted by why it was skipped (no leading schema_version, an unknown one, *.tmp, *.rejected, not .json, not a regular file such as a FIFO, a symlinked directory) and never parsed. A file that cannot be read, or whose body does not hold the version its head names, is listed as unreadable with its reason. Artifacts come newest first by the time they record, by mtime when they record none. No field is shared by every kind, so dokimi/runs/kinds.py names per kind where the subject, the model and the time are; a kind without one reads as empty, never as a guess from the path. A test fails when a kind of SCHEMA_VERSIONS has no row. Refs intel#278 Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
dokimi runs list prints one row per artifact under the runs root: schema_version, subject, model, recorded time (UTC, to the minute; the file's mtime marked * when the artifact records none) and the path relative to the root, newest first. Under the table a summary counts the artifacts, every file skipped by reason (unknown schema_versions by name) and the unreadable files with the first one's reason. --kind KIND (repeatable, a kind of SCHEMA_VERSIONS) and --subject (an exact match on one of the artifact's subjects) filter the rows. --json prints the rows with the intel#94 field names (schema_version, kind, run_id, subject, model, generated_at, path, size_bytes; run_id is null until intel#277) plus root, total, skipped, unknown_versions and unreadable. A missing default results/ prints "no runs" and exits 0. A DOKIMI_RUNS_DIR that does not exist or is not a directory, or a results/ that is not a directory, exits 2 through InputError. --subject is a new local flag, allowlisted with its reason; --kind was already on the list. The CLI snapshot is regenerated. Refs intel#278 Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
dokimi runs show PATH prints one artifact: its path, kind and schema_version, the result of its schema check, subject, model, time (the file's mtime when the artifact records none), size, the length of the lists it holds and the entries next to it, each .json one with the schema_version it opens with (the first 20 named, the rest counted). PATH is the path runs list prints, relative to the root, or an absolute or working-directory path that lies inside the root. The document is checked against the schema of the version it declares, so a superseded version is checked against its own schema. A failed check prints "schema: invalid, N errors" and every error and exits 0: showing is not a gate. A path outside the root, a missing file, a directory, an entry that is not a regular file (a FIFO), a file that does not open with the schema_version of a registered kind, or one that does not parse, exits 2. Control characters in the printed fields are escaped. --json prints the runs list row fields plus root, schema, schema_valid, schema_errors, contents, siblings, siblings_total and siblings_error. The kinds table gains the lists to count per kind; a test checks that each is a top-level array of the kind's schema. Refs intel#278 Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
docs/runs.md: what the runs root is and how DOKIMI_RUNS_DIR moves it, an example of each command, which file counts as an artifact and why the rest are counted but not read, where subject, model and time come from, what is not listed yet (no run_id grouping until intel#277; repo-runs/, answers/ and eval/ written without --run-id, and output flags pointing outside the root), the --json fields and the exit codes. The page goes in the mkdocs nav and the docs index, and quickstart's output layout points to it. The static benchmark's max_docs rises to 36 for the new page, as its spec asks. Refs intel#278 Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
github-actions
Bot
added
the
cli-interface-change
CLI interface change detectedThis PR changes dokimi's user-facing CLI surface (commands and/or options): + added command `dokimi runs`
+ option `--help/-h`
+ added command `dokimi runs list`
+ option `--help/-h`
+ option `--json`
+ option `--kind`
+ option `--subject`
+ added command `dokimi runs show`
+ argument `path`
+ option `--help/-h`
+ option `--json`Updated on every push; the |
|
✅ OpenCodeReview: Review complete: 0 finding(s) across 23 selected item(s). Review details
|
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #307 +/- ##
==========================================
+ Coverage 93.92% 93.98% +0.05%
==========================================
Files 284 289 +5
Lines 36767 37150 +383
Branches 6200 6242 +42
==========================================
+ Hits 34532 34914 +382
- Misses 1437 1439 +2
+ Partials 798 797 -1
🚀 New features to boost your workflow:
|
…t crash head_version keeps any character but an ASCII control, so an unknown schema_version could carry a Unicode line separator (U+2028) into the runs list summary and the runs show error. Both now go through the one escaping helper, plain(), moved to dokimi/runs/scan.py. tmpfs keeps an mtime past the year 9999, where datetime.fromtimestamp raises: the text of runs list and runs show crashed on an untimed artifact with such an mtime (--json did not). The time column and the show line now print it as @<seconds>. Refs intel#278 Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
On Python 3.11-3.13 Path.exists(), is_dir() and is_file() raise PermissionError for a path under a directory that cannot be searched. RunsRoot.exists() (a DOKIMI_RUNS_DIR below such a directory) and runs show's resolve() (an address in such a directory) let it out, and the command crashed with a traceback and exit 1. Both now raise InputError with the reason, so the command prints it and exits 2. Python 3.14 answers False there, and the message is the existing "does not exist" / "no file"; the tests accept either. Refs intel#278 Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
runs show checked that the address lay inside the root as spelled, so results/linked -> /elsewhere let runs show linked/eval.json read a file outside the root (and list its neighbours), and runs list read a file symlink that pointed outside. intel#278 lists only what is under the root. The rule is now the same for both commands: a file is read only when its real path lies inside the real root. runs list counts a file symlink that leads out as skipped (linked_outside, "symlinks to files outside the root (not read)"); runs show exits 2 with "... leads outside the runs root"; a neighbour that links out is named but its head is not read. A symlink inside the root, and a root that is itself a symlink, work as before, and runs show keeps the name the address used. Symlinked directories are still not walked by runs list. Refs intel#278 Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
…pened Two documented behaviours had no test (Codecov on intel#307): runs list prints a recorded time it cannot parse as written, and runs show exits 2 on an artifact it may not open. The open error now takes its wording from read_error_reason, as every other read error does; the message is unchanged. Refs intel#278 Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
…n fail to docs/runs.md said a file that does not parse is listed as unreadable. Only a file that opens with a registered schema_version is parsed; a broken file of no registered kind is counted as skipped. Say so, and that the text summary names the first unreadable file while --json lists all. Refs intel#278 Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
cli.main prints an InputError as it is, so a runs show address or a DOKIMI_RUNS_DIR with a newline or an escape sequence reached stderr raw. Every message of resolve(), _version(), show(), RunsRoot.exists() and runs show's missing-root error now names them through plain(); the paths used to read files are unchanged. Refs intel#278 Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
A junction is not a symlink, and os.walk(followlinks=False) still descends into one on Python 3.11-3.13, so runs list read and listed the files of a junction that points outside the root (runs show already refused them by real path), and a junction back into the root would be walked twice. A directory that is a symlink or a junction (st_reparse_tag IO_REPARSE_TAG_MOUNT_POINT, which 3.11 reads too) is now pruned before the walk descends and counted as "symlinked directories and junctions (not walked)". The test runs on Windows only. Refs intel#278 Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
resolve() checked the address as the filesystem resolves it, then made it absolute with os.path.abspath, which folds "link/.." before the symlink is followed: with results/link -> results/nested/deep, runs show link/../eval.json checked results/nested/eval.json and read results/eval.json. The path is now made absolute with Path.absolute(), which keeps "..", and an address with ".." is shown and read by its real path. Refs intel#278 Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
The test nested 100 000 arrays to make json raise RecursionError. On Python 3.14 the C parser is bounded by the C stack, not the recursion limit, and parses that document, so the test failed on CI's 3.14 leg while the scanner was right. The parse now raises RecursionError through a patched load_document, and the test checks that the file is listed as unreadable with that reason, on every interpreter. Refs intel#278 Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
docs/runs.md and the changelog fragment said a file is an artifact when it opens with a registered schema_version. head_version reads only the first 256 bytes, so a file whose first key comes later (after 1000 spaces, say) is skipped. Say that; save_artifact's output always fits. Refs intel#278 Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
resolve() accepts an address whose real path is inside the root, and keeps the spelled path for display. With results/linked -> /elsewhere and /elsewhere/return.json -> results/real.json, runs show linked/return.json was accepted, and its neighbours were listed from /elsewhere: outside file names and schema_versions in the output. When the spelled directory does not resolve inside the root, the neighbours now come from the artifact's own (real) directory. Refs intel#278 Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
| if ".." not in spelled.parts and inside(root, spelled): # keep the name runs list prints | ||
| return spelled, root.absolute() |
There was a problem hiding this comment.
Must-fix: account for .. in the root as well as in the address. For example, from /tmp/work, with an artifact at /tmp/results/e.json, DOKIMI_RUNS_DIR=../results python cli.py runs show /tmp/results/e.json passes the containment checks but returns /tmp/results/e.json alongside /tmp/work/../results. read_artifact() then calls path.relative_to(root), which is lexical and raises ValueError, so a valid artifact exits 2. Use the resolved-path fallback when the absolute root contains .., and add a regression test for an absolute address with a parent-relative runs root.
There was a problem hiding this comment.
Fixed in 34e76e1. Reproduced: from work/ with DOKIMI_RUNS_DIR=../results, runs show <absolute path of results/e.json> exited 2 with the relative_to error, while runs show e.json worked. A .. in the root now takes the real-path branch as a .. in the address does, so both addresses show e.json. Regression test added; the runs tests also pass on Windows.
…h .. resolve() kept the spelled path when the address had no "..", and returned it with root.absolute(). For a root such as DOKIMI_RUNS_DIR=../results that base still holds "..", and read_artifact's lexical relative_to failed: an absolute address of a valid artifact exited 2. A ".." in the root now takes the real-path branch too, so the address is shown relative to the real root. Refs intel#278 Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
napetrov
left a comment
napetrov
left a comment
There was a problem hiding this comment.
Review at head 34e76e1 (delta 33f8422..34e76e1: bdc14ac, 942ecdf, 2f31691, 34e76e1; no merges of main in the range).
The fix in 34e76e1 holds by probe: with DOKIMI_RUNS_DIR=../results, an absolute address under the root now exits 0 (it exited 2 at 33f8422). Symlinks that lead outside the root are still refused. Neighbour listing from inside the root (2f31691) is covered by its test.
Tests on the head: runs show, runs scan, runs cli, cli flags, cli surface and the import-architecture test pass (208 passed, 2 skipped). ruff check and mypy are clean on the touched runs modules.
The four behaviours I checked are fine, each with a docs note:
stampwritesschema_versionlast, so results from readiness, environment and budget are not listed byruns list. Docs note: one sentence under "Nothing is dropped silently" saying that such files are skipped as having no leadingschema_version.--subjectis a row filter, not an input. Docs note: one sentence next to the--kind/--subjectbullet saying it filters rows and selects no subject for other commands.runs showon a*.rejectedfile exits 0 and checks the schema, whileruns listskips*.rejectedby name. Docs note: one sentence in theruns showparagraph saying a*.rejectedpayload is readable byshowand exits 0.- A path through a symlinked directory inside the root works with
show, and the path it prints depends on how the root is spelled (as spelled for an absolute root, canonical for a root spelled with..);runs listnever prints the symlinked path. Docs note: one sentence saying the printed path can differ from theruns listpath.
One inline nit, still present at this head and not introduced by the delta. Not blocking.
| if stamp: | ||
| return stamp.strftime("%Y-%m-%d %H:%M") | ||
| if artifact.generated_at: # not ISO 8601, or out of range: shown as written | ||
| return artifact.generated_at |
There was a problem hiding this comment.
nit: _when returns the raw generated_at string for a time that is not ISO 8601, with no *. The mtime fallback below it carries one. runs list sorts such a row by its file mtime, so a row that prints not a time can sort above a row recorded 2026-10-29 (probe: q1.json with generated_at: "not a time" and today's mtime, against arms/a.json). Suggest marking the raw string too, or showing the mtime with *.
napetrov
left a comment
napetrov
left a comment
There was a problem hiding this comment.
Checked against #94 (§2.3, Q6, R1–R3, J11, F2) and #278. Aligned overall: runs root results/ + DOKIMI_RUNS_DIR, scan with no index (the index is an optional cache per Q6), pre-envelope files listed (R1), --kind/--subject, exit 2 for bad input, show reads and does not gate, --root banned, #94 field names. "Refs" rather than "Fixes" #278 is right.
Two things to settle before merge, so later stages don't need a breaking change:
-
runs showpositional. Here it is a file path; #94 J11 hasruns show $RUN(a run, later with anexplainsection). Once #277 bringsrun_id, the same positional would need to take both a run and a file. Could we decide now how a run is named, e.g.runs show RUNplusruns show --artifact PATH, or a documented rule for telling the two apart? Otherwise the switch needs a tombstone later. -
--jsonbeforecli_output.v1. main has no envelope yet, so this is fine today, but the raw shape (root,artifacts,skipped, ...) becomes something scripts depend on until F2. Please add one line todocs/runs.mdsaying the shape is pre-envelope and F2 wraps it, keeping the field names.
Nit: the text table heads the column SCHEMA, but the filter is --kind; a KIND column would match the flag.
|
Heads-up on merge order: this conflicts with #284 (#94 A1) in |
Brings in intel#284 (route table), intel#254 (flag registry, CLI snapshot) and the cli_routes.v1 artifact from main. Resolved: - cli.py: runs and data both register; the table lists runs. - cli_routes.py: runs is a root in "Analysis and data" with runs list and runs show as free leaves. - runs/kinds.py: cli_routes.v1 gets its counted lists, so runs show and runs list handle the new artifact kind. - Tests: the expected route sections, the scan and show kind tables include cli_routes.v1. - cli-interface.json regenerated with make cli-snapshot. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
Merged
Full suite on the merge (py3.13 venv): 11041 passed, 7 skipped. Your commits are unchanged. |
| shown = Shown(artifact=artifact, schema=schema, errors=schema_errors(schema, data), | ||
| contents=contents(kind, data)) |
There was a problem hiding this comment.
Must-fix: schema validation can still raise an unhandled RecursionError after JSON loading succeeds. For example, an arms.v1 document with a deeply nested array in model can parse successfully but exceed the recursion limit when jsonschema formats that value for its string-type error. schema_errors() does not catch this, and the CLI catches only InputError/EmptyQuestionError, so runs show emits a traceback instead of a controlled diagnostic. Handle recursion failures around validation too, and add a test covering validation-time recursion (the scan test only covers loading).
|
Conflicts were additive: keep both the runs root (intel#307) and the backends root (main) in cli.py, ROOTS/LEAVES of cli_routes.py and the Analysis-and-data help test, runs first. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
test_a_junction_is_counted_and_not_walked skips off Windows, so no CI job ran the only test of the runs-root containment on Windows. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
|
Main reached 36 docs with intel#255, and this branch adds docs/runs.md, so the merged tree has 37. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
|
Summary
Refs #278, refs #94 (§2.3, Q6). Two new free, read-only commands over the runs root (
results/in the working directory, orDOKIMI_RUNS_DIR):runs list: one row per artifact (schema_version, subject, model, time, path), newest first;--kind(repeatable),--subject,--json. A summary counts every skipped file by reason and names the unreadable ones.runs show PATH: one artifact with its kind, schema check (errors listed), fields, list lengths and neighbouring files;--json.Writers do not change and nothing is written (no
index.jsonl). New packagedokimi/runs/(tracks layer:scan.py,kinds.py,show.py), CLI incommands/runs.py, docs indocs/runs.md.ResultsAggregatoris not reused: it reads eval files only, and parses them. NotFixes #278: grouping byrun_idwaits for #277.Four commits, each green on its own: scan,
runs list,runs show, docs.Evidence
Speed (500 artifacts;
scan()in process): 495 small artifacts take 0.04 s. Adding 5 artifacts of 58 MB takes 0.87 s, because each is parsed once. 5 non-artifact JSON files of 50 MB take 0.0002 s, because only their first 256 bytes are read.runs listwall time is 0.8 s and 1.7 s, mostly CLI start-up (dokimi --version0.76 s), so no cache is needed.Bad input, run by hand.
DOKIMI_RUNS_DIRthat is missing or a file.results/.runs show: exit 0 on a schema-invalid artifact. Exit 2 on a non-artifact, an unknown version, a directory, a FIFO,../x.json,/etc/passwd, or a truncated file.Tests.
test_runs_scan.py,test_runs_cli.pyandtest_runs_show.pyrun offline through the CLI.--json, and the schema check of a superseded version.Checks.
showexiting 1 on an invalid schema, no inside-the-root check.\paths, other drives and..\behave as on Linux.make ci-local: 0 failed (10691 passed).Decisions
Each choice below is in the code already; the link points at it.
Choices that change visible behaviour if revisited
--jsonfield names (runs/scan.py:100): the CLI API: the evaluation loop at the top level, subject and eval groups below, one enforced flag vocabulary (proposal v2.1) #94 vocabulary, i.e.schema_version,kind,run_id,subject,model,generated_at(evaluated_atfor an eval),path,size_bytes.show --jsonadds its own fields. F2 wraps the output incli_output.v1and keeps them.runs/scan.py:43): a missing defaultresults/means "no runs" and exits 0. ADOKIMI_RUNS_DIRthat does not exist or is a file exits 2, as bad input does since fix(cli): input and usage refusals exit 2, not 1 (#94 A2) #285.showon a schema-invalid artifact (commands/runs.py:131) exits 0 and lists the errors: it reads, it does not gate. Verdicts stay with §2.6 / J11/J12: distinct inconclusive verdict and blocking policy (AC-J11-002, AC-J12-002) #99.Choices that can only be extended later
run_idyet, sorun_idisnulluntil Artifact envelope for run artifacts, not inputs (#94 B1/B1b) #277.--verdict: almost no artifact has a verdict beforeresult.v1.repo-runs/(noschema_version; R2 moves it), and the default outputs ofanswers/evalwithout--run-id,eval panel-score,questions generate|refine,repo reachandtools preference.docs/runs.mdnames them all.DOKIMI_RUNS_DIR:--rootis banned incli_flags.py. A positional root can come later.runs/scan.py:143):save_artifactwritesschema_versionfirst, and only known kinds are parsed. Artifacts saved before refactor(artifacts): write answers, eval and arms through save_artifact #27 are counted as "no leading schema_version".runs/kinds.py:35): a table names the field for each kind. Without such a field the column shows-, never a guess from the path. Answers use their records' sharedlibrary_name. When an artifact has several values, all are kept, and--subjectmatches any of them.--subjectmatches exactly (case-sensitive).showtakes a path inside the root: relative to the root, absolute, or from the working directory. Addressing byrun_idcan come after Artifact envelope for run artifacts, not inputs (#94 B1/B1b) #277.index.jsonl: the scan is fast enough (Evidence).These are the defaults this PR picks, not a settled contract: any of them can change here before merge, or move to #94/#278.
Merge Danger
Door: two-way. New commands and package; no existing behaviour or schema changes.
Blast Radius: only the
runsgroup. Shared files get small edits:cli.py: one import and oneregister.dokimi/__init__.pydocstring, the layer table and the CLI snapshot.--subjectadded, with its reason.max_docs: 35 → 36, as its spec asks.Overlap:
cli_routes.pyneedsRoot("runs", "The loop")andLeaf(("runs", "list"|"show"), FREE).FIELDSand one in the test.cli.pyimports, a trivial conflict.Whichever PR merges second makes these edits.
Not in this PR
run_id,--verdict,runs explain|resume|replay|export|import|rm|gc|baseline, andindex.jsonl.cli_output.v1(F2).Found while testing, in code this PR does not change:
artifacts.stamp()putsschema_versionlast. As a result, the--jsonoutput ofrepo readiness|environment|budgetandtasks inspect, saved underresults/, is not listed.questions generaterecords noproduct, so--subjectcannot find those sets.Checklist
pytestpasses (10691 passed, 57 skipped)ruff,mypy, and schema checks pass (make ci-local)docs/runs.md, nav and index; docs code-reference test andmkdocs build --strictpassAGENTS.md: mutations, bad input, Windows run, an independent review pass