Repository navigation
Expand file tree
/
Copy patherrors_coverage.py
More file actions
887 lines (782 loc) · 41.4 KB
/
Copy patherrors_coverage.py
File metadata and controls
887 lines (782 loc) · 41.4 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
#!/usr/bin/env python3
"""errors_coverage — does every catalogued error name its fix?
`errors name the fix` has been a language-level commitment since unbound v1.1
§22 — the highest-leverage machine-authorship affordance in Planes, because a
message that reports a token mismatch without saying what to write instead
leaves an author (human or machine) with a true statement and no next move.
S8 counted it for the first time. C2 closed the shortfall and gave the
measurement a floor.
WHAT IS MEASURED — the inclusion rule, stated (C2, A.2 item 2)
The catalogue is `grammar/errors.json`, generated by `grammar_gen.py` from the
raise sites themselves, so this measures the code and not a hand-kept list.
Each entry carries a `kind`, and the rule for it lives in `grammar_gen.py`
beside the class list it governs:
error the entry constructs one of the seven error classes the reference
implementation defines. Measured here.
report the entry is text a plane renders for a person without raising —
rules.py's `render_violation` and `_render_vacuous_text` (B4;
`Violation.render()`/`Violation.as_json()` delegate to these).
Counted and listed, never measured: an error stops a program and
leaves its author needing a next move, and a report does not.
Four report lines used to be counted as passes, which inflated the 70 S8
measured. They are not errors, and the number that matters is the one that is
a work list.
THREE STATES, AND WHY NOT A PERCENTAGE (C2, A.3)
names a fix the entry carries a `fix` field (the third
constructor argument), or its message template
runs to more than one line — the continuation
clause every fix-naming message in this repo
uses. A site that delegates to
grammar/messages/amber.json counts here only if
every template it delegates to carries its own
`fix`; the delegating share is reported
separately so the dependency stays visible.
deliberately names none the raise site states, in words, why it names
none — the `no_fix` argument, read as a literal
at the site exactly as the message is. Two
shapes qualify: a message the language did not
write (`fail`'s own text, an `or fail` re-tag),
and a gate too generic to know what the author
meant (`expect`). Marked, never silent.
should name one and does not the work list. Its target is zero — not that
the first state reaches 100%, which it cannot
once deliberate silences exist and would be
the wrong thing to chase if it could.
An entry whose message is not a literal at the raise site is *unreadable* to
this checker. That is not a state of its own: unreadable-with-a-stated-reason
is a decision, and unreadable-without-one is shortfall. It is reported as a
figure so it stays visible, and it is never counted as a pass.
It REPORTS; it never fails. A message with no fix clause is a fact to weigh
and schedule, not a build break — and gating CI on it would mean the first
person to add an honest one-line error has to write prose before they can
commit. Same contract as corpus_coverage.py: this always exits 0.
python3 errors_coverage.py # the report
python3 errors_coverage.py --json # machine-readable
"""
import json
import os
import sys
REPO = os.path.dirname(os.path.abspath(__file__))
ERRORS_PATH = os.path.join(REPO, "grammar", "errors.json")
AMBER_PATH = os.path.join(REPO, "grammar", "messages", "amber.json")
# The shape every fix-naming message in this repo uses: the detail line, then a
# continuation clause on its own line. `check_binding_name`'s builtin-collision
# message is the reference instance. The test is "more than one line" rather
# than the exact two-space indent, because a message composed across branches
# (rules.py's vacuous-rule reports) is catalogued as its raw pieces and the
# indent the renderer adds is not part of the template.
FIX_CONTINUATION = "\n"
# The same clause, as it is written in Planes source: a literal backslash-n
# inside a string, not a real newline. `classify()` reads the reference's
# rendered template; the self-hosted scanner reads source text.
FIX_CONTINUATION_LITERAL = "\\n "
NAMES_FIX = "names a fix"
DELIBERATE = "deliberately names none"
SHORTFALL = "should name one and does not"
STATES = (NAMES_FIX, DELIBERATE, SHORTFALL)
ERROR_KIND = "error"
REPORT_KIND = "report"
# ============================================ a fourth dimension: intent (A-Q19)
#
# The three states above answer "does this site's message carry a fix
# clause at all". They do not answer a different question a fix clause can
# still fail: "does the fix it names hold in EVERY situation this site can
# fire from" — `cannot-combine`'s old clause named a real fix (`text of`)
# that was the *wrong* one whenever the site fired from the arithmetic
# side rather than the text-building side, and it counted as NAMES_FIX
# the whole time. That defect is invisible to a naming check and visible
# only to someone reading the site against every way it can be reached.
#
# This is deliberately not an audit of the other ~110 sites — the standing
# anti-escalation rule gives a found instance a point fix, not a mandate to
# re-read the catalogue. What a single instance CAN do cheaply is make the
# next one visible: every site is either ASSESSED (a build looked at it
# against this question and recorded the answer, changing it or not) or
# UNASSESSED (no build has). The count is reported, never a work list —
# unassessed is the honest default for a site nobody has looked at yet, not
# a claim that it is wrong.
#
# Keyed by the reference catalogue's own `id` (grammar/errors.json) for the
# reference side, and by (file, function) for the self-hosted side, where
# `self_hosted_sites()` has no single stable id.
ASSESSED_THIS_BUILD = frozenset({
# A-Q19's own site: the fix corrected to name both directions.
"interp.cannot-combine.apply_op",
# Read alongside it because it makes the same claim `number of` now
# falsifies -- "Planes has no text-to-number builtin" stopped being
# true the moment this build shipped one.
"interp.not-a-number.builtin-1",
# The three new sites `number of` itself raises from.
"interp.not-text.builtin",
"interp.not-a-number.builtin-2",
"interp.not-a-number.builtin-3",
# Every other site this build found by searching for `text of` as a
# named remedy (§3.2's literal instruction) and read against whether
# the fix holds in every situation the site fires from. Each of these
# is genuinely one-directional (the site only ever wants text), so
# none needed to change -- assessed and left alone is still assessed.
"interp.not-text.require_text", # lower / upper / normalize
"interp.not-text.require_target", # ask / read / write's target
"interp.not-text.membership", # `in` over text
"interp.cannot-join.builtin-2", # join's per-item guard
"interp.fail-message-not-text.exec_stmt-1", # fail's message field
# Not a `text of` site, but read for the same reason `whole`'s was:
# it is the other generic arithmetic guard next to the builtin this
# build added, and the same question applies. Left unchanged because
# it names no specific conversion to get wrong.
"interp.not-a-number.arith",
# host-error-reachability-and-fix-clauses: reports/REPORT_ERROR_MESSAGES.md
# read the first 60 catalogued sites against this exact question and
# found nine whose fix misdirected for at least one situation the site
# can fire from. Each was corrected; `interp.cannot-combine.apply_op` and
# `interp.not-text.membership` above already carry this build's fix
# (independently found by both audits) and are not repeated here.
"interp.cannot-compare.equal-1", # nothing: the nested-path case
"interp.cannot-compare.equal-2", # nothing: the whole-value case
"interp.cannot-compare.equal-4", # equal(): list/record, not just num/text
"interp.not-a-yes-no.condition", # and/or/not/where, not just `if`
"interp.unrecognized-record-format.records_from_json", # which side is stale
"interp.write-failed.eval", # OSError has more causes than ENOENT/EACCES
"interp.recursion-too-deep.call", # plain numeric recursion has no collection
"interp.not-a-number.builtin-4", # sine: not itself wrong, made complete
"lexer.grammar-data-missing._load_vocabulary-3", # same stale-side question as above
# Read alongside them and confirmed correct as written, not changed:
# compare()'s guard (`<` `>` `<=` `>=`) shares equal-4's old (pre-F7-split)
# text, but only numbers and text are ever orderable, same-typed or not —
# so the audit's fix belongs to equal() only, and this site is assessed
# and left alone, same as C2's own precedent above.
"interp.cannot-compare.compare",
})
SELF_HOSTED_ASSESSED_THIS_BUILD = frozenset({
("interp.planes", "apply-plus"),
("interp.planes", "builtin-whole"),
("interp.planes", "builtin-number-of"),
# host-error-reachability-and-fix-clauses (see the reference set above).
("interp.planes", "values-equal"),
("interp.planes", "as-condition"),
("interp.planes", "member-of"),
("interp.planes", "builtin-sine"),
("interp.planes", "compare-values"),
})
def intent_assessment(entries, assessed_ids):
"""How many of `entries` (each carrying an `id`) are in `assessed_ids`.
Returns {"assessed": N, "unassessed": N} — reported, never a work list;
see the module note above ASSESSED_THIS_BUILD."""
assessed = sum(1 for e in entries if e["id"] in assessed_ids)
return {"assessed": assessed, "unassessed": len(entries) - assessed}
def self_hosted_intent_assessment(sites, assessed_keys):
"""The same question, for the self-hosted sites keyed by (file, function)
instead of an id string."""
all_sites = [s for v in sites.values() for s in v]
assessed = sum(1 for s in all_sites if (s[0], s[4]) in assessed_keys)
return {"assessed": assessed, "unassessed": len(all_sites) - assessed}
def load(path):
with open(path, encoding="utf-8") as f:
return json.load(f)
def amber_templates_all_name_a_fix(amber):
"""Whether every amber template carries its own `fix`. A delegation is
only as good as what it delegates to."""
templates = amber.get("templates", [])
return bool(templates) and all(t.get("fix") for t in templates)
def delegates(entry):
return "amber.json" in (entry.get("note") or "")
def classify(entry, amber_ok):
"""One of the three states. Order matters: a site that names a fix names
one whether or not it also carries a `no_fix` reason, and a stated reason
only settles a site that names none."""
if entry.get("fix"):
return NAMES_FIX
template = entry.get("template")
if template and FIX_CONTINUATION in template:
return NAMES_FIX
if delegates(entry):
return NAMES_FIX if amber_ok else SHORTFALL
if entry.get("no_fix"):
return DELIBERATE
return SHORTFALL
def is_unreadable(entry):
"""The message is not a literal at the raise site, so this checker cannot
read whether it names a fix. A figure, not a state — see the module
docstring."""
if entry.get("fix") or delegates(entry):
return False
return entry.get("template") is None
def coverage():
cat = load(ERRORS_PATH)
amber_ok = amber_templates_all_name_a_fix(load(AMBER_PATH))
errors = [e for e in cat["entries"]
if e.get("kind", ERROR_KIND) == ERROR_KIND]
reports = [e for e in cat["entries"] if e.get("kind") == REPORT_KIND]
buckets = {s: [] for s in STATES}
for e in errors:
buckets[classify(e, amber_ok)].append(e)
def brief(e, *extra):
out = {"id": e["id"], "source": e["source"], "class": e["class"]}
for k in extra:
out[k] = e.get(k)
return out
return {
"total": len(cat["entries"]),
"errors": len(errors),
"reports": [brief(e) for e in reports],
"amber_templates_all_name_a_fix": amber_ok,
"delegating": sum(1 for e in errors if delegates(e)),
"counts": {s: len(buckets[s]) for s in STATES},
"work_list": [brief(e, "template") | {"unreadable": is_unreadable(e)}
for e in buckets[SHORTFALL]],
"deliberate": [brief(e, "no_fix") for e in buckets[DELIBERATE]],
"unreadable": [brief(e) for e in errors if is_unreadable(e)],
"intent": intent_assessment(errors, ASSESSED_THIS_BUILD),
}
# ============================================ the self-hosted stack (C4 / §158)
#
# Everything above measures the reference implementation, whose raise sites are
# Python. REPORT_DETAIL_CONVERGENCE.md §6 recorded that `grammar/interp.planes`
# "cannot name a fix anywhere" and was outside the commitment as measured —
# correctly, because there was no slot to name one in. §158 opened the slot:
# `error-fix-of` exists, so the self-hosted files are now inside the
# commitment and can be counted against it for the first time.
#
# The figure is reported on its own and is NOT merged into the reference's
# work list. Two reasons. The reference's list is at zero and merging would
# hide that; and driving this one to zero is a body of message-writing work
# with its own ruling, not a side effect of opening the slot. A non-zero
# self-hosted work list is the honest output of the build that made it
# measurable.
SELF_HOSTED_GLOB = ("interp.planes", "parser.planes", "lexer.planes",
"json.planes")
def self_hosted_sites():
"""Every place a `grammar/*.planes` file raises, and whether it names a fix.
Three shapes, read off the source the same way grammar_gen.py reads the
Python raise sites — structurally, from the call, not from a hand-kept list:
error-fix-of of "tag", <detail>, <fix> names a fix
error-of of "tag", <detail> names none
fail <expr> as <tag> depends on <expr>
The last one splits. `fail d as unknown-name` re-raises a detail some other
site already wrote, which is the same shape as `or fail`'s re-tag and is
marked deliberate for the same stated reason: the message belongs to
whoever raised it. `fail "..." as tag` writes its own message here and has
nowhere put a fix, so it is shortfall.
Each site is `(file, line, source, tag, function)`.
The TAG is read from the call — a literal at `error-of of "..."`, the target
of `fail ... as <tag>` — and is None where the site names it dynamically
(`error-of of stmt.tag`). C5 needs it to split the shortfall against the
reference catalogue; None is carried rather than guessed, and reported as
its own figure.
The FUNCTION is the enclosing `to <name>`, tracked while walking. Every
definition in these files is at column 0 — Planes has no nested `to` — so
the nearest preceding one at lower indentation is simply the nearest
preceding one. C6 needs it for the second matching pass, because half the
reference catalogue has no tag to be found by.
"""
import re
sites = {NAMES_FIX: [], DELIBERATE: [], SHORTFALL: []}
for name in SELF_HOSTED_GLOB:
path = os.path.join(REPO, "grammar", name)
if not os.path.exists(path):
continue
fn = None
continuing = _continuation_helpers(path)
for n, line in enumerate(open(path, encoding="utf-8"), 1):
if line.lstrip().startswith("#"):
continue
defn = re.match(r"^to ([\w-]+)", line)
if defn:
fn = defn.group(1)
src = line.strip()[:96]
for m in re.finditer(r"\berror-fix-of of ", line):
if _defines(line, m.start()):
continue
sites[NAMES_FIX].append(
(name, n, src, _tag_at(line, m.end()), fn))
for m in re.finditer(r"(?<!-)\berror-of of ", line):
if _defines(line, m.start()):
continue
sites[SHORTFALL].append(
(name, n, src, _tag_at(line, m.end()), fn))
# D: the self-hosted counterpart of the reference's `no_fix`. The
# reason is a literal at the site, exactly as the message is, and
# is never rendered — it exists so a deliberate silence can be told
# from a gap on this side too.
for m in re.finditer(r"\berror-no-fix-of of ", line):
if _defines(line, m.start()):
continue
sites[DELIBERATE].append(
(name, n, src, _tag_at(line, m.end()), fn))
m = re.match(r"\s*fail\s+(.*?)\s+as\s+([\w-]+)\s*$", line)
if m:
arg = m.group(1)
# A bare name re-raises a message written elsewhere; anything
# else (a literal, a concatenation) is a message this site
# wrote.
#
# D: and a message this site wrote can now name a fix, through
# §158's record form — `fail { message: ..., fix: ... } as tag`
# is the ONLY way a `fail` site can name one, and this scanner
# predated the form and could not see it. Without this branch
# the work list could never reach zero for the ~30 `fail`-shaped
# sites however many clauses were written.
deliberate = re.fullmatch(r"[A-Za-z][\w-]*(\.[\w-]+)*", arg)
if deliberate:
bucket = DELIBERATE
elif arg.lstrip().startswith("{") and re.search(r"(?<!-)\bfix:",
arg):
bucket = NAMES_FIX
elif arg.lstrip().startswith("{") and re.search(r"\bno-fix:",
arg):
# The self-hosted `no_fix`: a reason, as a literal at the
# site. Ignored by all three interpreters exactly as the
# reference's is never rendered, so the message is
# unchanged and the silence is marked rather than silent.
bucket = DELIBERATE
elif _names_fix_inline(arg, continuing):
bucket = NAMES_FIX
else:
bucket = SHORTFALL
sites[bucket].append((name, n, src, m.group(2), fn))
return sites
def _continuation_helpers(path):
"""Functions in this file whose body writes a fix on a continuation line.
A syntax error names its fix the way the reference does — inline, after a
`\n ` — rather than in a field, because both implementations render one
string and the two are asserted byte-identical. `classify()` already reads
exactly this shape on the reference side (FIX_CONTINUATION). The self-hosted
half could not, so a message composed in a helper (`fail
unterminated-string-message of line as ...`) read as a silence when the
clause was right there. One hop, resolved structurally, same as everything
else this scanner reads.
"""
import re
helpers, current, body = set(), None, []
with open(path, encoding="utf-8") as fh:
for line in fh:
m = re.match(r"^to ([\w-]+)", line)
if m:
if current and any(FIX_CONTINUATION_LITERAL in b for b in body):
helpers.add(current)
current, body = m.group(1), []
body.append(line)
if current and any(FIX_CONTINUATION_LITERAL in b for b in body):
helpers.add(current)
return helpers
def _names_fix_inline(arg, helpers):
"""Whether a `fail <arg> as tag` message carries its fix on a continuation
line — written here, or written by the one helper it calls."""
import re
if FIX_CONTINUATION_LITERAL in arg:
return True
head = re.match(r"\s*([A-Za-z][\w-]*)", arg)
return bool(head) and head.group(1) in helpers
def _defines(line, start):
"""Whether the match at `start` is a function DEFINITION, not a call.
C6 / Ruling 1. `to error-of of tag, detail:` and `to error-fix-of of tag,
detail, fix:` are the definitions of the two helpers this scanner looks
for, and it matched both — the first into the shortfall, the second into
names-a-fix. Two errors in opposite directions, which is why 113 and 72
looked stable across two builds while each was one high. `_tag_at`'s
docstring already named the case; nothing acted on it.
The test is the general one and not a list of two names: a match defines if
everything before it on the line is exactly `to`. It deliberately does NOT
skip every line beginning `to `, because Planes allows a single inline
statement as a function body, so `to f of x: give fail-of of (error-of of
"y", z), env` is a real raise site on a `to` line and must stay counted.
"""
return line[:start].strip() == "to"
def _tag_at(line, pos):
"""The tag a `... of ` call names at `pos`, or None if it is not a literal.
`error-of of "not-a-record", ...` reads; `error-of of stmt.tag, ...` does
not, and neither does the two-line definition `to error-of of tag, detail:`.
"""
import re
m = re.match(r'"([^"]+)"', line[pos:])
return m.group(1) if m else None
# ------------------------------------------- the shortfall, split (C5 / Ruling 2)
#
# 72 of 113 self-hosted raise sites name no fix, and that arrived as one number.
# It is two bodies of work, and only one of them needs anybody to write prose.
#
# The premise of the three-implementation convergence is that the answer does
# not depend on which implementation ran the program. So any self-hosted message
# whose tag the reference also raises with a catalogued `fix` already has a
# clause written, tested, and byte-identical on the Python side; porting it is
# mechanical. Only a site with no reference twin needs authorship.
#
# THE MATCH IS ON THE TAG, AND A TAG IS NOT A MESSAGE
#
# Tags are deliberately shared across many messages
# (REPORT_DETAIL_CONVERGENCE.md §4 item 4 is explicit about this): ten of them
# here map to more than one catalogued entry. A tag match is therefore evidence
# that *a* clause exists to port, not proof it is the right clause for this
# site. The multiplicity is reported as its own figure rather than resolved, so
# the build that reads this number decides per site and is never told the
# mapping is one-to-one when it is not.
#
# This produces a number. It ports nothing, and it does not merge into the
# reference's work list, which is at zero.
#
# ------------------------------------- the second key: the function (C6/Ruling 2)
#
# C5's own finding was that a matcher whose key half the catalogue lacks is a
# matcher that understates, and it left the 28 standing as a ceiling rather
# than a measurement: 51 of the 109 catalogued reference errors carry no tag,
# so `grammar/parser.planes`'s `fail "line 3: ..." as parse-error` could not
# match `parser.py`'s syntax message however nearly identical the two are.
#
# Since #29 a catalogue id carries the enclosing function, promoted from the
# `raised_in` field the entry already held. The self-hosted files are ports of
# the reference and their functions carry corresponding names across one
# spelling convention — `parser.py`'s `read_effect_word` against
# `grammar/parser.planes`'s `read-effect-word`. So a second pass runs over
# whatever the tag pass did not match, keyed on that name normalised across
# `_` and `-`.
#
# A FUNCTION MATCH IS WEAKER EVIDENCE THAN A TAG MATCH and gets its own state
# rather than being folded into the first. The correspondence is inferred from
# a naming convention that nothing asserts, and a reference function may hold
# several raises, so multiplicity bites harder here — both figures are reported.
# A matcher that hides which key it used is worse than one that understates,
# because the next build has to decide per site and must see which evidence it
# is deciding on.
HAS_TWIN = "has a reference twin"
PROBABLE_TWIN = "has a probable twin"
NO_TWIN = "no reference twin"
def _normalise_fn(name):
"""One spelling convention across two languages. `_resolve_subject` and
`resolve-subject` are the same function; a leading underscore is Python's
private marker and carries no meaning in Planes."""
return (name or "").replace("_", "-").strip("-")
def reference_fix_tags():
"""tag -> the catalogued reference entries that raise it AND name a fix.
Only PlanesError and GrammarDataError carry a tag, and four PlanesError
sites do not carry one either. Everything else — 29 PlanesSyntaxError
entries, 7 RuleConflict, 6 PlanesAmbiguity, 3 ModuleError, 2
RuleNotSupported — has no tag at all, so those entries are not addressable
by tag and cannot be twins however close the message is.
`untagged_reference_entries()` reports how many that is, because a matcher
whose key half the catalogue does not have is a matcher that understates.
"""
cat = load(ERRORS_PATH)
amber_ok = amber_templates_all_name_a_fix(load(AMBER_PATH))
twins: dict[str, list[str]] = {}
for e in cat["entries"]:
if e.get("kind", ERROR_KIND) != ERROR_KIND:
continue
if e.get("tag") and classify(e, amber_ok) == NAMES_FIX:
twins.setdefault(e["tag"], []).append(e["id"])
return twins
def reference_fix_functions():
"""normalised function name -> the catalogued entries raised there that name
a fix. The second key, and the one that reaches the untagged half."""
cat = load(ERRORS_PATH)
amber_ok = amber_templates_all_name_a_fix(load(AMBER_PATH))
fns: dict[str, list[str]] = {}
for e in cat["entries"]:
if e.get("kind", ERROR_KIND) != ERROR_KIND:
continue
if e.get("raised_in") and classify(e, amber_ok) == NAMES_FIX:
fns.setdefault(_normalise_fn(e["raised_in"]), []).append(e["id"])
return fns
def untagged_reference_entries():
"""(entries with no tag, entries in total). The ceiling on what TAG-matching
can ever see — which is why there is a second key."""
entries = [e for e in load(ERRORS_PATH)["entries"]
if e.get("kind", ERROR_KIND) == ERROR_KIND]
return sum(1 for e in entries if not e.get("tag")), len(entries)
def split_shortfall(shortfall, twins=None, fns=None):
"""The shortfall, as port-versus-write. Sub-counts sum to the input, always.
Two passes, in order, and the order is the confidence ranking: a tag is
asserted at both sites, a function name is a convention nothing asserts.
Whatever survives both needs a clause written.
A site whose tag is unreadable (`error-of of stmt.tag`) still gets the
second pass — the enclosing function is readable even when the tag is not.
What survives falls to NO_TWIN, the conservative side, because the
alternative is claiming a clause exists to port when nothing checked.
"""
twins = reference_fix_tags() if twins is None else twins
fns = reference_fix_functions() if fns is None else fns
out = {HAS_TWIN: [], PROBABLE_TWIN: [], NO_TWIN: [],
"multiplicity": [], "multiplicity_by_function": [],
"tag_unreadable": []}
for site in shortfall:
tag, fn = site[3], _normalise_fn(site[4])
if tag is None:
out["tag_unreadable"].append(site)
if tag and tag in twins:
out[HAS_TWIN].append(site)
if len(twins[tag]) > 1:
out["multiplicity"].append(site)
elif fn and fn in fns:
out[PROBABLE_TWIN].append(site)
if len(fns[fn]) > 1:
out["multiplicity_by_function"].append(site)
else:
out[NO_TWIN].append(site)
return out
def render_split(split, twins, fns):
ambiguous = sorted({s[3] for s in split["multiplicity"]})
amb_fn = sorted({_normalise_fn(s[4]) for s in
split["multiplicity_by_function"]})
lines = ["", " THE SHORTFALL SPLIT — port versus write "
"(C5 / Ruling 2, extended C6). A number,", " not the clauses.", ""]
lines.append(f" {HAS_TWIN:<24} {len(split[HAS_TWIN]):>3} by TAG — the "
"reference raises this tag and")
lines.append(" "
" names a fix; porting it is mechanical")
lines.append(f" {PROBABLE_TWIN:<24} {len(split[PROBABLE_TWIN]):>3} by "
"FUNCTION — the reference answers this in a")
lines.append(" "
" correspondingly-named function. Weaker evidence:")
lines.append(" "
" the correspondence is a naming convention, not")
lines.append(" "
" an assertion anywhere")
lines.append(f" {NO_TWIN:<24} {len(split[NO_TWIN]):>3} survives both "
"passes; needs a clause written")
lines.append("")
lines.append(" The two are NOT merged. A matcher that hides which key it "
"used is worse than")
lines.append(" one that understates, because the next build decides per "
"site and has to see")
lines.append(" which evidence it is deciding on.")
lines.append("")
lines.append(f" Of the {len(split[HAS_TWIN])} matched by tag, "
f"{len(split['multiplicity'])} carry a tag mapping to more "
"than one entry")
lines.append(f" — {len(ambiguous)} such tags:")
lines.extend(_wrap_counted(ambiguous, twins))
lines.append(f" Of the {len(split[PROBABLE_TWIN])} matched by function, "
f"{len(split['multiplicity_by_function'])} sit in a reference "
"function holding")
lines.append(f" more than one raise — {len(amb_fn)} such functions:")
lines.extend(_wrap_counted(amb_fn, fns))
lines.append(" Multiplicity bites harder on the second pass: a tag names "
"one kind of failure,")
lines.append(" a function may raise several. Reported, never resolved.")
if split["tag_unreadable"]:
lines.append("")
lines.append(f" Tag unreadable at the site: "
f"{len(split['tag_unreadable'])} — these get the second "
"pass anyway, because")
lines.append(" the enclosing function is readable when the tag is not:")
for f, n, src, _t, fn in split["tag_unreadable"]:
where = ("no twin" if (f, n, src, _t, fn) in split[NO_TWIN]
else PROBABLE_TWIN)
lines.append(f" {f}:{n} [{where}] {src}")
# THE CEILING, REWRITTEN. C5 left this saying what tag-matching could not
# see. The second pass moved it, and the honest statement is that a ceiling
# has moved, not that it has gone: say what is STILL unmatchable.
untagged, entries = untagged_reference_entries()
lines.append("")
lines.append(f" THE CEILING, MOVED — NOT GONE. {untagged} of the {entries} "
"catalogued reference errors")
lines.append(" carry no tag at all (PlanesSyntaxError, RuleConflict, "
"PlanesAmbiguity, ModuleError,")
lines.append(" RuleNotSupported, and four PlanesError sites). The function "
"pass reaches into")
lines.append(f" that half and matched "
f"{len(split[PROBABLE_TWIN])} of them. What it CANNOT reach "
"is a self-hosted")
lines.append(" function the port renamed, or one with no reference "
"counterpart at all — and")
lines.append(" nothing distinguishes those two from each other here. So "
f"{len(split[NO_TWIN])} remains an")
lines.append(" upper bound on the authorship work, a tighter one than "
"before and still not a")
lines.append(" measurement.")
lines.append("")
# The population this phase exists for: the lexer and parser sites, which
# ALL landed in "no twin" under tag-matching alone. Named individually.
syntax = [s for s in split[HAS_TWIN] + split[PROBABLE_TWIN] + split[NO_TWIN]
if s[0] in ("parser.planes", "lexer.planes")]
state_of = {}
for label in (HAS_TWIN, PROBABLE_TWIN, NO_TWIN):
for s in split[label]:
state_of[(s[0], s[1])] = label
lines.append(f" THE {len(syntax)} LEXER AND PARSER SITES — every one of "
"them landed in 'no twin'")
lines.append(" under tag-matching alone, which is the whole reason for a "
"second key:")
for s in sorted(syntax, key=lambda s: (s[0], s[1])):
label = state_of[(s[0], s[1])]
mark = {HAS_TWIN: "tag ", PROBABLE_TWIN: "fn ", NO_TWIN: "-- "}[label]
lines.append(f" [{mark}] {s[0]}:{s[1]:<5} {str(s[4]):<26} "
f"{s[3] or '(dynamic)'}")
matched = sum(1 for s in syntax if state_of[(s[0], s[1])] != NO_TWIN)
lines.append(f" {matched} of {len(syntax)} now matched; "
f"{len(syntax) - matched} still need a clause written.")
return "\n".join(lines)
def _wrap_counted(names, table):
"""A name list with each entry's catalogue count, wrapped to the report."""
if not names:
return [" none"]
out, row = [], " "
for name in names:
if len(row) + len(name) + 6 > 72:
out.append(row)
row = " "
row += f" {name} ({len(table[name])})"
out.append(row)
return out
def render_self_hosted(sites):
total = sum(len(v) for v in sites.values())
lines = ["", "=" * 72, ""]
lines.append("SELF-HOSTED STACK — grammar/*.planes, measured for the first "
"time (C4 / §158).")
lines.append("Before this build there was no slot for a fix in a "
"self-hosted error, so these")
lines.append("sites were outside the commitment rather than failing it "
"(REPORT_DETAIL_CONVERGENCE.md §6).")
lines.append("")
lines.append(f" {total} raise sites across "
f"{', '.join(SELF_HOSTED_GLOB)}:")
for label in STATES:
n = len(sites[label])
pct = f"{100 * n / total:.0f}%" if total else "-"
lines.append(f" {label:<28} {n:>3} of {total} ({pct})")
lines.append("")
lines.append(f" THE SELF-HOSTED WORK LIST: {len(sites[SHORTFALL])}. "
"Reported, not merged into the")
lines.append(" reference's list above, which is at zero — and not driven "
"to zero here: that is")
lines.append(" message-writing work with its own ruling, not a side "
"effect of opening the slot.")
lines.append("")
intent = self_hosted_intent_assessment(sites, SELF_HOSTED_ASSESSED_THIS_BUILD)
lines.append(f" The same fourth question as the reference report: "
f"{intent['unassessed']} of {total} self-hosted")
lines.append(f" sites are unassessed for whether their fix holds in "
f"every situation ({intent['assessed']} assessed).")
lines.append("")
by_file: dict[str, int] = {}
for site in sites[SHORTFALL]:
by_file[site[0]] = by_file.get(site[0], 0) + 1
for f in SELF_HOSTED_GLOB:
if f in by_file:
lines.append(f" {f:<20} {by_file[f]:>3} name no fix")
twins, fns = reference_fix_tags(), reference_fix_functions()
lines.append(render_split(split_shortfall(sites[SHORTFALL], twins, fns),
twins, fns))
return "\n".join(lines)
def render_report(cov):
lines = []
lines.append("errors_coverage: grammar/errors.json against "
"`errors name the fix`")
lines.append("=" * 72)
lines.append("")
lines.append(f" {cov['total']} catalogued message sites: "
f"{cov['errors']} errors, {len(cov['reports'])} rule-plane "
f"reports.")
lines.append(" An entry is an error iff it constructs one of the seven "
"error classes the")
lines.append(" reference defines (the rule, with its exclusions, is in "
"grammar_gen.py). A")
lines.append(" report is text a plane renders without raising; it is "
"listed, not measured.")
lines.append("")
total = cov["errors"]
c = cov["counts"]
for label in STATES:
n = c[label]
pct = f"{100 * n / total:.0f}%" if total else "-"
lines.append(f" {label:<28} {n:>3} of {total} ({pct})")
lines.append("")
lines.append(f" THE WORK LIST: {c[SHORTFALL]}. Its target is zero.")
lines.append(f" Of the {c[NAMES_FIX]} that name one, {cov['delegating']} "
f"delegate to grammar/messages/amber.json,")
lines.append(" which counts only because every template there carries "
"its own fix.")
lines.append(f" Unreadable at the raise site: {len(cov['unreadable'])} "
f"— never counted as a pass.")
lines.append("")
intent = cov["intent"]
lines.append(" A fourth question, orthogonal to the three above: does "
"the fix a site names")
lines.append(f" hold in every situation that site can fire from? "
f"{intent['unassessed']} of {total} sites have not")
lines.append(f" been read against that question by any build yet "
f"({intent['assessed']} have). Not a work")
lines.append(" list — a site nobody has looked at is not a site known "
"to be wrong.")
lines.append("")
if not cov["amber_templates_all_name_a_fix"]:
lines.append(" WARNING: an amber template carries no `fix` field; "
"delegating sites counted as naming none.")
lines.append("")
if cov["work_list"]:
lines.append(f"SHOULD NAME ONE AND DOES NOT ({len(cov['work_list'])}) "
"— the work list, reported not enforced.")
lines.append("Measured structurally: a message that names its fix "
"inline in prose is counted")
lines.append("here too, because a catalogue needs the fix in a field "
"or on its own line.")
for e in cov["work_list"]:
mark = " (unreadable here)" if e["unreadable"] else ""
lines.append(f" {e['source']:<22} {e['id']}{mark}")
if e["template"]:
lines.append(f" {e['template']}")
lines.append("")
if cov["deliberate"]:
lines.append(f"DELIBERATELY NAMES NONE ({len(cov['deliberate'])}) — "
"each with the reason its raise site states.")
lines.append("Not a gap and not a pass: a distinct decision, so the "
"work list above stays")
lines.append("a work list as it approaches its floor.")
for e in cov["deliberate"]:
lines.append(f" {e['source']:<22} {e['id']}")
lines.append(f" {e['no_fix']}")
lines.append("")
if cov["reports"]:
lines.append(f"rule-plane reports ({len(cov['reports'])}) — "
"catalogued as text a person reads, and not")
lines.append("measured against a commitment about errors:")
for e in cov["reports"]:
lines.append(f" {e['source']:<22} {e['id']}")
lines.append("")
lines.append("This is a report, never a gate. A message with no fix clause "
"is work to schedule,")
lines.append("not a build to break — and an honest one-line error should "
"not be un-committable.")
return "\n".join(lines)
def main(argv):
args = argv[1:]
cov = coverage()
sites = self_hosted_sites()
if "--json" in args:
def rows(v):
return [{"file": f, "line": n, "source": s, "tag": t,
"function": fn} for f, n, s, t, fn in v]
twins, fns = reference_fix_tags(), reference_fix_functions()
split = split_shortfall(sites[SHORTFALL], twins, fns)
cov["self_hosted"] = {k: rows(v) for k, v in sites.items()}
cov["self_hosted_intent"] = self_hosted_intent_assessment(
sites, SELF_HOSTED_ASSESSED_THIS_BUILD)
cov["self_hosted_split"] = {
"has_a_reference_twin": rows(split[HAS_TWIN]),
"has_a_probable_twin": rows(split[PROBABLE_TWIN]),
"no_reference_twin": rows(split[NO_TWIN]),
"counts": {HAS_TWIN: len(split[HAS_TWIN]),
PROBABLE_TWIN: len(split[PROBABLE_TWIN]),
NO_TWIN: len(split[NO_TWIN]),
"sum": len(split[HAS_TWIN]) + len(split[PROBABLE_TWIN])
+ len(split[NO_TWIN])},
"multiplicity": {
"sites": len(split["multiplicity"]),
"tags": {t: twins[t]
for t in sorted({s[3] for s in split["multiplicity"]})},
},
"multiplicity_by_function": {
"sites": len(split["multiplicity_by_function"]),
"functions": {
f: fns[f] for f in
sorted({_normalise_fn(s[4])
for s in split["multiplicity_by_function"]})},
},
"tag_unreadable": rows(split["tag_unreadable"]),
}
print(json.dumps(cov, indent=2))
else:
print(render_report(cov))
print(render_self_hosted(sites))
return 0 # a report never fails
if __name__ == "__main__":
sys.exit(main(sys.argv))