SBOM: say what the document does not cover, via --coverage - #186
Merged
villelaitila merged 4 commits intoAug 24, 2026
Merged
Conversation
…s own slot The coverage ledger answers half of what a BOM is for -- what the document does not cover -- and until now it could only be said in Softagram-specific metadata properties, which a consumer has no reason to read. CycloneDX defines compositions[].aggregate for exactly this claim, so the claim now has a home in a vocabulary consumers already know. The four counts stay where they are. A composition holds aggregate, assemblies, dependencies and vulnerabilities and nothing else, with additionalProperties false, so a ten-category taxonomy with counts and samples genuinely has no home there -- the module's existing comment was right about that and is left standing. What it was silent about is completeness, which is the part that does fit. 'complete' is unreachable by construction. The enum defines it as "no further relationships ... are KNOWN to exist"; the ledger proves only that every element the walk SAW was classified, never that the analyzer saw everything, and between those two lies every dependency an unrun analyzer would have found. Claiming completeness makes a consumer stop looking, which is the wrong direction to be wrong in for a security artifact, so the strongest honest claim is the enum's own 'unknown'. A test walks a grid of ledger states and asserts none reaches it. An empty walk is 'not_specified' rather than 'unknown': a model with no External subtree cannot distinguish an estate that depends on nothing from an analyzer that never ran, and 'unknown' would assert the best effort 'not_specified' declines to assert. Nothing calls this yet, so no emitted document differs by a byte -- asserted by the existing call-graph test over all three public entry points, which still passes unchanged. Values enumerated from bom-1.7.schema.json, the specVersion this generator declares, rather than from recollection. Suite 485 -> 492, seven added: one guards the hand-built ledger against drifting from the real report's shape, so the grid test cannot pass while proving nothing.
…ject coverage_compositions builds the claim; this puts it on a document. The subject is the metadata component, because the claim is about the assembly of THAT document rather than of the model -- a per-element SBOM and the single-document one describe different assemblies and must not both claim to describe the model. A document whose metadata component carries no bom-ref names no subject, and the composition is then emitted with no assemblies rather than with an empty array. An empty array reads as "these zero components are incomplete"; an absent key reads as "the subject was not named", which is what is true. Default-off stays in the call graph rather than moving to a parameter default, matching attach_coverage_summary. A completeness claim nobody asked for is still a claim, so the guard is asserted over all three public entry points -- the same test shape that already guards the summary, because the failure it prevents is the same: a call site added by reflex changing every emitted document. Suite 492 -> 497. Pre-registered as 495 and reported here as measured: the default-off guard is parametrized over three entry points, so three test functions collect as five tests. Counting functions instead of collected tests is the same error this project pinned a rule against; the number stands as collected rather than being reconciled after the fact.
…document The claim and its attachment existed; nothing could ask for them. A capability reachable only by importing two modules and calling three functions in the right order is half-shipped, and the half that is missing is the half a user sees. Single-document mode only, enforced rather than documented. The coverage report is model-wide, so attaching it to a per-element SBOM would claim that THIS document's third-party assembly is incomplete because some other subtree's is -- a statement the document carries no evidence for. --coverage with --level or --element-path is a CLI error, matching how every other unsupported combination here is handled. Per-document coverage is a real thing to build; asserting it before it exists is not. The import of external_identification is function-local and that is the design, not a workaround: that module imports this one, the direction is deliberate -- the generator emits, the report reads what was emitted -- and a module-level import would make it a cycle. A CLI is the composition above both and the one place entitled to know about each. No library signature changed, so no programmatic caller sees anything new. Default-off, asserted: a document produced without the flag is byte-for-byte what 1.13.0 produced. Every changed hunk in the generator is inside __main__, which is checkable from the diff rather than from this claim. Both reachable outcomes are exercised on fixtures measured to produce them -- 'unknown' where every external is identified, 'incomplete' where one is not. The first fixture chosen was in the quiet state, where the flag could have been wired to a constant and still passed; the second is what makes the test able to fail. The two rejection tests assert the exact parser message, because with the flag absent they passed on argparse's "unrecognized arguments: --coverage" -- a pass for the wrong reason that would have survived the flag doing nothing. Emitted compositions were validated against definitions.compositions in bom-1.7.schema.json out of band; the committed test keeps an offline proxy (the key set the schema defines, since it sets additionalProperties false). Suite 497 -> 502, four collected added and one more than the 501 registered: parametrizing the happy path over two fixtures added the fifth.
The flag is documented where the other SBOM options are, with the three reachable aggregate values and the reason the fourth is unreachable: 'complete' asserts that no further components are known to exist, and an analyzer that was not run leaves no evidence that it was missing. Also corrects a line this project's own 1.13.0 invalidated. The page still said an element publishing several packages "keeps the element name and an empty purl"; A6 removed that key rather than emptying it, and a test now asserts no fixture carries an empty purl in any mode. The old behaviour is named with the version that changed it, because a reader hitting an older document needs to know which of the two they are looking at.
Softagram Impact Report for pull/186 (head commit: 86b4286)TL;DR Arch. Impact: -2 | Changed code files: 4 | Directly impacted code files: 2⭐ Change Overview
⭐ Details of Dependency Changes (diagram)
🤖 AGENTS - machine-readable impact data (4 files changed, 2 impacted, +25/-0 deps)Change overviewHead Added dependencies (25)
Removed dependencies (0)None. Impacted files (2)Unchanged files that directly depend on files changed in this PR - check them for behavioral impact. Grouped by changed file:
Complete data
[] 📄 Full report
Impact Report explained. Give feedback on this report to support@softagram.com |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The coverage ledger added in 1.13.0 reached no BOM —
attach_coverage_summaryhad no caller. This makes it reachable and adds the completeness claim in CycloneDX's own slot.--coverage(single-SBOM mode only) adds:metadata.propertiescounting what was and was not identified underExternalcompositionsentry:{"aggregate": "incomplete", "assemblies": ["/Org"]}completeis never emitted. It asserts that no further components are known to exist; the ledger proves only that every element the walk saw was classified.unknownis the spec's word for a best-effort of inconclusive completeness, and is the strongest honest claim here.Single-SBOM only, enforced. The report is model-wide, so on a
--leveldocument it would claim that document's assembly is incomplete because another subtree's is. Rejected at the CLI.Opt-in. Verified identical default output vs
mainacross 33 (model, mode) pairs / 11 models, normalising only the UUID serial, timestamp and BOM-links. Every generator hunk is inside__main__.Suite 485 → 502. Emitted compositions validated against
bom-1.7.schema.json.Also fixes a doc line 1.13.0 made stale (
purlis now omitted, not empty).