Repository navigation
Conversation
README.md says 2026/final/ is the canonical source for the published 2026 release and 2026/working/ is retained as historical working files, but 2026/working/polling/scripts/sync_sprint2.py (GenAI-Security-Project#164) still actively reads "existing entries" from 2026/working/. For 8 of 11 LLM00-LLM10 IDs, 2026/working/ and 2026/final/ assign the ID to two different risks (see issue GenAI-Security-Project#166 for the full table and a read-only python -c reproduction), so any tooling trusting 2026/working/'s numbering silently attaches the wrong topic to the wrong ID. Add .github/scripts/check_working_final_entry_drift.py: it maps each LLM##_*.md file's ID to its filename under both directories and flags any ID whose filename differs between them. Rather than requiring the trees to match outright (the drift is a known, undecided condition, not something this change resolves), it pins the currently observed mismatched-ID set as KNOWN_DRIFT_IDS and fails only if the *actual* drift set changes, so a future PR that touches either directory's numbering has to consciously update the baseline instead of silently shifting it further. test_check_working_final_entry_drift.py covers matching trees, a single mismatched ID, IDs present on only one side, matching the known baseline, a newly drifted ID outside the baseline, and a previously tracked ID being resolved without updating the baseline, all against synthetic fixture trees. A new workflow runs both the self-tests and the validator itself on changes under 2026/working/, 2026/final/, or either script. Signed-off-by: Raghuvar Karthik Durga <177662415+drk-ram@users.noreply.github.com>
drk-ram
requested review from
rocklambros and
virtualsteve-star
as code owners
November 4, 2026 20:11
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Companion to #166.
README.mdsays2026/final/is the canonical source for the published2026 release and
2026/working/is retained as historical working files,but
2026/working/polling/scripts/sync_sprint2.py(#164) still activelyreads "existing entries" from
2026/working/. For 8 of 11LLM00-LLM10IDs,
2026/working/and2026/final/assign the ID to two different risks(see #166 for the full table and a read-only
python -creproduction), soany tooling that trusts
2026/working/'s numbering silently attaches thewrong topic to the wrong ID.
Changes
.github/scripts/check_working_final_entry_drift.py: maps eachLLM##_*.mdfile's ID to its filename under both directories and flagsany ID whose filename differs between them. It does not require the two
trees to match outright -- the drift documented in [2026 errata] 2026/working/ and 2026/final/ disagree on which risk LLM03-LLM10 identifies (slug/numbering drift) #166 is a known,
undecided condition, not something this PR resolves. Instead it pins the
currently observed mismatched-ID set as
KNOWN_DRIFT_IDSand fails onlyif the actual drift set changes (grows, shrinks, or shifts), so a
future change to either directory's numbering has to consciously update
the baseline (with a note on why) instead of silently shifting it
further.
.github/scripts/test_check_working_final_entry_drift.py: coversmatching trees, a single mismatched ID, IDs present on only one side,
matching the known baseline, a newly drifted ID outside the baseline, and
a previously tracked ID being resolved without updating the baseline, all
against synthetic fixture trees built in isolated temporary directories.
.github/workflows/check-working-final-entry-drift.yml: runs thevalidator self-tests and then the validator itself on changes under
2026/working/,2026/final/, or either script.No existing
2026/working/or2026/final/content is modified.Validation
From
.github/scripts/on Python 3.11.9:python -B -m unittest discover -v -p "test_check_working_final_entry_drift.py"-- passed (9 tests).python -B check_working_final_entry_drift.py-- passed: reports the 8 known-drifted IDs from [2026 errata] 2026/working/ and 2026/final/ disagree on which risk LLM03-LLM10 identifies (slug/numbering drift) #166 and exits 0 (observed drift matches the tracked baseline).python -B -m py_compile check_working_final_entry_drift.py test_check_working_final_entry_drift.py-- passed.git diff --cached --check-- passed.