Skip to content

Add script to predict and mark required phrases in license rules - #5262

Open
Kaushik-Kumar-CEG wants to merge 1 commit into
aboutcode-org:developfrom
Kaushik-Kumar-CEG:gsoc/inject-ml-phrases
Open

Add script to predict and mark required phrases in license rules#5262
Kaushik-Kumar-CEG wants to merge 1 commit into
aboutcode-org:developfrom
Kaushik-Kumar-CEG:gsoc/inject-ml-phrases

Conversation

@Kaushik-Kumar-CEG

@Kaushik-Kumar-CEG Kaushik-Kumar-CEG commented Aug 9, 2026

Copy link
Copy Markdown

Part of #5243. Depends on #5222.

Adds a command that loads the hardened required phrase model, predicts BIOES spans for eligible license rules, validates the candidates with ScanCode's existing helpers, and marks accepted phrases with {{ }}.

The command:

  • accepts a local final model directory or Hugging Face repository
  • validates the final model and its published artifact hashes before inference
  • preserves complete-word token alignment when input is truncated
  • skips rules that already contain required phrases or cannot be updated
  • applies changes in memory and writes each changed rule once
  • supports expression filtering, dry runs, limits, and verbose output
  • records ml_model as the phrase source

Validation:

  • 167 training, export, and integration tests pass
  • Ruff and git diff --check pass
  • the command was tested with the final hardened model

An installable package version is also available in scancode-required-phrases on branch gsoc/prediction-injection. It includes the add-model-required-phrases entry point, tests, and documentation. Its complete test suite passes with 185 tests, and its documentation and distributions build successfully.

The license index must be rebuilt after writing rules

disclosure : used Claude to help review and clean up few bugs in script

@Kaushik-Kumar-CEG Kaushik-Kumar-CEG changed the title Add tool to mark required phrases using the trained tagger Add script to predict and mark required phrases in license rules Aug 9, 2026
@Kaushik-Kumar-CEG
Kaushik-Kumar-CEG marked this pull request as draft August 9, 2026 16:25
@Kaushik-Kumar-CEG
Kaushik-Kumar-CEG marked this pull request as ready for review September 7, 2026 13:40
Signed-off-by: Kaushik Kumar <kaushikrjpm10@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant