Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# Corpus changes require review from a repo member with write access.
# See README.md for the corpus update process.
/inbox/ @GangGreenTemperTatum @mamicidal @rocklambros @sclintonowasp
/corpus/ @GangGreenTemperTatum @mamicidal @rocklambros @sclintonowasp
42 changes: 42 additions & 0 deletions .github/ISSUE_TEMPLATE/propose-resource.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
name: Propose a corpus resource
description: Suggest an OWASP GenAI Security Project document for the Advisor corpus, without opening a PR.
title: "Resource: "
labels: ["corpus-proposal"]
body:
- type: markdown
attributes:
value: |
A maintainer will check the license and add it through `inbox/`.
If you'd rather do it yourself, see `inbox/README.md`.
- type: input
id: title
attributes:
label: Title
validations:
required: true
- type: input
id: url
attributes:
label: Source URL
description: genai.owasp.org resource page, direct PDF link, or upstream repo path.
validations:
required: true
- type: input
id: initiative
attributes:
label: Initiative
description: e.g. llm-top10, agentic-top10, data-security, mcp-security, red-teaming, governance, incident-response, or a new one.
- type: input
id: supersedes
attributes:
label: Replaces an existing resource?
description: The MANIFEST.yaml id it supersedes, if any.
- type: textarea
id: license
attributes:
label: License
description: Where does the document state its license? (e.g. "CC BY-SA 4.0 on page 2")
- type: textarea
id: notes
attributes:
label: What is it, and when should the Advisor use it?
88 changes: 88 additions & 0 deletions .github/workflows/process-inbox.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
name: PR - Process inbox

# Turns submissions in inbox/ into corpus/ files + MANIFEST.yaml entries on the
# PR branch, so reviewers see exactly what will merge. See inbox/README.md.

on:
pull_request:
paths:
- "inbox/**"

permissions:
contents: write
pull-requests: write

concurrency:
group: process-inbox-${{ github.event.pull_request.number }}
cancel-in-progress: true

jobs:
process:
runs-on: ubuntu-latest
env:
# Same-repo PRs get the processed files committed back. Fork PRs can't
# be written to, so they get a dry run and the report in the job summary.
SAME_REPO: ${{ github.event.pull_request.head.repo.full_name == github.repository }}
steps:
- uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.ref }}
repository: ${{ github.event.pull_request.head.repo.full_name }}
# Commits pushed with the default GITHUB_TOKEN don't re-trigger
# workflows. Set CORPUS_BOT_TOKEN (a GitHub App or fine-grained
# token with contents:write) so validate-corpus re-runs on the
# processed commit.
token: ${{ secrets.CORPUS_BOT_TOKEN || github.token }}

- uses: actions/setup-python@v5
with:
python-version: "3.11"

- name: Install dependencies
run: pip install -r scripts/requirements.txt

- name: Process inbox
id: process
run: |
args="--report inbox-report.md"
if [ "$SAME_REPO" != "true" ]; then args="$args --dry-run"; fi
set +e
python scripts/process_inbox.py $args
echo "exit=$?" >> "$GITHUB_OUTPUT"
cat inbox-report.md >> "$GITHUB_STEP_SUMMARY"

- name: Commit processed submissions
if: env.SAME_REPO == 'true'
run: |
git config user.name "corpus-inbox-bot"
git config user.email "actions@users.noreply.github.com"
git add -A corpus/ inbox/
if git diff --cached --quiet; then
echo "Nothing to commit."
else
git commit -m "Process inbox submissions into corpus/"
git push
fi

- name: Comment report on PR
if: env.SAME_REPO == 'true'
env:
GH_TOKEN: ${{ github.token }}
PR: ${{ github.event.pull_request.number }}
run: |
{
cat inbox-report.md
echo
echo "_Reviewers: confirm the license and the manifest entry before merging._"
} > comment.md
gh api "repos/${{ github.repository }}/issues/$PR/comments" -F body=@comment.md > /dev/null

- name: Fail if anything was held
if: steps.process.outputs.exit != '0'
run: |
echo "::error::Some inbox submissions were held -- see the report in the job summary / PR comment."
exit 1

- name: Validate corpus
if: env.SAME_REPO == 'true'
run: python scripts/validate_corpus.py --repo-root .
1 change: 1 addition & 0 deletions .github/workflows/validate-corpus.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ on:
pull_request:
paths:
- "corpus/**"
- "inbox/**"
- "scripts/validate_corpus.py"

jobs:
Expand Down
3 changes: 2 additions & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,8 @@
## Adding a corpus resource

See "Adding or updating a resource" in [README.md](./README.md). In short: drop the
file(s) in `corpus/<initiative>/`, add a `corpus/MANIFEST.yaml` entry, open a PR --
file and a `.yaml` sidecar in `inbox/` and open a PR (see [inbox/README.md](inbox/README.md)),
or do it by hand: file(s) in `corpus/<initiative>/`, a `corpus/MANIFEST.yaml` entry, a PR --
CI validates the manifest, and PDF text extraction runs automatically after merge.

## Changing the tooling
Expand Down
16 changes: 15 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,17 +22,31 @@ corpus/
MANIFEST.yaml # catalog of every resource -- the single source of truth
<initiative>/... # vendored content, grouped by initiative
_extracted/ # auto-generated PDF text (search index only, never edit)
inbox/ # submission drop zone -- always empty on main
scripts/
validate_corpus.py # MANIFEST checks, run by CI on every PR
process_inbox.py # inbox/ -> corpus/ + MANIFEST entry, with license check
extract_pdf_text.py # regenerates corpus/_extracted/ from vendored PDFs
requirements.txt
.github/workflows/
validate-corpus.yaml # PR check
process-inbox.yaml # PR: processes inbox/ submissions onto the PR branch
extract-pdf-text.yaml # runs on push to main, commits refreshed _extracted/
```

## Adding or updating a resource

**Easiest: use the inbox.** Put the file and a small `.yaml` sidecar in
`inbox/` and open a PR. The `process-inbox` workflow checks the document for an
explicit license statement, moves it into `corpus/`, writes the `MANIFEST.yaml`
entry (and marks anything it supersedes), and comments on the PR with what it
did. See [inbox/README.md](inbox/README.md). Don't have the file, or don't want
to open a PR? Use the **Propose a corpus resource** issue form instead.

The manual steps below are what the inbox automates, and still apply for
anything it can't handle (e.g. turning an existing `linked` entry into a
vendored one).

1. **Confirm the license.** Only vendor (copy in) a document whose license is
explicitly stated in the source -- most OWASP GenAI Security Project output is
CC BY-SA 4.0. If you can't confirm it, catalog it as a `linked` entry instead
Expand Down Expand Up @@ -70,7 +84,7 @@ scripts/
python scripts/validate_corpus.py
```

6. **Open a PR.** `validate-corpus.yaml` checks that every entry has its required
6. **Open a PR.** `validate-corpus.yaml` checks that `inbox/` is empty and that every entry has its required
fields, ids are unique, `status` is valid, and `path` exists (or is `null` for
`linked`). CODEOWNERS review is required for `corpus/`.
7. **After merge,** `extract-pdf-text.yaml` extracts text from every `current` or
Expand Down
52 changes: 52 additions & 0 deletions inbox/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# inbox/ -- submit new corpus resources here

Drop a new resource (PDF, markdown, JSON, or a folder of them) in this folder in
a pull request. The `process-inbox` workflow checks it and moves it into
`corpus/` with a `MANIFEST.yaml` entry, committed back to your PR branch. A
CODEOWNER then reviews and merges.

**This folder is always empty on `main`.** CI fails while anything other than
this README is here, so nothing unreviewed is ever published.

## How to submit

1. Add the file, e.g. `inbox/My-Guide-v1.0.pdf`.
2. Add a sidecar with the same name and a `.yaml` extension,
e.g. `inbox/My-Guide-v1.0.yaml`:

```yaml
title: "My Guide" # defaults to the PDF's metadata title
initiative: mcp-security # required: corpus/ subfolder (new ones OK)
source_url: https://genai.owasp.org/resource/... # or source_repo (+ source_path)
version: "1.0" # defaults to the version in the file name
published: "2026-10-01" # optional
folder: companions # optional subfolder under the initiative
supersedes: some-existing-id # optional: marks that entry superseded
status: current # current (default) | draft | linked
id: my-guide-1.0 # optional: defaults to the file name
notes: >
What it is and when the Advisor should use it.
```

3. Open a PR. The workflow comments with what it did, or why a submission was
held.

To work locally instead, run `python scripts/process_inbox.py` (add
`--dry-run` to preview) and commit the result.

## License rules

The processor only moves a file into `corpus/` when the document itself states
one of these licenses: CC BY-SA 4.0, CC BY 4.0, Apache-2.0 or MIT. A vague
"released under creative commons licenses" doesn't count.

- **No license statement found:** the file is held. Either confirm the license
with the initiative lead and add a statement, or set `status: linked`.
- **`status: linked`:** catalogs the resource with `path: null` and
`license: UNCONFIRMED`, and deletes the file. The Advisor links to
`source_url` and doesn't quote from it. A sidecar on its own (no file) works
for this too.
- **Any other license (e.g. NC/ND variants):** held for a maintainer decision.

Reviewers still confirm the license before merging. The detector helps
reviewers; it doesn't replace them.
Loading
Loading