Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
370 changes: 370 additions & 0 deletions .github/ci-paths.toml

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion .github/instructions/hsm.instructions.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,6 @@ matrix.
- [ ] HSM model enum updated in `crate/server/src/config/` or `crate/hsm/base_hsm/`
- [ ] Wizard step in `crate/server/src/config/wizard/hsm_wizard.rs`
- [ ] Test vectors in `test_data/vectors/hsm/<model>/`
- [ ] CI added to `.github/workflows/test_all.yml` (software HSMs: list the leaf in `.mise/tasks/test/hsm/_default` and add the model to `test-hsm`'s `bench`; real hardware: add a `test-hsm-hardware` cell with `lock` and its secrets in `test-group.yml`, plus `skip` it in `test-hsm`), and the upstream `env:` gate (`print_error` when `CI=true` and a secret is unset, then `skip_test`) in `.mise/tasks/test/hsm/<model>` for hardware HSMs
- [ ] CI added to `.github/workflows/test_all.yml` and `.github/ci-paths.toml` (software HSMs: list the leaf in `.mise/tasks/test/hsm/_default` and add the model to the `bench` of `[units.hsm]`; real hardware: add a `test-hsm-hardware` cell with `lock` and its secrets in `test-group.yml`, plus `skip` it in the `skip:` of `test-suites`), and the upstream `env:` gate (`print_error` when `CI=true` and a secret is unset, then `skip_test`) in `.mise/tasks/test/hsm/<model>` for hardware HSMs

> Rule 4.13 of `/kms-sync-rules`.
18 changes: 7 additions & 11 deletions .github/workflows/README_main.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# CI workflows: `main.yml`

`main.yml` is the PR / tag / nightly test pipeline. Its jobs are direct callers, so the run page groups them by area (`test / …`, `supply-chain / …`, `no-nix / …`):
`main.yml` -> `supply-chain.yml`, `no-nix.yml`, `forward_proxy.yml`, `test_windows.yml`, `test_all.yml` -> `test-db.yml`, `test-interop.yml`, `test-group.yml` / `cargo-publish.yml`.
`main.yml` -> `supply-chain.yml`, `no-nix.yml`, `forward_proxy.yml`, `test_windows.yml`, `test_all.yml` -> `test-group.yml` / `cargo-publish.yml`.
Packaging and releases are described in [README_nightly.md](README_nightly.md).

## Triggers and concurrency
Expand Down Expand Up @@ -59,8 +59,8 @@ flowchart TB
```mermaid
flowchart TB
Cache["nix-dev-shell-cache<br/>OpenSSL 3.1.2 / 3.6.5 + SoftHSM2 built once per Nix input hash"]
subgraph NixJobs["One job per test group, each run by test-group.yml (Nix dev shell)"]
Groups["test-cloud (upstream only) · test-db · test-hsm · test-interop · test-obs<br/>test-pkcs11 · test-pki · test-spire · test-ui"]
subgraph NixJobs["One job per test group or leg, each run by test-group.yml (Nix dev shell)"]
Groups["test-cloud (upstream only) · test-suites (db · hsm · interop · obs · pkcs11 · pki · spire legs) · test-ui"]
Hsm["test-hsm-hardware<br/>proteccio · crypt2pay<br/>not on PR"]
end
Cache --> Groups & Hsm
Expand All @@ -76,17 +76,13 @@ Each test job runs one group of `.mise/tasks/test` (`mise run test:<group> --var
|Job|Group|
|---|---|
|`test-cloud`|`cloud` (upstream only: needs the Google, AWS and Azure secrets)|
|`test-db`|`test-db.yml`: one `test-group.yml` job per `db` leaf and variant (`sqlite`, `psql`, `mysql`, `mariadb`, `percona`, `redis` non-FIPS only), with the database environment variables|
|`test-hsm`|`hsm` without the real HSMs (`skip: hsm:crypt2pay hsm:proteccio`), then the `bench` models|
|`test-interop`|`test-interop.yml`: one `test-group.yml` job per `interop` leaf (`ase`, `db2`, `edb-tde`, `iris`, `jose`, `kmipkit`, `pykmip` non-FIPS only; `kmip-go` both variants), so a hang or leftover state in one leaf stays in its own job|
|`test-obs`|`obs`|
|`test-pkcs11`|`pkcs11`|
|`test-pki`|`pki`|
|`test-spire`|`spire`|
|`test-suites`|one `test-group.yml` job per leg of the `suites-matrix` that `ci_changes.py` builds from the `variants` of `.github/ci-paths.toml`: the `db` leaves (with the database environment variables), `hsm` without the real HSMs (`skip: hsm:crypt2pay hsm:proteccio`) then the `bench` models, the `interop` leaves (a hang or leftover state in one leaf stays in its own job), `obs`, `pkcs11`, `pki` and `spire`; a suite that does not run has no leg, so no row|
|`test-ui`|`ui`|
|`test-hsm-hardware`|real HSMs `hsm:proteccio`, `hsm:crypt2pay`, one leaf per cell (never on pull requests)|

`test-ui` runs without cloud or test secrets, because it executes `pnpm install`; `test-spire` receives only the Docker Hub login. Each caller passes `test-group.yml` only the secrets its leaves use. `test-cloud` skips on forks and fork pull requests, `test-hsm-hardware` never runs on pull requests and only in the upstream repository. In both jobs a task whose required `env:` variables are unset (a deleted or renamed secret, a typo in the mapping) fails instead of skipping, because its upstream gate calls `print_error` when `CI=true`, so a green cell proves its tasks ran. A cell's `lock` (`hsm-proteccio`, `hsm-crypt2pay`) is its concurrency group: the HSMs refuse concurrent runs, and `test-k8s-hsm` in `packaging-docker.yml` (also never on pull requests) shares the groups. A cell's `bench` is the list of HSM models whose benchmarks run after its tests, off pull requests. Pull requests cover the HSM code paths with the software simulators of `test-hsm` (`softhsm2`, `utimaco`, `crypt2pay-simulator`, `kryoptic`).
`test-suites` is the only job whose secrets depend on the leg (`secrets:` expressions on `matrix.suite`): the Docker Hub login for every leg, the EDB and IRIS tokens for `interop:*` legs, the Splunk password for `obs`. `cloud` and `ui` stay separate jobs so that their secret scope is set by the job, not by an expression.

`test-ui` runs without cloud or test secrets, because it executes `pnpm install`. `test-cloud` skips on forks and fork pull requests, `test-hsm-hardware` never runs on pull requests and only in the upstream repository. In both jobs a task whose required `env:` variables are unset (a deleted or renamed secret, a typo in the mapping) fails instead of skipping, because its upstream gate calls `print_error` when `CI=true`, so a green cell proves its tasks ran. A cell's `lock` (`hsm-proteccio`, `hsm-crypt2pay`) is its concurrency group: the HSMs refuse concurrent runs, and `test-k8s-hsm` in `packaging-docker.yml` (also never on pull requests) shares the groups. A cell's `bench` is the list of HSM models whose benchmarks run after its tests, off pull requests. Pull requests cover the HSM code paths with the software simulators of the `hsm` legs of `test-suites` (`softhsm2`, `utimaco`, `crypt2pay-simulator`, `kryoptic`).

A leaf that fails only when grouped (port, Docker or environment leak) or only in one variant: add its `<group>:<leaf>` id to the `skip` input of its job, with a one-line reason comment.

Expand Down
26 changes: 26 additions & 0 deletions .github/workflows/linkcheck.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
---
name: Link check

# Link checking hits external sites (flaky on PRs): nightly only. A workflow of its own, so pull requests
# do not list it as a skipped job.
on:
schedule:
- cron: 0 3 * * * # Daily at 03:00 UTC

jobs:
linkcheck:
name: Check documentation links
runs-on: ubuntu-24.04
permissions:
contents: read
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
submodules: recursive
persist-credentials: false

- name: Link check
uses: lycheeverse/lychee-action@e7477775783ea5526144ba13e8db5eec57747ce8 # v2
with:
args: --config lychee.toml 'documentation/docs/**/*.md'
fail: false
76 changes: 56 additions & 20 deletions .github/workflows/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,46 +25,98 @@ jobs:
uses: ./.github/workflows/cla.yml
secrets: inherit

# Runs only the jobs whose files (.github/ci-paths.toml) the PR changes; tags, schedule and dispatch run everything.
changes:
name: Detect changed areas
runs-on: ubuntu-24.04
permissions:
contents: read
outputs:
skip: ${{ steps.classify.outputs.skip }}
suites-matrix: ${{ steps.classify.outputs.suites-matrix }}
run-test: ${{ steps.classify.outputs.run-test }}
run-windows: ${{ steps.classify.outputs.run-windows }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
persist-credentials: false
# On pull_request the checkout is the test merge commit, whose first parent is the base branch tip.
fetch-depth: 2

- uses: ./.github/actions/install-mise

- name: Classify changed files
id: classify
env:
EVENT_NAME: ${{ github.event_name }}
GITHUB_TOKEN: ${{ github.token }}
run: |
mise run dev:ci-changes --self-test
if [ "${EVENT_NAME}" = "pull_request" ]; then
mise run dev:ci-changes --base HEAD^1
else
mise run dev:ci-changes --all
fi

cargo-clippy:
uses: Cosmian/reusable_workflows/.github/workflows/clippy.yml@9079c7a06025e1373a9be950fd94a36464ded0b9 # develop
needs: changes
if: ${{ !contains(fromJSON(needs.changes.outputs.skip), 'cargo-clippy') }}

cargo-deny:
uses: Cosmian/reusable_workflows/.github/workflows/cargo-audit.yml@9079c7a06025e1373a9be950fd94a36464ded0b9 # develop
name: Security Audit

cargo-machete:
uses: Cosmian/reusable_workflows/.github/workflows/cargo-machete.yml@9079c7a06025e1373a9be950fd94a36464ded0b9 # develop
needs: changes
if: ${{ !contains(fromJSON(needs.changes.outputs.skip), 'cargo-machete') }}

supply-chain:
uses: ./.github/workflows/supply-chain.yml

forward-proxy:
uses: ./.github/workflows/forward_proxy.yml
needs: changes
if: ${{ !contains(fromJSON(needs.changes.outputs.skip), 'forward-proxy') }}

no-nix:
uses: ./.github/workflows/no-nix.yml
needs: changes
if: ${{ !contains(fromJSON(needs.changes.outputs.skip), 'no-nix') }}

windows:
# Same gating as `test` (no dependabot PRs) plus no fork PRs. Windows is slow: PRs run the
# quick suite; tags and nightly run the full one.
needs: changes
if: >-
((github.event_name == 'pull_request' && !startsWith(github.ref_name, 'dependabot/') && github.actor != 'dependabot[bot]')
(((github.event_name == 'pull_request' && !startsWith(github.ref_name, 'dependabot/') && github.actor != 'dependabot[bot]')
|| startsWith(github.ref, 'refs/tags/') || github.event_name == 'schedule' || github.event_name == 'workflow_dispatch')
&& (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository)
&& (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository))
&& needs.changes.outputs.run-windows == 'true'
uses: ./.github/workflows/test_windows.yml
with:
archive-name: windows
# test_kms_server and ckms lib tests are covered on Linux; the full suite runs off-PR only.
full-suite: ${{ github.event_name != 'pull_request' }}
skip-units: ${{ needs.changes.outputs.skip }}

test: # Run all tests (PRs, tags and the nightly schedule / manual dispatch)
if: (github.event_name == 'pull_request' && !startsWith(github.ref_name, 'dependabot/') && github.actor != 'dependabot[bot]') || startsWith(github.ref,
'refs/tags/') || github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
needs: changes
if: >-
((github.event_name == 'pull_request' && !startsWith(github.ref_name, 'dependabot/') && github.actor != 'dependabot[bot]')
|| startsWith(github.ref, 'refs/tags/') || github.event_name == 'schedule' || github.event_name == 'workflow_dispatch')
&& needs.changes.outputs.run-test == 'true'
uses: ./.github/workflows/test_all.yml
secrets: inherit
with:
skip-units: ${{ needs.changes.outputs.skip }}
suites-matrix: ${{ needs.changes.outputs.suites-matrix }}

build-documentation:
name: Build documentation
needs: changes
if: ${{ !contains(fromJSON(needs.changes.outputs.skip), 'build-documentation') }}
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
Expand All @@ -84,22 +136,6 @@ jobs:
export PATH="$HOME/.cargo/bin:$PATH"
mise run docs:mdbook-build

linkcheck:
# Link checking hits external sites (flaky on PRs): nightly only.
if: github.event_name == 'schedule'
name: Check documentation links
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
submodules: recursive

- name: Link check
uses: lycheeverse/lychee-action@e7477775783ea5526144ba13e8db5eec57747ce8 # v2
with:
args: --config lychee.toml 'documentation/docs/**/*.md'
fail: false

public_documentation:
if: github.event.pull_request.head.repo.full_name == github.repository && !startsWith(github.ref_name, 'dependabot/') && github.actor != 'dependabot[bot]'
|| startsWith(github.ref, 'refs/tags/')
Expand Down
62 changes: 0 additions & 62 deletions .github/workflows/test-db.yml

This file was deleted.

37 changes: 27 additions & 10 deletions .github/workflows/test-group.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,11 +22,6 @@ on:
required: false
type: string
default: ''
extra-env:
description: KEY=VALUE lines appended to GITHUB_ENV before the tests
required: false
type: string
default: ''
secrets:
DOCKERHUB_USERNAME: {required: false}
DOCKERHUB_TOKEN: {required: false}
Expand Down Expand Up @@ -74,11 +69,33 @@ jobs:
cleanup-runner: 'true'
dev-shell-cache: 'true'

- name: Export suite environment
if: inputs.extra-env != ''
env:
EXTRA_ENV: ${{ inputs.extra-env }}
run: printf '%s\n' "$EXTRA_ENV" >> "$GITHUB_ENV"
- name: Export database suite environment
if: startsWith(inputs.suite, 'db:')
run: |
cat >> "$GITHUB_ENV" <<'EOF'
POSTGRES_USER=kms
PGUSER=kms
POSTGRES_PASSWORD=kms
POSTGRES_DB=kms
POSTGRES_HOST=localhost
POSTGRES_PORT=5432
KMS_POSTGRES_URL=postgresql://kms:kms@127.0.0.1:5432/kms
MYSQL_DATABASE=kms
MYSQL_ROOT_PASSWORD=kms
MYSQL_HOST=localhost
MYSQL_PORT=3306
KMS_MYSQL_URL=mysql://root:kms@localhost:3306/kms
PERCONA_HOST=localhost
PERCONA_PORT=3307
KMS_PERCONA_URL=mysql://root:kms@localhost:3307/kms
MARIADB_HOST=localhost
MARIADB_PORT=3308
KMS_MARIADB_URL=mysql://root:kms@localhost:3308/kms
KMS_SQLITE_PATH=data/shared
REDIS_HOST=localhost
REDIS_PORT=6379
KMS_REDIS_URL=redis://localhost:6379
EOF

- name: Docker login (Docker Hub)
if: ${{ env.DOCKERHUB_USERNAME != '' }}
Expand Down
46 changes: 0 additions & 46 deletions .github/workflows/test-interop.yml

This file was deleted.

Loading
Loading