Skip to content

ci: read the project board App key from Secrets Manager via OIDC - #764

Merged
eliseong merged 2 commits into
mainfrom
ci/project-app-key-secrets-manager
Nov 4, 2026
Merged

eliseong merged 2 commits into
mainfrom
ci/project-app-key-secrets-manager

Conversation

@LeeroyHannigan

@LeeroyHannigan LeeroyHannigan commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

What

The project status sweep mints a GitHub App installation token every five minutes. Until now the App's private key was a GitHub environment secret. With this change, the job reads the key at run time from AWS Secrets Manager. The read goes through an OIDC role that trusts only the project-board environment. #744 made the same change for the Docker Hub credential.

  • The job has id-token: write and assumes the role for a 15-minute session.
  • The secrets action exports the key as an environment variable, and the mint step reads it from there. The log masks the key whole and line by line.
  • The step after minting blanks every variable that the secrets action exported, and the AWS credentials. After that step, the environment of the sweep step holds neither the App key nor the AWS credentials. The sweep uses the installation token.
  • The sweep step can still request an OIDC token, assume the role and read the key, because the job has id-token: write. The cleanup step cannot prevent this. The runner sets ACTIONS_ID_TOKEN_REQUEST_URL and ACTIONS_ID_TOKEN_REQUEST_TOKEN in the environment of every step. The comment above the cleanup step says this too. The same code owners approve changes to the sweep script and to this workflow.
  • The setup notes at the top of the workflow say where the key is and how to rotate it. They also show how to build the secret's JSON and keep the PEM's line breaks.

Rollout: the role and the secret exist. After the first green run, delete the environment secret and the old key in the App's settings. If the secret holds the same key as the environment secret, rotate it first. Generate a new key, store it in the secret, and wait for a green run.

Why

A person who has the App's private key can act as the App until someone rotates the key. The App can write to the organization's projects. With OIDC, a run can read the key only from main in the project-board environment. CloudTrail records each read. To rotate the key, an admin updates the secret.

Testing done

  • actionlint with shellcheck is clean.
  • I read the role's two policies from IAM. The trust policy has one statement: sts:AssumeRoleWithWebIdentity for GitHub's OIDC provider, with StringEquals on sub and aud and no wildcards. The permission policy has one statement, which allows secretsmanager:GetSecretValue on the board App's secret.
  • The IAM policy simulator allows the role to read that secret. It denies PutSecretValue and DeleteSecret on that secret, and GetSecretValue on a secret whose name starts with the same text.
  • The first scheduled run after the merge is the live test of the role and the secret.

Checklist

  • I have read CONTRIBUTING.md
  • All tests pass (cargo test --workspace)
  • Code is formatted (cargo fmt --check)
  • Clippy is clean (cargo clippy -- -W clippy::pedantic)
  • I have added or updated tests for new functionality
  • I have updated documentation if behavior changed
  • Breaking changes are noted below (if any)
  • If this changes the wire protocol, Storage trait, auth model, on-disk
    format, or public CLI surface, an RFC has been accepted or is linked
    below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: n/a. The change is to one CI workflow, and it follows #744.


By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.

The project status sweep minted its GitHub App token from a private key
stored as a GitHub environment secret. Read the key at run time from AWS
Secrets Manager through an OIDC role that trusts only the project-board
environment, as the Docker Hub credential is now read.

The job gets id-token: write and a 15-minute AWS session. After minting
the token, a step blanks every variable the secrets action exported and
the AWS credentials, so the sweep script runs with only the installation
token, as before. The header's setup notes cover where the key lives, how
to build the secret's JSON, and how to rotate the key.
@LeeroyHannigan LeeroyHannigan added the v1 Needed for the 1.0 release label Nov 2, 2026
@navareamol navareamol added this to the 1.0 milestone Nov 3, 2026
@LeeroyHannigan LeeroyHannigan added the release-blocker Must ship for its milestone's release label Nov 3, 2026
@eliseong eliseong self-assigned this Nov 3, 2026

@eliseong eliseong left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran the changed steps on a local machine with the pinned action versions. A local HTTP server replaced Secrets Manager, and a second local server replaced the GitHub API. The workflow code works as intended. One statement in the PR description is not correct.

Results:

  1. The secrets action exports the key as PROJECT_APP_PRIVATE_KEY. The mint step reads that variable. The test server verified the JWT signature with the original key, and the signature was valid.
  2. The log masking is correct. The action masks the full value. The runner then masks each line of the value. No line of the key was visible in the step output.
  3. The cleanup step writes an empty value for each name in SECRETS_LIST_CLEAN_UP and for the three AWS credential variables. After the cleanup step, no variable in the job environment contained key data.
  4. The post-job cleanup of each action still passes after the cleanup step.
  5. I tested these secret formats: PKCS#1, PKCS#8, CRLF line endings, literal \n sequences, no final newline, and extra JSON keys. The mint step made a token for each format.
  6. I tested these incorrect formats: a raw PEM that is not JSON, an incorrect JSON key name, an empty value, and a truncated PEM. Each run failed before the sweep step. No error message showed key data.
  7. I tested these Secrets Manager errors: throttling, access denied, not found, and a server error. Each run failed before the sweep step. No error message showed key data.
  8. actionlint 1.7.12 with shellcheck 0.11.0 shows no errors for this file.

Change requested:

The PR description says: "The sweep script then runs with only the installation token, as it did before." This is not correct. The job has id-token: write, so the sweep step can request an OIDC token, assume the role, and read the key. The cleanup step cannot prevent this, because the runner adds ACTIONS_ID_TOKEN_REQUEST_URL and ACTIONS_ID_TOKEN_REQUEST_TOKEN to the environment of each step. The comment in the workflow file states this correctly. Please make the PR description agree with that comment.

Question:

Please confirm the trust policy of ExtendDBProjectBoardApp. The comments say that the role trusts only the project-board environment, but the repo does not show the policy. Does the trust policy use StringEquals with these two conditions and no wildcards?

  • token.actions.githubusercontent.com:sub equals repo:ExtendDB/extenddb:environment:project-board
  • token.actions.githubusercontent.com:aud equals sts.amazonaws.com

Also, does the permission policy allow only secretsmanager:GetSecretValue on extenddb/project-board-app?

@eliseong eliseong assigned eliseong and unassigned eliseong Nov 3, 2026
@eliseong

eliseong commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator

Update on the two rollout preconditions.

The secret extenddb/project-board-app is in place.

The role ExtendDBProjectBoardApp is not created yet. The rollout note says to merge only after the role exists, so this is not ready to merge. Until the role exists, every scheduled run fails at the AWS step, once every 5 minutes.

When you create the role, set its trust policy to StringEquals with these two conditions and no wildcards:

  • token.actions.githubusercontent.com:sub equals repo:ExtendDB/extenddb:environment:project-board
  • token.actions.githubusercontent.com:aud equals sts.amazonaws.com

Set the permission policy to one statement: secretsmanager:GetSecretValue on arn:aws:secretsmanager:us-east-1:975306274793:secret:extenddb/project-board-app-*.

The branch was 90 commits behind main. Main does not change the sweep
workflow or the sweep script, and the merge has no conflicts.
@LeeroyHannigan

Copy link
Copy Markdown
Collaborator Author

@eliseong On your two points:

  1. The description now says what the workflow comment says. After the cleanup step, the sweep step has neither the App key nor the AWS credentials. It can still request an OIDC token and read the key, because the job has id-token: write. The runner sets the token request variables for every step.
  2. The role exists now. I read both of its policies back from IAM:
    • Trust policy: one statement, sts:AssumeRoleWithWebIdentity for GitHub's OIDC provider in the role's account. The only condition operator is StringEquals, with token.actions.githubusercontent.com:sub equal to repo:ExtendDB/extenddb:environment:project-board and token.actions.githubusercontent.com:aud equal to sts.amazonaws.com. There are no wildcards.
    • Permission policy: one inline policy with one statement, secretsmanager:GetSecretValue on that secret. The resource ends in -??????, so it matches only the 6-character suffix that Secrets Manager adds to the name. The -* form in your comment also matches a longer name that starts with the same text, for example the same name with -extra added. No managed policy is attached, and the role has no permissions boundary. The secret uses the default aws/secretsmanager key, so the role needs no KMS permission.
    • The IAM policy simulator allows GetSecretValue on the secret and denies PutSecretValue and DeleteSecret on it. It also denies GetSecretValue on a secret with that longer name.

I merged main into the branch. Could you take another look?

@eliseong eliseong left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both points are resolved.

The description now matches the workflow comment about the OIDC token.

The role exists and is scoped as requested: one trust statement with no wildcards, and read access to the one secret only.

The workflow file is unchanged from the version I reviewed, so the earlier results still hold. Approving.

@eliseong
eliseong added this pull request to the merge queue Nov 4, 2026
Merged via the queue into main with commit f898103 Nov 4, 2026
32 checks passed
@eliseong
eliseong deleted the ci/project-app-key-secrets-manager branch November 4, 2026 22:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release-blocker Must ship for its milestone's release security v1 Needed for the 1.0 release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants