ReleaseGate

An openly licensed research artifact for measuring one production decision:

A new container image is available. Given the release evidence and this deployment's configuration, should an operator promote it or hold it?

Warning

Research software, not a production safety certification. Every final full run false-promoted 87 or more of the 128 test instances that should have been held (the best, GPT-5.5, false-promoted exactly 87), before selective thresholding. Do not grant the stack unattended production authority. Promotion is disabled unless an external signed-webhook receiver is deliberately configured.

What is here

Folder	Purpose	Maturity
ReleaseGateBench/	Dataset, schemas, miners, offline harness, baselines, scorer, annotations, frozen runs	Frozen benchmark v1.0.0
ReleaseGateCore/	Single-call reference SUT implementing decide(instance) -> AgentOutput	Research reference implementation
ReleaseGateStack/	Diun → Windmill → isolated Dokploy staging → gated signed-webhook workflow	Experimental; staging-oriented
Thesis/	Master thesis LaTeX source and compiled EPFLMain.pdf	Final, defended 2026

Quick start

Python 3.11–3.13.

git clone https://github.com/pmdlt/ReleaseGate.git
cd ReleaseGate

python -m venv .venv && source .venv/bin/activate
python -m pip install \
  -e './ReleaseGateCore[dev]' \
  -e './ReleaseGateBench[dev]' \
  -e './ReleaseGateStack[dev]'

python -m pytest ReleaseGateCore/tests ReleaseGateBench/tests ReleaseGateStack/tests

Validate the frozen artifact (offline: schemas, IDs, split isolation, label alignment, prediction coverage, manifests, input fingerprints):

python ReleaseGateBench/scripts/validate_release.py --all

shasum -a 256 ReleaseGateBench/data/v1.0.0/test.inputs.jsonl
# ce56a3c235f8cc8203977e8f92aa5122fc2b125c04ce1f87caacee574fbda6e1

Re-score a frozen run (no API key needed):

rg-score \
  --predictions ReleaseGateBench/runs/v1.0.0/b3_kimi_k2.6.rerun.test.jsonl \
  --labels ReleaseGateBench/data/v1.0.0/test.labels.jsonl \
  --output /tmp/kimi-k2.6.metrics.json

Re-running ReleaseGateCore against a provider is optional and costs money:

python -m pip install -e './ReleaseGateCore[litellm]'

rg-run --sut b3 --model openai/your-model \
  --inputs ReleaseGateBench/data/v1.0.0/dev.inputs.jsonl \
  --output /tmp/releasegate-predictions.jsonl

Do not use the test labels to tune prompts, models, or thresholds.

Documentation

• ReleaseGateBench/README.md — benchmark guide, then SPECIFICATION.md (design, schemas, gold policy, RQs), DATASHEET.md (composition, limitations, release contract), and runs/v1.0.0/README.md (result-file map).
• ReleaseGateCore/README.md — Python API, retry policy, uncertain-hold fallback, injected test client.
• ReleaseGateStack/README.md — read its safety notes first; the stack has destructive staging APIs, so use a disposable environment.
• Thesis/EPFLMain.pdf — Trustworthy Infrastructure Autonomy, the master thesis this artifact accompanies. Build from source with cd Thesis && make (XeLaTeX, latexmk, and minted required).

Public labels make v1.0.0 fully replayable. Future versions should mine releases after a declared cutoff and keep test labels private until evaluation closes.

Known limitations

• Ground truth uses an outcome oracle. 74 test holds are marked latent: their failure was not knowable from the evidence available at decision time.
• Six self-hosted applications and Docker Compose only.
• Staging is single-run, liveness-only, and available for 27 instances.
• The contextual-security subset (six within-release contrast pairs) is underpowered.
• One final seed per model; run-to-run variance is not estimated.
• Release profiles share evidence, but the bootstrap and McNemar calculations operate at the instance level, so their intervals and p-values are exploratory.
• Provider behavior, model aliases, and pricing change; frozen predictions stay reproducible even when a live endpoint does not.

Citation

GitHub can export a citation from CITATION.cff. Plain text:

Paul Madelénat. ReleaseGate: A benchmark and reference system for deployment-contextual release-promotion decisions. Version 1.0.0, 2026. https://github.com/pmdlt/ReleaseGate

To cite the thesis itself:

Paul Madelénat. Trustworthy Infrastructure Autonomy: Verify-in-the-loop agents for release promotion on cloud and operator-managed production systems. MSc thesis, School of Computer and Communication Sciences, Swiss Federal Institute of Technology Lausanne (EPFL), 2026.

License

• Apache-2.0: software and configuration under ReleaseGateBench/, ReleaseGateCore/, ReleaseGateStack/, .github/, plus root build files.
• CC BY 4.0: benchmark curation, labels, annotations, frozen results, documentation, and the thesis text and figures under Thesis/.
• LPPL-1.3c: the LaTeX template in Thesis/EPFLThesis.cls and Thesis/Configurations/, derived from the IPLeiria Thesis template by José Areia.
• Third-party release notes, issue excerpts, advisories, names, trademarks, and bundled fonts remain subject to their original rights.

See LICENSE and THIRD_PARTY_NOTICES.md.