LayerRAG-Bench is an open benchmark and Python toolkit for diagnosing cross-layer reliability failures in agentic retrieval-augmented generation systems.
A RAG answer can be grounded in retrieved text and still be operationally wrong: the evidence may be stale, inaccessible, incomplete, attached to corrupted metadata, or drawn from the wrong session. LayerRAG-Bench introduces controlled faults at those layers and measures whether a mitigation repairs the failure it is designed to address.
The completed cross-provider matrix contains 38,880 task-level evaluation records across nine models from OpenAI, Anthropic, and Gemini.
| Scenario | Strict success | Repair success |
|---|---|---|
| Schema drift | 0.000 | 0.913 |
| Stale index | 0.000 | 0.000 |
| Missing tool output | 0.000 | 0.000 |
| Permission denied | 0.000 | 0.000 |
| Wrong session state | 0.000 | 0.000 |
| Partial tool timeout | 0.314 | 0.308 |
Schema normalization repairs schema drift, but it does not restore missing or stale evidence, bypass access controls, reconstruct incomplete output, or correct wrong-session context. The benchmark is designed to measure that specificity rather than rank commercial models by aggregate score.
The enterprise_v2 benchmark contains:
| Component | Count |
|---|---|
| Enterprise domains | 8 |
| Policy specifications | 80 |
| Documents | 160 |
| Question-answer tasks | 240 |
| Fault scenarios | 9 |
| Contract modes | 2 |
The scenarios cover clean execution, stale indexes, schema drift, missing tool output, conflicting evidence, permission denial, partial tool timeouts, corrupted metadata, and wrong session state.
The repository also includes:
- deterministic and live evaluation harnesses;
- framework-neutral runtime reliability controls;
- an optional LangChain integration;
- privacy-safe result archives for aggregate recomputation;
- the LaTeX paper, generated tables, and citation metadata.
LayerRAG-Bench requires Python 3.11 and uses
uv.
uv sync --devInstall the optional LangChain integration with:
uv sync --dev --extra langchainGenerate the enterprise benchmark:
uv run layerrag-generate-enterprise-v2Run the deterministic benchmark and contract comparison:
uv run layerrag-run-benchmark
uv run layerrag-run-contract-modesRun the offline LangChain examples:
uv run --extra langchain python examples/langchain_reliability_quickstart.py
uv run --extra langchain python examples/langchain_agent_end_to_end.pyRuntime controls use observable evidence, tool behavior, and trusted application context. Benchmark scenario labels are never passed to the runtime policy.
The committed privacy-safe archives contain all scored records required to recompute the quantitative results without provider calls.
uv run layerrag-recompute-public-results \
--artifact-dir artifacts/public_results \
--json-output /tmp/layerrag-results.json \
--markdown-output /tmp/layerrag-results.mdThe recomputation covers scenario-by-contract and provider aggregates, strict-to-repair changes, groundedness false positives, hierarchical bootstrap intervals, and the focused LangChain study.
See
docs/public_artifact_recomputation.md
for the archive schema, privacy exclusions, and provenance fields.
The venue-neutral manuscript is assembled from paper/main.tex.
scripts/build_paper.sh
python scripts/check_paper_build_log.py --log-path paper/main.logThe paper reports the benchmark design, cross-provider study, retriever sensitivity audit, focused LangChain integration, uncertainty analysis, representative traces, reproducibility details, and limitations.
See paper/README.md for the source graph and build details.
The commands documented in this README are the supported public interface for version 1.0.0. Additional experiment-control and audit entry points are retained for exact research provenance, but they are not required for ordinary benchmark use.
Useful analysis commands include:
uv run layerrag-export-paper-tables
uv run layerrag-export-research-diagnostic-tables
uv run layerrag-export-langchain-runtime-evaluation
uv run layerrag-export-submission-release-preflightsrc/layerrag/ benchmark, evaluation, reliability, and integration code
artifacts/public_results privacy-safe archives and reviewed aggregate results
examples/ runnable framework-neutral and LangChain examples
paper/ manuscript source, tables, and bibliography
docs/ public guides, audit notes, and provenance material
tests/ unit and integration tests
The curated documentation index is in docs/README.md.
- The enterprise corpus is synthetic and policy-like.
- The 240 task rows are three variants of 80 policy specifications, not 240 independent content units.
- Historical live prompts exposed a version-local answer field in document metadata; the current code removes it.
- The live matrix is fixed-budget and does not establish statistically powered rankings of commercial models.
- The permission scenario simulates a failure mode; it does not certify production access-control enforcement.
- Privacy-safe archives support quantitative recomputation but exclude raw model prose and unrestricted qualitative analysis.
The complete limitations are in paper/sections/limitations.tex.
Citation metadata is available in CITATION.cff.
@software{shams2026layerrag,
author = {Musa Shams},
title = {LayerRAG-Bench: A Cross-Layer Reliability Benchmark for
Agentic Retrieval-Augmented Generation},
year = {2026},
version = {1.0.0},
url = {https://github.com/MusaShams/layerrag-bench}
}LayerRAG-Bench is released under the MIT License.