diff --git a/.gitignore b/.gitignore index 7a60b85..53c4d63 100644 --- a/.gitignore +++ b/.gitignore @@ -1,2 +1,3 @@ __pycache__/ *.pyc +service.yaml diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index fc4501c..0e4bd20 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -5,18 +5,23 @@ ``` aveproject/ave (source of truth) | - | raw.githubusercontent.com fetch, on startup + every 15 min + | raw.githubusercontent.com: latest records + manifest, + | on startup + every 15 min + | github.com release assets: a pinned release, on first use + | api.github.com: the list of releases, at most hourly v -in-memory cache (a Python dict, nothing more) +in-memory caches (Python dicts, nothing more): latest, a few pinned +releases, the releases list | - | four read-only routes + | read-only routes, under /v1 (the unversioned paths are aliases) v -GET / service info, including cache_age_seconds -GET /records full list -GET /records/{ave_id} one record -GET /records/{ave_id}/mitigation one record's neutral mitigation object only -GET /search?q= substring match across title, description, attack_class, - behavioral_fingerprint +GET /v1/ service info, api_version, catalog, cache_age_seconds +GET /v1/records full list +GET /v1/records/{ave_id} one record +GET /v1/records/{ave_id}/mitigation one record's neutral mitigation object only +GET /v1/search?q= substring match across title, description, attack_class, + behavioral_fingerprint +GET /v1/releases the catalog releases, and which can be pinned ``` ## Design decisions worth knowing before changing anything @@ -45,6 +50,27 @@ control, is defined in `aveproject/ave`'s own schema, not in any single implementation's docs; that boundary belongs to the standard, and this service simply respects it. +**Every data response names its catalog.** A catalog is the latest data +(`AVE-Catalog: latest`) or one release (`AVE-Catalog: v1.2.0`), and its +`sha256`, `schema_version`, `generated_at` and `source_commit` come from the +manifest beside the records file, never from guesses. The latest records and +manifest are fetched separately, so a refresh that gets two that do not match +(a commit landing between the two fetches) is refused like any other failed +refresh, and the last known-good cache keeps serving. The `ETag` is derived +from the records `sha256`, the path without its `/v1` prefix and the query, +so an alias and its `/v1` path share one. + +**A pinned release is verified before it is served, and only then cached.** +`?release=` downloads `SHA256SUMS`, the manifest and the records file, and +runs the same checks as `aveproject/ave`'s `scripts/check_release_assets.py`. +A release that fails is a `502` and is not cached, so a later request tries +again. Releases never change, so the cache has no expiry, only a size bound +(`RELEASE_CACHE_SIZE`, oldest use evicted first). + +**The releases list spends as little of GitHub's unauthenticated rate limit +as it can**: cached for an hour, then revalidated with `If-None-Match`, so an +unchanged list costs a `304`. A failed refresh keeps serving the last list. + **Rate limiting exists because the hosting context has real, finite headroom**, not as defensive-by-default paranoia. See `CHANGELOG.md` for when and why this was added. diff --git a/CHANGELOG.md b/CHANGELOG.md index e7ad9c7..c3d6ce8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,8 +5,32 @@ follows [Keep a Changelog](https://keepachangelog.com/). ## [Unreleased] +### Added + +- Version 1 of the contract under `/v1/`: `/v1/`, `/v1/records`, + `/v1/records/{ave_id}`, `/v1/records/{ave_id}/mitigation`, `/v1/search`. + The unversioned paths stay as aliases with identical behaviour (#49). +- Every data response names the catalog it came from: `AVE-API-Version`, + `AVE-Catalog` (`latest` or a release tag), `AVE-Records-SHA256`, + `AVE-Schema-Version`, `AVE-Source-Commit` when known, plus `ETag`, + `Last-Modified` and `Cache-Control`. A matching `If-None-Match` returns + `304`. The root adds `api_version`, `schema_version` and a `catalog` + object read from the latest manifest. +- `?release=vX.Y.Z` on every data route serves that catalog release of + `aveproject/ave`, verified against its `SHA256SUMS` and manifest before + use and cached in memory (a few releases). An unknown release is a `404`; + one that fails verification is a `502` and is not cached. +- `GET /v1/releases`: the catalog releases, from the GitHub Releases API, + cached for an hour and revalidated with conditional requests. + ### Changed +- A refresh of the latest data now also fetches its manifest, and a records + file that does not match the manifest's `sha256` or `record_count` counts + as a failed refresh (the last known-good cache keeps serving). +- CORS exposes the `AVE-*` headers and `ETag`, so browser scripts can read + them. + - Hosting switched from the planned Render deployment to Google Cloud Run for the first production deployment (`--min-instances 0`, `--max-instances 3`, `us-central1`), accepting the card-on-file diff --git a/CLAUDE.md b/CLAUDE.md index ac6d16f..68dd219 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -21,8 +21,9 @@ mid-session. ## Resilience rules -- Every network call this service makes (currently: one, the fetch to - `aveproject/ave`) must handle failure without taking down unrelated +- Every network call this service makes (currently: the latest records + and manifest from `aveproject/ave`, a pinned release's assets, and the + GitHub Releases API) must handle failure without taking down unrelated requests. A dependency being briefly unreachable is a routine event, not an incident; the cache-serves-stale-on-refresh-failure pattern in `_ensure_fresh()` is the model to follow for any future external call @@ -34,8 +35,8 @@ mid-session. decorator every existing route has. A route added without it is a gap in the rate-limiting coverage, not an oversight to catch later. - Timeouts on every outbound call, no exceptions. The existing fetch uses - `timeout=10`; any new outbound call needs an explicit timeout of its - own, never the client library's default. + `timeout=10` through `_client()`; any new outbound call needs an explicit + timeout of its own, never the client library's default. ## Security rules diff --git a/CONTEXT.md b/CONTEXT.md index 2126aad..a471e47 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -7,8 +7,9 @@ any prior history with this project. ## What this is A read-only reference API for AVE (Agentic Vulnerability Enumeration). -Four endpoints: list records, get one record, get one record's neutral -mitigation object, search. No auth, no write path, ever. +A handful of endpoints under `/v1`: list records, get one record, get one +record's neutral mitigation object, search, and list the catalog releases +that can be pinned. No auth, no write path, ever. ## What this is not @@ -28,8 +29,8 @@ addition here. Keep this boring on purpose. ## Data flow, one direction `aveproject/ave` publishes `dist/ave-records-latest.json`. This service -fetches it, caches it in memory, and serves it back out through four thin -endpoints. Nothing here originates data, transforms it meaningfully, or +fetches it (or a pinned release of it), caches it in memory, and serves it +back out through a few thin endpoints. Nothing here originates data, transforms it meaningfully, or stores a durable copy anywhere. If a task description involves writing new record content, generating new fields, or anything that isn't "fetch, cache, serve," stop and check whether the task actually belongs in diff --git a/README.md b/README.md index 561403a..e8ed99e 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,7 @@ Read-only reference implementation of the [AVE](https://aveproject.org) ## Quick start ```bash -curl -s https://api.aveproject.org/records/AVE-2026-00046 | jq . +curl -s https://api.aveproject.org/v1/records/AVE-2026-00046 | jq . ``` ## What this is, and isn't @@ -29,17 +29,88 @@ depends on this specific deployment staying available. ## Endpoints +Version 1 of the contract lives under `/v1`: + | Method | Path | Description | |---|---|---| -| GET | `/` | Service info, record count, cache age | -| GET | `/records` | All records | -| GET | `/records/{ave_id}` | One record | -| GET | `/records/{ave_id}/mitigation` | The record's neutral `mitigation` object only | -| GET | `/search?q=...` | Substring search across title, description, attack class, behavioral fingerprint | +| GET | `/v1/` | Service info, API version, the catalog being served, cache age | +| GET | `/v1/records` | All records, deprecated ones included with their `status` | +| GET | `/v1/records/{ave_id}` | One record | +| GET | `/v1/records/{ave_id}/mitigation` | The record's neutral `mitigation` object only | +| GET | `/v1/search?q=...` | Substring search across title, description, attack class, behavioral fingerprint | +| GET | `/v1/releases` | The catalog releases of `aveproject/ave`, and which can be pinned | | GET | `/docs` | Interactive OpenAPI docs (auto-generated) | -No authentication. No write endpoints. Rate limited to 60 requests/minute -per IP as a basic abuse backstop, generous for any legitimate use. +The unversioned paths (`/`, `/records`, `/records/{ave_id}`, +`/records/{ave_id}/mitigation`, `/search`) are aliases of their `/v1` paths +with identical bodies and headers, so existing callers keep working. New +callers should use `/v1`. + +A deprecated record is served with its `status`; only an id that was never +published is a `404`. + +No authentication. No write endpoints. Rate limited to 60 requests a minute +per IP for each path, as a basic abuse backstop, generous for any legitimate +use. + +## Which data you got + +Every data response (records, one record, a mitigation, a search) carries +headers saying which catalog it came from: + +| Header | Value | +|---|---| +| `AVE-API-Version` | `1`, the contract version (on every response) | +| `AVE-Catalog` | `latest`, or the release tag, for example `v1.2.0` | +| `AVE-Records-SHA256` | SHA-256 of the records file the answer came from | +| `AVE-Schema-Version` | The schema version every record in it declares | +| `AVE-Source-Commit` | The `aveproject/ave` commit it was built from, when known (releases) | +| `ETag`, `Last-Modified` | For conditional requests; `Last-Modified` is the file's `generated_at` | + +`GET /v1/` reports the same for the latest data in a `catalog` object, read +from the manifest that sits next to the records file. + +## Pinning a release + +Add `?release=vX.Y.Z` to any data route to read a fixed catalog release +instead of the latest data: + +```bash +curl -s "https://api.aveproject.org/v1/records/AVE-2026-00046?release=v1.2.0" +``` + +The service downloads that release's `ave-records.json`, checks it against +the release's `SHA256SUMS` and manifest before serving anything from it, and +keeps the last few releases in memory. An unknown release is a `404`; a +release whose files do not agree is a `502` and is not cached. Releases from +`v1.2.0` on can be pinned; `GET /v1/releases` lists them. These checks prove +the files belong together, not who published them; to check provenance, use +`gh attestation verify` on a downloaded release. + +## Caching + +Send the `ETag` back as `If-None-Match` and an unchanged answer comes back +as an empty `304`: + +```bash +curl -s -D - -o /dev/null https://api.aveproject.org/v1/records/AVE-2026-00046 +curl -s -o /dev/null -w '%{http_code}\n' \ + -H 'If-None-Match: ""' \ + https://api.aveproject.org/v1/records/AVE-2026-00046 +``` + +The latest data is `Cache-Control: public, max-age=300`; a pinned release +never changes, so it is `public, max-age=86400, immutable`. + +## Not for scanning hot paths + +This service runs as a single instance that scales to zero when idle, so the +first request after a quiet spell waits for a cold start, and every caller +shares one instance and its rate limit. Use it for lookups, links and +spot checks. A scanner that resolves `ave_id` values on every finding should +download a catalog release once (see +[Releases and pinning](https://github.com/aveproject/ave#releases-and-pinning) +in `aveproject/ave`) and read it locally. ## Running locally @@ -50,10 +121,12 @@ uvicorn main:app --reload ## Data source -Fetches `dist/ave-records-latest.json` from +Fetches `dist/ave-records-latest.json` and its manifest from [aveproject/ave](https://github.com/aveproject/ave) on startup and every 15 -minutes thereafter. This repo holds no copy of the record data itself; -`aveproject/ave` is the single source of truth. +minutes thereafter. A pinned release is downloaded from that repo's GitHub +Releases on first use, and the list of releases comes from the GitHub +Releases API, cached for an hour. This repo holds no copy of the record data +itself; `aveproject/ave` is the single source of truth. ## Deployment diff --git a/constants.py b/constants.py index a8b1d2a..47dcdd0 100644 --- a/constants.py +++ b/constants.py @@ -1,6 +1,7 @@ SERVICE_NAME = "AVE Reference API" STANDARD_URL = "https://aveproject.org" REPO_URL = "https://github.com/aveproject/ave" +API_VERSION = "1" MSG_RECORD_NOT_FOUND = "No record found for ave_id {ave_id}" MSG_MITIGATION_NOT_FOUND = "No mitigation object found for ave_id {ave_id}" @@ -14,3 +15,22 @@ MSG_INDEX_REFRESH_FAILED = ( "Index could not refresh the cache, reporting the last known state: {error}" ) +MSG_LATEST_SHA256_MISMATCH = ( + "The latest records file does not match the sha256 in its manifest; " + "the two were probably fetched from different commits" +) +MSG_LATEST_COUNT_MISMATCH = ( + "The latest records file does not hold the record_count in its manifest" +) +MSG_RELEASE_NOT_FOUND = ( + "No catalog release {tag} with ave-records.json, ave-records.manifest.json " + "and SHA256SUMS; see /v1/releases for the releases that can be pinned" +) +MSG_RELEASE_UNVERIFIED = "Release {tag} failed verification and was not used: {reason}" +MSG_RELEASE_FETCH_FAILED = "Release {tag} could not be downloaded: {error}" +MSG_RELEASES_UNAVAILABLE = ( + "The list of releases could not be fetched from GitHub and none is cached yet" +) +MSG_RELEASES_REFRESH_FAILED = ( + "Releases refresh failed, continuing to serve the last known list: {error}" +) diff --git a/main.py b/main.py index 4e493ac..c2e44c2 100644 --- a/main.py +++ b/main.py @@ -1,19 +1,35 @@ +import hashlib +import json import logging +import re import time +from collections import OrderedDict +from datetime import datetime, timezone +from email.utils import format_datetime +from typing import Annotated import httpx from fastapi import FastAPI, HTTPException, Query, Request from fastapi.middleware.cors import CORSMiddleware +from fastapi.responses import JSONResponse, Response from slowapi import Limiter, _rate_limit_exceeded_handler from slowapi.errors import RateLimitExceeded from slowapi.util import get_remote_address from constants import ( + API_VERSION, MSG_INDEX_REFRESH_FAILED, MSG_INITIAL_FETCH_FAILED, + MSG_LATEST_COUNT_MISMATCH, + MSG_LATEST_SHA256_MISMATCH, MSG_MITIGATION_NOT_FOUND, MSG_RECORD_NOT_FOUND, MSG_REFRESH_FAILED, + MSG_RELEASE_FETCH_FAILED, + MSG_RELEASE_NOT_FOUND, + MSG_RELEASE_UNVERIFIED, + MSG_RELEASES_REFRESH_FAILED, + MSG_RELEASES_UNAVAILABLE, REPO_URL, SERVICE_NAME, STANDARD_URL, @@ -26,15 +42,70 @@ "https://raw.githubusercontent.com/aveproject/ave/main/" "dist/ave-records-latest.json" ) +MANIFEST_URL = ( + "https://raw.githubusercontent.com/aveproject/ave/main/" + "dist/ave-records-latest.manifest.json" +) +RELEASE_ASSET_URL = "https://github.com/aveproject/ave/releases/download/{tag}/{name}" +RELEASES_API_URL = "https://api.github.com/repos/aveproject/ave/releases?per_page=100" REFRESH_INTERVAL_SECONDS = 15 * 60 +RELEASES_REFRESH_SECONDS = 60 * 60 +RELEASE_CACHE_SIZE = 3 +OUTBOUND_TIMEOUT_SECONDS = 10 + +RECORDS_ASSET = "ave-records.json" +MANIFEST_ASSET = "ave-records.manifest.json" +SUMS_ASSET = "SHA256SUMS" +RELEASE_TAG_PATTERN = r"^v(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)$" +LATEST_CACHE_CONTROL = "public, max-age=300" +RELEASE_CACHE_CONTROL = "public, max-age=86400, immutable" +RELEASES_CACHE_CONTROL = "public, max-age=300" +# the root's catalog object, in this order; schema_version sits beside it +CATALOG_FIELDS = ( + "channel", + "record_count", + "sha256", + "generated_at", + "source_commit", + "release", +) + +_cache: dict = {"records": [], "fetched_at": 0.0, "catalog": None} +# pinned releases never change, so they are kept until evicted, oldest use first +_release_cache: OrderedDict = OrderedDict() +_releases: dict = {"items": None, "etag": None, "fetched_at": 0.0} +_release_manifests: dict = {} -_cache: dict = {"records": [], "fetched_at": 0.0} +Release = Annotated[ + str | None, + Query( + pattern=RELEASE_TAG_PATTERN, + description=( + "Serve this catalog release, for example v1.2.0, instead of the " + "latest data. See /v1/releases." + ), + ), +] limiter = Limiter(key_func=get_remote_address) app = FastAPI( title=SERVICE_NAME, - description="A read-only reference implementation of the AVE standard's record lookup API.", + description=( + "A read-only reference implementation of the AVE standard's record " + "lookup API.\n\n" + "Version 1 of the contract lives under `/v1/`. The unversioned paths " + "(`/`, `/records`, `/records/{ave_id}`, `/records/{ave_id}/mitigation`, " + "`/search`) are aliases of the `/v1` paths with identical behaviour.\n\n" + "Every data response states the catalog it came from in its headers " + "(`AVE-Catalog`, `AVE-Records-SHA256`, `AVE-Schema-Version`, " + "`AVE-Source-Commit` when known), carries an `ETag`, and answers a " + "matching `If-None-Match` with `304`. Add `?release=vX.Y.Z` to any data " + "route to pin a catalog release.\n\n" + "This service runs as a single instance that scales to zero. Use it for " + "lookups, not in a scanner's hot path: download a catalog release from " + "aveproject/ave for that." + ), ) app.state.limiter = limiter app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler) @@ -44,6 +115,15 @@ allow_origins=["*"], # public, read-only data, no reason to restrict allow_methods=["GET"], allow_headers=["*"], + # browsers hide non-safelisted response headers from scripts unless listed + expose_headers=[ + "AVE-API-Version", + "AVE-Schema-Version", + "AVE-Catalog", + "AVE-Records-SHA256", + "AVE-Source-Commit", + "ETag", + ], ) @@ -56,12 +136,58 @@ async def add_security_headers(request: Request, call_next): return response +@app.middleware("http") +async def add_api_version_header(request: Request, call_next): + response = await call_next(request) + response.headers["AVE-API-Version"] = API_VERSION + return response + + +def _client(**kwargs) -> httpx.AsyncClient: + # every outbound call has an explicit timeout, never the library default + return httpx.AsyncClient(timeout=OUTBOUND_TIMEOUT_SECONDS, **kwargs) + + +def _sha256(data: bytes) -> str: + return hashlib.sha256(data).hexdigest() + + +def _latest_catalog(body: bytes, manifest: dict) -> tuple[list, dict]: + """The records in the latest file, and the catalog its manifest describes. + + The two files are fetched separately, so a refresh that straddles a commit + can get a records file and a manifest that do not belong together; that is + refused (the last known-good cache keeps serving) rather than labelled + with the wrong sha256. + """ + sha256 = _sha256(body) + if manifest.get("sha256", sha256) != sha256: + raise ValueError(MSG_LATEST_SHA256_MISMATCH) + records = json.loads(body) + if manifest.get("record_count", len(records)) != len(records): + raise ValueError(MSG_LATEST_COUNT_MISMATCH) + return records, { + "channel": "latest", + "schema_version": manifest.get("schema_version"), + "record_count": len(records), + "sha256": sha256, + "generated_at": manifest.get("generated_at"), + "source_commit": manifest.get("source_commit"), + "release": None, + } + + async def _refresh_cache() -> None: - async with httpx.AsyncClient(timeout=10) as client: - response = await client.get(SOURCE_URL) - response.raise_for_status() - records = response.json() + async with _client() as client: + records_response = await client.get(SOURCE_URL) + records_response.raise_for_status() + manifest_response = await client.get(MANIFEST_URL) + manifest_response.raise_for_status() + records, catalog = _latest_catalog( + records_response.content, manifest_response.json() + ) _cache["records"] = records + _cache["catalog"] = catalog _cache["fetched_at"] = time.time() @@ -77,6 +203,169 @@ async def _ensure_fresh() -> None: logger.warning(MSG_REFRESH_FAILED.format(error=e)) +class ReleaseNotFound(Exception): + pass + + +class ReleaseUnverified(Exception): + pass + + +def _verify_release(tag: str, files: dict[str, bytes]) -> tuple[list, dict]: + """The records of a downloaded release, once its three files agree. + + The same checks as aveproject/ave's scripts/check_release_assets.py: every + file matches SHA256SUMS, and the manifest matches the records file and the + tag. They prove the files belong together, not who published them; + `gh attestation verify` is the check for that. + """ + sums = {} + for line in files[SUMS_ASSET].decode("utf-8", "replace").splitlines(): + parts = line.split() + if len(parts) == 2: + sums[parts[1].lstrip("*")] = parts[0].lower() + for name in (RECORDS_ASSET, MANIFEST_ASSET): + if sums.get(name) != _sha256(files[name]): + raise ReleaseUnverified(f"{name} does not match {SUMS_ASSET}") + try: + manifest = json.loads(files[MANIFEST_ASSET]) + records = json.loads(files[RECORDS_ASSET]) + except ValueError as e: + raise ReleaseUnverified(f"unreadable JSON: {e}") from None + if not isinstance(manifest, dict) or not isinstance(records, list): + raise ReleaseUnverified("the manifest or the records file has the wrong shape") + if manifest.get("sha256") != _sha256(files[RECORDS_ASSET]): + raise ReleaseUnverified(f"the manifest's sha256 is not that of {RECORDS_ASSET}") + if manifest.get("release") != tag: + raise ReleaseUnverified( + f"the manifest names release {manifest.get('release')!r}" + ) + if manifest.get("record_count") != len(records): + raise ReleaseUnverified("the manifest's record_count is not the record count") + schema_version = manifest.get("schema_version") + if any( + not isinstance(record, dict) or record.get("schema_version") != schema_version + for record in records + ): + raise ReleaseUnverified("a record's schema_version is not the manifest's") + source_commit = manifest.get("source_commit") + if not isinstance(source_commit, str) or not re.fullmatch( + r"[0-9a-f]{40}", source_commit + ): + raise ReleaseUnverified("the manifest's source_commit is not a commit SHA") + return records, { + "channel": tag, + "schema_version": schema_version, + "record_count": len(records), + "sha256": manifest["sha256"], + "generated_at": manifest.get("generated_at"), + "source_commit": source_commit, + "release": tag, + } + + +async def _download_release(tag: str) -> dict[str, bytes]: + files = {} + async with _client(follow_redirects=True) as client: + for name in (SUMS_ASSET, MANIFEST_ASSET, RECORDS_ASSET): + response = await client.get(RELEASE_ASSET_URL.format(tag=tag, name=name)) + if response.status_code == 404: + raise ReleaseNotFound(tag) + response.raise_for_status() + files[name] = response.content + return files + + +async def _release_catalog(tag: str) -> tuple[list, dict]: + if tag in _release_cache: + _release_cache.move_to_end(tag) + return _release_cache[tag] + try: + records, catalog = _verify_release(tag, await _download_release(tag)) + except ReleaseNotFound: + raise HTTPException( + status_code=404, detail=MSG_RELEASE_NOT_FOUND.format(tag=tag) + ) from None + except ReleaseUnverified as e: + detail = MSG_RELEASE_UNVERIFIED.format(tag=tag, reason=e) + logger.warning(detail) + raise HTTPException(status_code=502, detail=detail) from None + except httpx.HTTPError as e: + raise HTTPException( + status_code=502, detail=MSG_RELEASE_FETCH_FAILED.format(tag=tag, error=e) + ) from None + _release_cache[tag] = (records, catalog) + while len(_release_cache) > RELEASE_CACHE_SIZE: + _release_cache.popitem(last=False) + return records, catalog + + +async def _catalog(release: str | None) -> tuple[list, dict]: + if release is None: + await _ensure_fresh() + return _cache["records"], _cache["catalog"] + return await _release_catalog(release) + + +def _catalog_headers(catalog: dict) -> dict[str, str]: + headers = { + "AVE-Catalog": catalog["channel"], + "AVE-Records-SHA256": catalog["sha256"], + } + if catalog.get("schema_version"): + headers["AVE-Schema-Version"] = catalog["schema_version"] + if catalog.get("source_commit"): + headers["AVE-Source-Commit"] = catalog["source_commit"] + return headers + + +def _etag(request: Request, catalog: dict) -> str: + # /records and /v1/records are one resource, so the path is taken without + # its /v1 prefix: an ETag from one is valid on the other + path = request.url.path.removeprefix("/v1") + query = sorted(request.query_params.multi_items()) + key = f"{API_VERSION}|{catalog['sha256']}|{path}|{query}" + return '"' + _sha256(key.encode())[:32] + '"' + + +def _etag_matches(if_none_match: str | None, etag: str) -> bool: + if not if_none_match: + return False + candidates = [candidate.strip() for candidate in if_none_match.split(",")] + return "*" in candidates or any( + candidate.removeprefix("W/") == etag for candidate in candidates + ) + + +def _http_date(timestamp: str | None) -> str | None: + if not timestamp: + return None + try: + moment = datetime.fromisoformat(timestamp.replace("Z", "+00:00")) + except ValueError: + return None + if moment.tzinfo is None: + return None + return format_datetime(moment.astimezone(timezone.utc), usegmt=True) + + +def _data_response(request: Request, payload, catalog: dict) -> Response: + etag = _etag(request, catalog) + headers = { + **_catalog_headers(catalog), + "ETag": etag, + "Cache-Control": ( + RELEASE_CACHE_CONTROL if catalog["release"] else LATEST_CACHE_CONTROL + ), + } + last_modified = _http_date(catalog.get("generated_at")) + if last_modified: + headers["Last-Modified"] = last_modified + if _etag_matches(request.headers.get("if-none-match"), etag): + return Response(status_code=304, headers=headers) + return JSONResponse(payload, headers=headers) + + @app.on_event("startup") async def startup() -> None: try: @@ -85,9 +374,12 @@ async def startup() -> None: logger.error(MSG_INITIAL_FETCH_FAILED.format(error=e)) -@app.get("/") +@app.get("/v1/", summary="Service status and the catalog being served") +@app.get("/", include_in_schema=False) @limiter.limit("60/minute") async def index(request: Request) -> dict: + """Service info, the API contract version, and the catalog the latest data + comes from, read from the manifest beside the records file.""" try: await _ensure_fresh() except Exception as e: @@ -103,62 +395,175 @@ async def index(request: Request) -> dict: status = "stale" else: status = "ok" + catalog = _cache["catalog"] return { "name": SERVICE_NAME, "standard": STANDARD_URL, "repo": REPO_URL, + "api_version": API_VERSION, "status": status, "record_count": len(_cache["records"]), "cache_age_seconds": age, + "schema_version": catalog["schema_version"] if catalog else None, + "catalog": {field: catalog[field] for field in CATALOG_FIELDS} + if catalog + else None, "docs": "/docs", } -@app.get("/records") +@app.get("/v1/records", summary="All records") +@app.get("/records", include_in_schema=False) @limiter.limit("60/minute") -async def list_records(request: Request) -> list[dict]: - await _ensure_fresh() - return _cache["records"] +async def list_records(request: Request, release: Release = None) -> Response: + """Every published record, deprecated ones included, with their status.""" + records, catalog = await _catalog(release) + return _data_response(request, records, catalog) -@app.get("/records/{ave_id}") +@app.get("/v1/records/{ave_id}", summary="One record") +@app.get("/records/{ave_id}", include_in_schema=False) @limiter.limit("60/minute") -async def get_record(request: Request, ave_id: str) -> dict: - await _ensure_fresh() - for record in _cache["records"]: +async def get_record( + request: Request, ave_id: str, release: Release = None +) -> Response: + """One record by ave_id. A deprecated record is served with its status; only + an id the catalog has never published is a 404.""" + records, catalog = await _catalog(release) + for record in records: if record.get("ave_id") == ave_id: - return record + return _data_response(request, record, catalog) raise HTTPException( - status_code=404, detail=MSG_RECORD_NOT_FOUND.format(ave_id=ave_id) + status_code=404, + detail=MSG_RECORD_NOT_FOUND.format(ave_id=ave_id), + headers=_catalog_headers(catalog), ) -@app.get("/records/{ave_id}/mitigation") +@app.get("/v1/records/{ave_id}/mitigation", summary="One record's mitigation") +@app.get("/records/{ave_id}/mitigation", include_in_schema=False) @limiter.limit("60/minute") -async def get_mitigation(request: Request, ave_id: str) -> dict: - await _ensure_fresh() - for record in _cache["records"]: +async def get_mitigation( + request: Request, ave_id: str, release: Release = None +) -> Response: + """The record's neutral mitigation object, and nothing else.""" + records, catalog = await _catalog(release) + for record in records: if record.get("ave_id") == ave_id: mitigation = record.get("mitigation") if mitigation is None: raise HTTPException( status_code=404, detail=MSG_MITIGATION_NOT_FOUND.format(ave_id=ave_id), + headers=_catalog_headers(catalog), ) - return mitigation + return _data_response(request, mitigation, catalog) raise HTTPException( - status_code=404, detail=MSG_RECORD_NOT_FOUND.format(ave_id=ave_id) + status_code=404, + detail=MSG_RECORD_NOT_FOUND.format(ave_id=ave_id), + headers=_catalog_headers(catalog), ) -@app.get("/search") +@app.get("/v1/search", summary="Search records") +@app.get("/search", include_in_schema=False) @limiter.limit("60/minute") -async def search(request: Request, q: str = Query(..., min_length=1)) -> list[dict]: - await _ensure_fresh() +async def search( + request: Request, q: str = Query(..., min_length=1), release: Release = None +) -> Response: + """Case-insensitive substring match across title, description, attack class + and behavioral fingerprint.""" + records, catalog = await _catalog(release) needle = q.lower() fields = ("title", "description", "attack_class", "behavioral_fingerprint") - return [ - record - for record in _cache["records"] - if any(needle in str(record.get(field, "")).lower() for field in fields) - ] + return _data_response( + request, + [ + record + for record in records + if any(needle in str(record.get(field, "")).lower() for field in fields) + ], + catalog, + ) + + +async def _release_entry(client: httpx.AsyncClient, release: dict) -> dict: + tag = release.get("tag_name") + assets = { + asset["name"]: asset["browser_download_url"] + for asset in release.get("assets", []) + } + pinnable = all( + name in assets for name in (RECORDS_ASSET, MANIFEST_ASSET, SUMS_ASSET) + ) + manifest = None + if pinnable: + manifest = _release_manifests.get(tag) + if manifest is None: + response = await client.get(assets[MANIFEST_ASSET]) + response.raise_for_status() + manifest = response.json() + _release_manifests[tag] = manifest + return { + "tag": tag, + "published_at": release.get("published_at"), + "pinnable": pinnable, + "schema_version": manifest.get("schema_version") if manifest else None, + "record_count": manifest.get("record_count") if manifest else None, + "sha256": manifest.get("sha256") if manifest else None, + "assets": assets, + } + + +async def _refresh_releases() -> None: + headers = { + "Accept": "application/vnd.github+json", + "X-GitHub-Api-Version": "2022-11-28", + } + if _releases["etag"]: + # a 304 answer does not spend the unauthenticated rate limit's budget + # on a list that has not changed + headers["If-None-Match"] = _releases["etag"] + async with _client(follow_redirects=True) as client: + response = await client.get(RELEASES_API_URL, headers=headers) + if response.status_code == 304 and _releases["items"] is not None: + _releases["fetched_at"] = time.time() + return + response.raise_for_status() + items = [ + await _release_entry(client, release) + for release in response.json() + if not release.get("draft") + ] + _releases["items"] = items + _releases["etag"] = response.headers.get("etag") + _releases["fetched_at"] = time.time() + + +async def _ensure_releases() -> None: + if ( + _releases["items"] is not None + and time.time() - _releases["fetched_at"] < RELEASES_REFRESH_SECONDS + ): + return + try: + await _refresh_releases() + except Exception as e: + if _releases["items"] is None: + raise HTTPException( + status_code=502, detail=MSG_RELEASES_UNAVAILABLE + ) from None + logger.warning(MSG_RELEASES_REFRESH_FAILED.format(error=e)) + + +@app.get("/v1/releases", summary="Catalog releases that can be pinned") +@limiter.limit("60/minute") +async def list_releases(request: Request) -> Response: + """aveproject/ave's catalog releases, newest first, from the GitHub Releases + API (cached for an hour). `pinnable` releases carry ave-records.json, its + manifest and SHA256SUMS, and work with `?release=`; their schema_version, + record_count and sha256 are read from the release's own manifest.""" + await _ensure_releases() + return JSONResponse( + _releases["items"], headers={"Cache-Control": RELEASES_CACHE_CONTROL} + ) diff --git a/tests/conftest.py b/tests/conftest.py index e5d752d..4826386 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -1,3 +1,5 @@ +import hashlib +import json import time import pytest @@ -24,8 +26,27 @@ "behavioral_fingerprint": "unconfirmed destructive action", "mitigation": None, }, + { + "ave_id": "AVE-0003", + "status": "deprecated", + "title": "Superseded class", + "description": "Merged into another record after review.", + "attack_class": "superseded", + "behavioral_fingerprint": "retired", + "mitigation": None, + }, ] +FAKE_CATALOG = { + "channel": "latest", + "schema_version": "1.2.0", + "record_count": len(FAKE_RECORDS), + "sha256": hashlib.sha256(json.dumps(FAKE_RECORDS).encode()).hexdigest(), + "generated_at": "2026-10-28T23:50:03.886Z", + "source_commit": None, + "release": None, +} + @pytest.fixture(autouse=True) def isolated_service_state(monkeypatch): @@ -34,11 +55,16 @@ def isolated_service_state(monkeypatch): async def fake_refresh_cache() -> None: main._cache["records"] = FAKE_RECORDS + main._cache["catalog"] = FAKE_CATALOG main._cache["fetched_at"] = time.time() monkeypatch.setattr(main, "_refresh_cache", fake_refresh_cache) main._cache["records"] = [] + main._cache["catalog"] = None main._cache["fetched_at"] = 0.0 + main._release_cache.clear() + main._releases.update(items=None, etag=None, fetched_at=0.0) + main._release_manifests.clear() main.limiter.reset() yield diff --git a/tests/test_main.py b/tests/test_main.py index e9e44f7..afef222 100644 --- a/tests/test_main.py +++ b/tests/test_main.py @@ -3,7 +3,7 @@ from fastapi.testclient import TestClient import main -from conftest import FAKE_RECORDS +from conftest import FAKE_CATALOG, FAKE_RECORDS def test_index_reports_service_metadata(client): @@ -144,6 +144,7 @@ async def refresh() -> None: if calls["n"] <= fail_first: raise RuntimeError("simulated network failure") main._cache["records"] = FAKE_RECORDS + main._cache["catalog"] = FAKE_CATALOG main._cache["fetched_at"] = time.time() monkeypatch.setattr(main, "_refresh_cache", refresh) diff --git a/tests/test_v1.py b/tests/test_v1.py new file mode 100644 index 0000000..8673144 --- /dev/null +++ b/tests/test_v1.py @@ -0,0 +1,621 @@ +import hashlib +import json +import time + +import httpx +import pytest +from fastapi.testclient import TestClient + +import main +from conftest import FAKE_CATALOG, FAKE_RECORDS + +# captured before the autouse fixture swaps it for a fake +REAL_REFRESH_CACHE = main._refresh_cache + +TAG = "v1.2.0" +COMMIT = "a" * 40 +RELEASE_RECORDS = [ + { + "ave_id": "AVE-0001", + "schema_version": "1.2.0", + "title": "Prompt injection via tool output, as pinned", + "description": "The record as it stood in the release.", + "attack_class": "instruction-injection", + "behavioral_fingerprint": "unexpected-tool-call-sequence", + "mitigation": {"summary": "Pinned mitigation."}, + }, +] + +DATA_ROUTES = [ + "/records", + "/records/AVE-0001", + "/records/AVE-0001/mitigation", + "/search?q=prompt", +] + + +def _sha(data: bytes) -> str: + return hashlib.sha256(data).hexdigest() + + +def make_release(tag=TAG, records=None, **manifest_overrides) -> dict[str, bytes]: + """The three files of a consistent release, built the way the catalog-release + workflow builds them, with SHA256SUMS written last.""" + body = json.dumps(records or RELEASE_RECORDS, indent=2).encode() + b"\n" + manifest = { + "release": tag, + "schema_version": "1.2.0", + "record_count": len(records or RELEASE_RECORDS), + "generated_at": "2026-10-29T04:51:04.844Z", + "source": "https://github.com/aveproject/ave", + "sha256": _sha(body), + "source_commit": COMMIT, + } + manifest.update(manifest_overrides) + manifest_bytes = json.dumps(manifest, indent=2).encode() + b"\n" + sums = ( + f"{_sha(body)} ave-records.json\n" + f"{_sha(manifest_bytes)} ave-records.manifest.json\n" + ).encode() + return { + "ave-records.json": body, + "ave-records.manifest.json": manifest_bytes, + "SHA256SUMS": sums, + } + + +class FakeGitHub: + """Serves release assets and the Releases API from memory, counting requests.""" + + def __init__(self, releases=None, api_releases=None, api_etag='"releases-1"'): + self.releases = releases if releases is not None else {TAG: make_release()} + self.api_releases = api_releases + self.api_etag = api_etag + self.api_status = 200 + self.calls: list[httpx.Request] = [] + + def handler(self, request: httpx.Request) -> httpx.Response: + self.calls.append(request) + path = request.url.path + if request.url.host == "api.github.com": + if self.api_status != 200: + return httpx.Response(self.api_status) + if request.headers.get("if-none-match") == self.api_etag: + return httpx.Response(304, headers={"etag": self.api_etag}) + return httpx.Response( + 200, json=self.api_releases, headers={"etag": self.api_etag} + ) + prefix = "/aveproject/ave/releases/download/" + if path.startswith(prefix): + tag, name = path[len(prefix) :].split("/", 1) + files = self.releases.get(tag) + if files and name in files: + return httpx.Response(200, content=files[name]) + return httpx.Response(404) + + def downloads(self) -> int: + return sum(1 for call in self.calls if "/releases/download/" in call.url.path) + + def api_calls(self) -> list[httpx.Request]: + return [call for call in self.calls if call.url.host == "api.github.com"] + + +@pytest.fixture +def github(monkeypatch): + fake = FakeGitHub() + + def client_factory(**kwargs): + return httpx.AsyncClient(transport=httpx.MockTransport(fake.handler), **kwargs) + + monkeypatch.setattr(main, "_client", client_factory) + return fake + + +# --- the contract and its headers -------------------------------------------------- + + +@pytest.mark.parametrize("prefix", ["", "/v1"]) +@pytest.mark.parametrize("route", DATA_ROUTES) +def test_every_data_route_states_its_catalog(client, prefix, route): + response = client.get(prefix + route) + + assert response.status_code == 200 + headers = response.headers + assert headers["AVE-API-Version"] == "1" + assert headers["AVE-Schema-Version"] == "1.2.0" + assert headers["AVE-Catalog"] == "latest" + assert headers["AVE-Records-SHA256"] == FAKE_CATALOG["sha256"] + assert "AVE-Source-Commit" not in headers # the latest manifest names none + assert headers["ETag"].startswith('"') and headers["ETag"].endswith('"') + assert headers["Last-Modified"] == "Wed, 28 Oct 2026 23:50:03 GMT" + assert headers["Cache-Control"] == "public, max-age=300" + + +@pytest.mark.parametrize("route", DATA_ROUTES) +def test_the_unversioned_path_is_an_exact_alias(client, route): + old = client.get(route) + new = client.get("/v1" + route) + + assert old.status_code == new.status_code == 200 + assert old.content == new.content + assert dict(old.headers) == dict(new.headers) + + +def test_the_root_is_an_alias_too(client): + old = client.get("/").json() + new = client.get("/v1/").json() + + old.pop("cache_age_seconds") + new.pop("cache_age_seconds") + assert old == new + + +def test_the_api_version_header_is_on_every_response(client): + for path in ["/", "/v1/", "/docs", "/records/does-not-exist"]: + assert client.get(path).headers["AVE-API-Version"] == "1" + + +def test_the_root_reports_the_catalog_it_serves(client): + body = client.get("/v1/").json() + + assert body["api_version"] == "1" + assert body["schema_version"] == "1.2.0" + assert body["catalog"] == { + "channel": "latest", + "record_count": len(FAKE_RECORDS), + "sha256": FAKE_CATALOG["sha256"], + "generated_at": "2026-10-28T23:50:03.886Z", + "source_commit": None, + "release": None, + } + assert body["status"] == "ok" + + +def test_the_root_reports_no_catalog_while_there_is_no_data(monkeypatch): + async def failing_refresh() -> None: + raise RuntimeError("simulated network failure") + + monkeypatch.setattr(main, "_refresh_cache", failing_refresh) + + with TestClient(main.app) as test_client: + body = test_client.get("/v1/").json() + + assert body["catalog"] is None + assert body["schema_version"] is None + assert body["api_version"] == "1" + + +def test_a_404_still_says_which_catalog_was_searched(client): + response = client.get("/v1/records/does-not-exist") + + assert response.status_code == 404 + assert response.headers["AVE-Catalog"] == "latest" + assert response.headers["AVE-Records-SHA256"] == FAKE_CATALOG["sha256"] + + +def test_browsers_may_read_the_catalog_headers(client): + response = client.get("/v1/records", headers={"Origin": "https://example.com"}) + + exposed = response.headers["access-control-expose-headers"] + for name in ["AVE-Catalog", "AVE-Records-SHA256", "ETag", "AVE-Source-Commit"]: + assert name in exposed + + +def test_deprecated_records_are_served_with_their_status(client): + response = client.get("/v1/records/AVE-0003") + + assert response.status_code == 200 + assert response.json()["status"] == "deprecated" + assert "AVE-0003" in [r["ave_id"] for r in client.get("/v1/records").json()] + + +# --- ETag and 304 ------------------------------------------------------------------- + + +@pytest.mark.parametrize("route", DATA_ROUTES) +def test_a_matching_if_none_match_gets_304(client, route): + etag = client.get("/v1" + route).headers["ETag"] + + response = client.get("/v1" + route, headers={"If-None-Match": etag}) + + assert response.status_code == 304 + assert response.content == b"" + assert response.headers["ETag"] == etag + assert response.headers["AVE-Catalog"] == "latest" + + +def test_the_alias_and_v1_share_an_etag(client): + etag = client.get("/records/AVE-0001").headers["ETag"] + + assert client.get("/v1/records/AVE-0001").headers["ETag"] == etag + assert ( + client.get("/v1/records/AVE-0001", headers={"If-None-Match": etag}).status_code + == 304 + ) + + +@pytest.mark.parametrize( + "header", + ['"something-else"', "", 'W/"something-else", "another"'], +) +def test_a_non_matching_if_none_match_gets_the_body(client, header): + response = client.get("/v1/records", headers={"If-None-Match": header}) + + assert response.status_code == 200 + assert len(response.json()) == len(FAKE_RECORDS) + + +def test_weak_list_and_wildcard_forms_match(client): + etag = client.get("/v1/records").headers["ETag"] + + for header in [f"W/{etag}", f'"other", {etag}', "*"]: + assert ( + client.get("/v1/records", headers={"If-None-Match": header}).status_code + == 304 + ) + + +def test_etags_differ_between_resources_and_queries(client): + etags = { + client.get(path).headers["ETag"] + for path in [ + "/v1/records", + "/v1/records/AVE-0001", + "/v1/records/AVE-0002", + "/v1/records/AVE-0001/mitigation", + "/v1/search?q=prompt", + "/v1/search?q=excessive", + ] + } + + assert len(etags) == 6 + + +def test_the_etag_changes_when_the_records_change(client): + before = client.get("/v1/records/AVE-0001").headers["ETag"] + main._cache["catalog"] = dict(FAKE_CATALOG, sha256="0" * 64) + + after = client.get("/v1/records/AVE-0001") + + assert after.headers["ETag"] != before + assert ( + client.get( + "/v1/records/AVE-0001", headers={"If-None-Match": before} + ).status_code + == 200 + ) + + +# --- pinning to a release ----------------------------------------------------------- + + +@pytest.mark.parametrize("route", DATA_ROUTES) +def test_a_pinned_release_is_served_with_its_own_headers(client, github, route): + joiner = "&" if "?" in route else "?" + response = client.get(f"/v1{route}{joiner}release={TAG}") + + assert response.status_code == 200 + headers = response.headers + assert headers["AVE-Catalog"] == TAG + assert headers["AVE-Source-Commit"] == COMMIT + assert headers["AVE-Records-SHA256"] == _sha( + github.releases[TAG]["ave-records.json"] + ) + assert headers["Cache-Control"] == "public, max-age=86400, immutable" + assert headers["Last-Modified"] == "Thu, 29 Oct 2026 04:51:04 GMT" + + +def test_a_pinned_release_serves_the_release_data_not_latest(client, github): + body = client.get(f"/v1/records/AVE-0001?release={TAG}").json() + + assert body["title"] == "Prompt injection via tool output, as pinned" + assert client.get(f"/v1/records/AVE-0002?release={TAG}").status_code == 404 + assert client.get("/v1/records/AVE-0002").status_code == 200 + + +def test_a_verified_release_is_downloaded_once(client, github): + client.get(f"/v1/records?release={TAG}") + first = github.downloads() + + client.get(f"/v1/records/AVE-0001?release={TAG}") + client.get(f"/records?release={TAG}") + + assert first == 3 + assert github.downloads() == 3 + + +def test_an_unknown_release_is_a_404_with_a_clear_message(client, github): + response = client.get("/v1/records?release=v9.9.9") + + assert response.status_code == 404 + assert "v9.9.9" in response.json()["detail"] + assert "/v1/releases" in response.json()["detail"] + assert "v9.9.9" not in main._release_cache + + +def test_a_release_without_the_three_assets_is_a_404(client, github): + files = make_release(tag="v1.1.0") + del files["SHA256SUMS"] + github.releases["v1.1.0"] = files + + assert client.get("/v1/records?release=v1.1.0").status_code == 404 + + +@pytest.mark.parametrize("bad", ["1.2.0", "v1.2", "latest", "v1.2.0-rc1"]) +def test_a_malformed_release_is_refused_before_any_download(client, github, bad): + response = client.get(f"/v1/records?release={bad}") + + assert response.status_code == 422 + assert github.downloads() == 0 + + +def test_a_tampered_asset_is_refused_and_not_cached(client, github): + github.releases[TAG]["ave-records.json"] += b" " + + first = client.get(f"/v1/records?release={TAG}") + second = client.get(f"/v1/records?release={TAG}") + + assert first.status_code == second.status_code == 502 + assert "SHA256SUMS" in first.json()["detail"] + assert TAG not in main._release_cache + assert github.downloads() == 6 # tried again, nothing was cached + + +def test_a_release_whose_upstream_fails_is_a_502_and_not_cached( + client, github, monkeypatch +): + def failing_handler(request): + raise httpx.ConnectError("simulated outage") + + monkeypatch.setattr(github, "handler", failing_handler) + + response = client.get(f"/v1/records?release={TAG}") + + assert response.status_code == 502 + assert TAG not in main._release_cache + + +def test_the_release_cache_is_bounded(client, github): + tags = [f"v1.{minor}.0" for minor in range(2, 2 + main.RELEASE_CACHE_SIZE + 1)] + for tag in tags: + github.releases[tag] = make_release(tag=tag) + assert client.get(f"/v1/records?release={tag}").status_code == 200 + + assert list(main._release_cache) == tags[1:] + client.get(f"/v1/records?release={tags[0]}") + assert github.downloads() == 3 * (len(tags) + 1) + + +def test_a_pinned_release_has_its_own_etag(client, github): + latest = client.get("/v1/records/AVE-0001").headers["ETag"] + pinned = client.get(f"/v1/records/AVE-0001?release={TAG}") + + assert pinned.headers["ETag"] != latest + assert ( + client.get( + f"/v1/records/AVE-0001?release={TAG}", + headers={"If-None-Match": pinned.headers["ETag"]}, + ).status_code + == 304 + ) + + +# --- release verification, check by check ----------------------------------------- + + +def test_a_consistent_release_verifies(): + records, catalog = main._verify_release(TAG, make_release()) + + assert records == RELEASE_RECORDS + assert catalog["release"] == TAG + assert catalog["source_commit"] == COMMIT + + +def _tampered(change) -> dict[str, bytes]: + files = make_release() + change(files) + return files + + +def _resummed(files: dict[str, bytes]) -> dict[str, bytes]: + """Rewrite SHA256SUMS to match whatever the files now hold, so only the + manifest checks can catch the change.""" + files["SHA256SUMS"] = ( + f"{_sha(files['ave-records.json'])} ave-records.json\n" + f"{_sha(files['ave-records.manifest.json'])} ave-records.manifest.json\n" + ).encode() + return files + + +@pytest.mark.parametrize( + "files, reason", + [ + ( + _tampered(lambda f: f.update({"ave-records.json": b"[]\n"})), + "ave-records.json does not match SHA256SUMS", + ), + ( + _tampered(lambda f: f.update({"ave-records.manifest.json": b"{}\n"})), + "ave-records.manifest.json does not match SHA256SUMS", + ), + (_resummed(make_release(sha256="0" * 64)), "sha256 is not that of"), + (_resummed(make_release(release="v1.1.0")), "names release 'v1.1.0'"), + (_resummed(make_release(record_count=5)), "record_count"), + (_resummed(make_release(schema_version="1.1.0")), "schema_version"), + (_resummed(make_release(source_commit="abc123")), "source_commit"), + ], +) +def test_each_inconsistency_is_refused(files, reason): + with pytest.raises(main.ReleaseUnverified, match=reason): + main._verify_release(TAG, files) + + +# --- /v1/releases ------------------------------------------------------------------- + + +def _api_release(tag, assets, published="2026-10-29T05:09:35Z"): + base = f"https://github.com/aveproject/ave/releases/download/{tag}/" + return { + "tag_name": tag, + "draft": False, + "published_at": published, + "assets": [ + {"name": name, "browser_download_url": base + name} for name in assets + ], + } + + +API_RELEASES = [ + _api_release(TAG, ["ave-records.json", "ave-records.manifest.json", "SHA256SUMS"]), + _api_release("v1.1.0", ["ave-records-v1.1.0.json"], "2026-06-24T00:00:00Z"), + _api_release("v1.0.0", [], "2026-06-18T00:00:00Z"), +] + + +def test_releases_lists_each_release_from_the_api(client, github): + github.api_releases = API_RELEASES + + response = client.get("/v1/releases") + + assert response.status_code == 200 + assert response.headers["AVE-API-Version"] == "1" + current, older, first = response.json() + manifest = json.loads(github.releases[TAG]["ave-records.manifest.json"]) + assert current == { + "tag": TAG, + "published_at": "2026-10-29T05:09:35Z", + "pinnable": True, + "schema_version": "1.2.0", + "record_count": 1, + "sha256": manifest["sha256"], + "assets": { + name: f"https://github.com/aveproject/ave/releases/download/{TAG}/{name}" + for name in ["ave-records.json", "ave-records.manifest.json", "SHA256SUMS"] + }, + } + assert older["pinnable"] is False and older["record_count"] is None + assert list(older["assets"]) == ["ave-records-v1.1.0.json"] + assert first["tag"] == "v1.0.0" and first["assets"] == {} + + +def test_releases_are_cached_for_an_hour_then_revalidated(client, github): + github.api_releases = API_RELEASES + client.get("/v1/releases") + client.get("/v1/releases") + assert len(github.api_calls()) == 1 + + main._releases["fetched_at"] = time.time() - main.RELEASES_REFRESH_SECONDS - 1 + downloads_before = github.downloads() + response = client.get("/v1/releases") + + revalidation = github.api_calls()[-1] + assert len(github.api_calls()) == 2 + assert revalidation.headers["if-none-match"] == '"releases-1"' + assert response.json()[0]["tag"] == TAG + assert github.downloads() == downloads_before # 304: nothing re-downloaded + + +def test_releases_serve_the_last_list_when_github_fails(client, github): + github.api_releases = API_RELEASES + client.get("/v1/releases") + main._releases["fetched_at"] = 0.0 + github.api_status = 503 + + response = client.get("/v1/releases") + + assert response.status_code == 200 + assert response.json()[0]["tag"] == TAG + + +def test_releases_are_a_502_when_github_fails_and_nothing_is_cached(client, github): + github.api_status = 503 + + response = client.get("/v1/releases") + + assert response.status_code == 502 + + +# --- the latest data and its manifest ----------------------------------------------- + + +def _latest_files(records=None, **manifest_overrides): + body = json.dumps(records or FAKE_RECORDS).encode() + manifest = { + "schema_version": "1.2.0", + "record_count": len(records or FAKE_RECORDS), + "generated_at": "2026-10-28T23:50:03.886Z", + "source": "https://github.com/aveproject/ave", + "sha256": _sha(body), + } + manifest.update(manifest_overrides) + return body, manifest + + +def test_the_latest_catalog_comes_from_its_manifest(): + body, manifest = _latest_files() + + records, catalog = main._latest_catalog(body, manifest) + + assert records == FAKE_RECORDS + assert catalog == dict(FAKE_CATALOG, sha256=_sha(body)) + + +@pytest.mark.parametrize( + "overrides, message", + [({"sha256": "0" * 64}, "sha256"), ({"record_count": 99}, "record_count")], +) +def test_a_latest_manifest_from_another_commit_is_refused(overrides, message): + body, manifest = _latest_files(**overrides) + + with pytest.raises(ValueError, match=message): + main._latest_catalog(body, manifest) + + +def test_the_real_refresh_reads_the_records_and_their_manifest(monkeypatch): + body, manifest = _latest_files() + + def handler(request): + if request.url.path.endswith("ave-records-latest.json"): + return httpx.Response(200, content=body) + if request.url.path.endswith("ave-records-latest.manifest.json"): + return httpx.Response(200, json=manifest) + return httpx.Response(404) + + monkeypatch.setattr( + main, + "_client", + lambda **kwargs: httpx.AsyncClient( + transport=httpx.MockTransport(handler), **kwargs + ), + ) + monkeypatch.setattr(main, "_refresh_cache", REAL_REFRESH_CACHE) + + with TestClient(main.app) as test_client: + response = test_client.get("/v1/records") + + assert response.status_code == 200 + assert response.headers["AVE-Records-SHA256"] == _sha(body) + + +def test_outbound_calls_have_an_explicit_timeout(): + assert main._client().timeout == httpx.Timeout(main.OUTBOUND_TIMEOUT_SECONDS) + + +# --- rate limiting covers the new routes -------------------------------------------- + + +@pytest.mark.parametrize("path", ["/v1/", "/v1/records", "/v1/records/AVE-0001"]) +def test_the_v1_paths_are_rate_limited(client, path): + # slowapi counts per IP per path, so each path has its own 60 a minute + for _ in range(60): + assert client.get(path).status_code == 200 + + assert client.get(path).status_code == 429 + + +def test_releases_are_rate_limited(client, github): + github.api_releases = API_RELEASES + for _ in range(60): + assert client.get("/v1/releases").status_code == 200 + + assert client.get("/v1/releases").status_code == 429