From 3c174a5de7c242f252bf680d58dd246911aced31 Mon Sep 17 00:00:00 2001 From: Scott Robinson Date: Wed, 4 Nov 2026 00:23:11 +0000 Subject: [PATCH] feat(container): publish a Cassandra-backed ExtendDB image Adds extenddb/extenddb-cassandra to the container release pipeline for the 1.0 release, built and released the same way as extenddb-mongodb. - Dockerfile.cassandra: the MongoDB image's Dockerfile with the cassandra feature, its notices file and labels. - docker-compose.cassandra.yml: local reference stack and what the release smoke test runs. One Cassandra 4.1 node with PasswordAuthenticator and CassandraAuthorizer, not published on the host; the node's entrypoint checks that both edits of cassandra.yaml took, since sed exits 0 whether or not a line matched. cassandra-auth-init replaces the stock superuser password on first start through 0600 cqlsh credentials files, fails closed if password authentication is off, and gives up after five minutes. All three passwords are required variables. Later starts run the migrate preview and run migrate --yes only when migrations are pending. - ci/smoke-test-cassandra-container.sh: image identity and hardening, Cassandra authentication and exposure, the application role's credentials and grants, a DynamoDB round trip, persistence across restart, SIGTERM, and, after recreating the stack, the bootstrap's migrate preview and the replaced superuser password. In build mode the image is built with docker build before the bounded compose up. A failed assertion names its line. - smoke-cassandra-image.yml runs that script in build mode on amd64 and arm64 runners for every pull request that changes the image, the compose file or the script, so a release dispatch meets a script that has passed on both. - release-image, promote-image, ghcr-mirror, sign-image and dockerhub-description gain extenddb-cassandra. release-image's gate now refuses a tag that has no Dockerfile or smoke script for the chosen image. The licences push trigger watches SOFTWARE-LICENSE-NOTICES-CASSANDRA.html. promote-image and sign-image read the image input from the environment instead of interpolating it into a script. - The two existing smoke scripts' "tool is absent" checks used `! command -v`, which never fails under `sh -e`; they now exit explicitly. - Docs: docker/README-cassandra.md, the Docker Hub overview, README.md, and docs/cassandra-setup.md (container image section; keyspace replication procedure including system_auth; the local node published on loopback only). Signed-off-by: Scott Robinson --- .github/workflows/dockerhub-description.yml | 24 +- .github/workflows/ghcr-mirror.yml | 3 +- .github/workflows/licenses.yml | 1 + .github/workflows/promote-image.yml | 8 +- .github/workflows/release-image.yml | 48 +- .github/workflows/sign-image.yml | 6 +- .github/workflows/smoke-cassandra-image.yml | 89 ++++ Dockerfile.cassandra | 76 +++ README.md | 7 +- ci/smoke-test-cassandra-container.sh | 485 ++++++++++++++++++++ ci/smoke-test-container.sh | 7 +- ci/smoke-test-mongodb-container.sh | 8 +- docker-compose.cassandra.yml | 261 +++++++++++ docker/README-cassandra.md | 265 +++++++++++ docker/dockerhub-overview-cassandra.md | 126 +++++ docs/cassandra-setup.md | 73 ++- 16 files changed, 1439 insertions(+), 48 deletions(-) create mode 100644 .github/workflows/smoke-cassandra-image.yml create mode 100644 Dockerfile.cassandra create mode 100755 ci/smoke-test-cassandra-container.sh create mode 100644 docker-compose.cassandra.yml create mode 100644 docker/README-cassandra.md create mode 100644 docker/dockerhub-overview-cassandra.md diff --git a/.github/workflows/dockerhub-description.yml b/.github/workflows/dockerhub-description.yml index 7f6be2863..70410872b 100644 --- a/.github/workflows/dockerhub-description.yml +++ b/.github/workflows/dockerhub-description.yml @@ -42,6 +42,7 @@ on: - 'docker/dockerhub-overview.md' - 'docker/dockerhub-overview-dev.md' - 'docker/dockerhub-overview-mongodb.md' + - 'docker/dockerhub-overview-cassandra.md' workflow_dispatch: inputs: image: @@ -51,6 +52,7 @@ on: options: - extenddb-postgres - extenddb-mongodb + - extenddb-cassandra - extenddb-dev permissions: @@ -58,7 +60,7 @@ permissions: # One group for every run: publishes are quick and rare, and serializing them # is simpler and strictly safer than per-image groups now that one push can -# carry both images. +# carry several images. concurrency: group: dockerhub-description cancel-in-progress: false @@ -71,8 +73,8 @@ jobs: # image. Push: the image(s) whose Overview file changed in the push, # resolved via the compare API rather than a full-history checkout. If the # push range cannot be resolved (forced push, branch creation), fall back to - # publishing both — the PATCH is idempotent, so over-publishing is harmless - # and strictly better than silently skipping one. + # publishing every image — the PATCH is idempotent, so over-publishing is + # harmless and strictly better than silently skipping one. plan: runs-on: ubuntu-latest timeout-minutes: 5 @@ -92,7 +94,7 @@ jobs: set -euo pipefail if [[ "$EVENT" == "workflow_dispatch" ]]; then case "$INPUT_IMAGE" in - extenddb-postgres|extenddb-mongodb|extenddb-dev) ;; + extenddb-postgres|extenddb-mongodb|extenddb-cassandra|extenddb-dev) ;; *) echo "::error::unknown image '$INPUT_IMAGE'"; exit 1 ;; esac IMAGES=$(jq -cn --arg i "$INPUT_IMAGE" '[$i]') @@ -107,13 +109,15 @@ jobs: fi IMAGES='[]' if [[ -z "$CHANGED" ]]; then - # Unresolvable range: publish both (idempotent). - IMAGES='["extenddb-postgres","extenddb-mongodb","extenddb-dev"]' + # Unresolvable range: publish every image (idempotent). + IMAGES='["extenddb-postgres","extenddb-mongodb","extenddb-cassandra","extenddb-dev"]' else grep -qx 'docker/dockerhub-overview.md' <<< "$CHANGED" \ && IMAGES=$(jq -c '. + ["extenddb-postgres"]' <<< "$IMAGES") grep -qx 'docker/dockerhub-overview-mongodb.md' <<< "$CHANGED" \ && IMAGES=$(jq -c '. + ["extenddb-mongodb"]' <<< "$IMAGES") + grep -qx 'docker/dockerhub-overview-cassandra.md' <<< "$CHANGED" \ + && IMAGES=$(jq -c '. + ["extenddb-cassandra"]' <<< "$IMAGES") grep -qx 'docker/dockerhub-overview-dev.md' <<< "$CHANGED" \ && IMAGES=$(jq -c '. + ["extenddb-dev"]' <<< "$IMAGES") fi @@ -160,6 +164,10 @@ jobs: FILE='docker/dockerhub-overview-mongodb.md' SHORT='DynamoDB-compatible API adapter backed by your own MongoDB 7+ replica set' ;; + extenddb-cassandra) + FILE='docker/dockerhub-overview-cassandra.md' + SHORT='DynamoDB-compatible API adapter backed by your own Apache Cassandra 4.1+ cluster' + ;; extenddb-dev) FILE='docker/dockerhub-overview-dev.md' SHORT='DynamoDB-compatible API over SQLite for local dev and CI. Plain HTTP, not for production.' @@ -276,6 +284,10 @@ jobs: FILE='docker/dockerhub-overview-mongodb.md' SHORT='DynamoDB-compatible API adapter backed by your own MongoDB 7+ replica set' ;; + extenddb-cassandra) + FILE='docker/dockerhub-overview-cassandra.md' + SHORT='DynamoDB-compatible API adapter backed by your own Apache Cassandra 4.1+ cluster' + ;; extenddb-dev) FILE='docker/dockerhub-overview-dev.md' SHORT='DynamoDB-compatible API over SQLite for local dev and CI. Plain HTTP, not for production.' diff --git a/.github/workflows/ghcr-mirror.yml b/.github/workflows/ghcr-mirror.yml index 7f28d3ec7..a5245b69a 100644 --- a/.github/workflows/ghcr-mirror.yml +++ b/.github/workflows/ghcr-mirror.yml @@ -50,6 +50,7 @@ on: options: - extenddb-postgres - extenddb-mongodb + - extenddb-cassandra default: extenddb-postgres # Copying between registries needs no repository content at all. @@ -86,7 +87,7 @@ jobs: run: | set -euo pipefail case "$RAW_IMAGE" in - extenddb-postgres|extenddb-mongodb) ;; + extenddb-postgres|extenddb-mongodb|extenddb-cassandra) ;; *) echo "::error::image '$RAW_IMAGE' is not in the allowlist"; exit 1 ;; esac KIND='' diff --git a/.github/workflows/licenses.yml b/.github/workflows/licenses.yml index a109b9751..e870cc0a1 100644 --- a/.github/workflows/licenses.yml +++ b/.github/workflows/licenses.yml @@ -30,6 +30,7 @@ on: - 'Cargo.lock' - 'SOFTWARE-LICENSE-NOTICES.html' - 'SOFTWARE-LICENSE-NOTICES-MONGODB.html' + - 'SOFTWARE-LICENSE-NOTICES-CASSANDRA.html' - 'SOFTWARE-LICENSE-NOTICES-DEV.html' - 'about.toml' - 'about.hbs' diff --git a/.github/workflows/promote-image.yml b/.github/workflows/promote-image.yml index 0f3f63e71..9c0254d7f 100644 --- a/.github/workflows/promote-image.yml +++ b/.github/workflows/promote-image.yml @@ -73,6 +73,7 @@ on: # ECR Public does not auto-create repositories on push. - extenddb-postgres - extenddb-mongodb + - extenddb-cassandra default: extenddb-postgres permissions: @@ -117,13 +118,14 @@ jobs: RAW_TAG: ${{ inputs.candidate_tag }} RAW_DIGEST: ${{ inputs.expected_digest }} RAW_MOVE_LATEST: ${{ inputs.move_latest }} + RAW_IMAGE: ${{ inputs.image }} run: | set -euo pipefail # Local allowlist enforcement: API-side choice enforcement is # undocumented, so the security property is proven here. - case "${{ inputs.image }}" in - extenddb-postgres|extenddb-mongodb) ;; - *) echo "::error::image '${{ inputs.image }}' is not in the allowlist"; exit 1 ;; + case "$RAW_IMAGE" in + extenddb-postgres|extenddb-mongodb|extenddb-cassandra) ;; + *) echo "::error::image '$RAW_IMAGE' is not in the allowlist"; exit 1 ;; esac if [[ ! "$RAW_VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then echo "::error::version '$RAW_VERSION' is not strict semver (MAJOR.MINOR.PATCH)" diff --git a/.github/workflows/release-image.yml b/.github/workflows/release-image.yml index 40e18dee9..f9226d925 100644 --- a/.github/workflows/release-image.yml +++ b/.github/workflows/release-image.yml @@ -73,6 +73,7 @@ on: options: - extenddb-postgres - extenddb-mongodb + - extenddb-cassandra default: extenddb-postgres # Nothing is committed, so no write scope on the repo is needed. @@ -146,6 +147,13 @@ jobs: SMOKE_SCRIPT='./ci/smoke-test-mongodb-container.sh' SMOKE_ENV='EXTENDDB_MONGODB_IMAGE' ;; + extenddb-cassandra) + IMAGE_REPO='extenddb/extenddb-cassandra' + DOCKERFILE='Dockerfile.cassandra' + LICENSE_BACKEND='cassandra' + SMOKE_SCRIPT='./ci/smoke-test-cassandra-container.sh' + SMOKE_ENV='EXTENDDB_CASSANDRA_IMAGE' + ;; *) echo "::error::image '$RAW_IMAGE' is not in the allowlist" exit 1 @@ -201,17 +209,25 @@ jobs: # distributed image than in source. The script pins cargo-about itself # and refuses to run against any other version. env: + IMAGE: ${{ steps.meta.outputs.image }} + DOCKERFILE: ${{ steps.meta.outputs.dockerfile }} + SMOKE_SCRIPT: ${{ steps.meta.outputs.smoke_script }} LICENSE_BACKEND: ${{ steps.meta.outputs.license_backend }} run: | set -euo pipefail - if [[ "$LICENSE_BACKEND" == mongodb ]] \ - && ! grep -q -- '--backend' devtools/generate-software-license-notices; then - echo "::error::tagged commit does not contain MongoDB image license support; use a tag created after MongoDB container support was merged" - exit 1 - fi + # A tag cut before an image's container support merged has no + # Dockerfile or smoke test for it; say so before a build is paid for. + # (Dockerfile.mongodb and the generator's --backend flag arrived + # together, so this also covers the earlier MongoDB-specific check.) + for f in "$DOCKERFILE" "$SMOKE_SCRIPT"; do + if [[ ! -f "$f" ]]; then + echo "::error::tagged commit has no ${f}; use a tag created after ${IMAGE} container support was merged" + exit 1 + fi + done cargo install --locked --version 0.9.0 --features cli cargo-about ./devtools/generate-software-license-notices --check \ - --backend '${{ steps.meta.outputs.license_backend }}' + --backend "$LICENSE_BACKEND" - name: Gate - the release freezes the catalog migrations it ships # RELEASED_THROUGH must reach the last migration id, so each @@ -308,21 +324,23 @@ jobs: || { echo "::error::expected ${{ matrix.arch }}, image reports $ACTUAL"; exit 1; } - name: Smoke test the built image - # EXTENDDB_IMAGE puts ci/smoke-test-container.sh into prebuilt mode: it - # reads VERSION / VCS_REF / BUILD_DATE back off the image labels, runs - # `compose up --no-build`, and asserts every container is running this - # exact image ID. + # Each image has its own smoke script and its own image variable (the + # gate picks both). Setting the variable puts the script into prebuilt + # mode: it reads VERSION / VCS_REF / BUILD_DATE back off the image + # labels, runs `compose up --no-build`, and asserts every container is + # running this exact image ID. env: IMAGE: ${{ needs.gate.outputs.image_repo }}:sha-${{ needs.gate.outputs.sha }}-${{ matrix.arch }} SMOKE_ENV: ${{ needs.gate.outputs.smoke_env }} SMOKE_SCRIPT: ${{ needs.gate.outputs.smoke_script }} run: | set -euo pipefail - if [[ "$SMOKE_ENV" == 'EXTENDDB_MONGODB_IMAGE' ]]; then - EXTENDDB_MONGODB_IMAGE="$IMAGE" "$SMOKE_SCRIPT" - else - EXTENDDB_IMAGE="$IMAGE" "$SMOKE_SCRIPT" - fi + case "$SMOKE_ENV" in + EXTENDDB_IMAGE) EXTENDDB_IMAGE="$IMAGE" "$SMOKE_SCRIPT" ;; + EXTENDDB_MONGODB_IMAGE) EXTENDDB_MONGODB_IMAGE="$IMAGE" "$SMOKE_SCRIPT" ;; + EXTENDDB_CASSANDRA_IMAGE) EXTENDDB_CASSANDRA_IMAGE="$IMAGE" "$SMOKE_SCRIPT" ;; + *) echo "::error::unknown smoke-test variable '$SMOKE_ENV'"; exit 1 ;; + esac - name: Save the tested image as an artifact # docker save preserves the image ID, so what publish pushes is exactly diff --git a/.github/workflows/sign-image.yml b/.github/workflows/sign-image.yml index b744de4f0..9841930bf 100644 --- a/.github/workflows/sign-image.yml +++ b/.github/workflows/sign-image.yml @@ -37,6 +37,7 @@ on: # signing, which distinguishes tiers automatically. - extenddb-postgres - extenddb-mongodb + - extenddb-cassandra - extenddb-dev default: extenddb-postgres reason: @@ -80,7 +81,7 @@ jobs: run: | set -euo pipefail case "$RAW_IMAGE" in - extenddb-postgres|extenddb-mongodb|extenddb-dev) ;; + extenddb-postgres|extenddb-mongodb|extenddb-cassandra|extenddb-dev) ;; *) echo "::error::image '$RAW_IMAGE' is not in the allowlist"; exit 1 ;; esac if [[ ! "$RAW_DIGEST" =~ ^sha256:[0-9a-f]{64}$ ]]; then @@ -154,9 +155,10 @@ jobs: # Free text; reaches the shell only through the environment, per # the repo convention (inputs.image above is allowlist-validated). REASON: ${{ inputs.reason }} + IMAGE: ${{ inputs.image }} run: | { - echo "### ${{ inputs.image }} signed (break-glass path)" + echo "### ${IMAGE} signed (break-glass path)" echo "" echo "| field | value |" echo "|---|---|" diff --git a/.github/workflows/smoke-cassandra-image.yml b/.github/workflows/smoke-cassandra-image.yml new file mode 100644 index 000000000..312232803 --- /dev/null +++ b/.github/workflows/smoke-cassandra-image.yml @@ -0,0 +1,89 @@ +# Copyright 2026 ExtendDB contributors +# SPDX-License-Identifier: Apache-2.0 +# +# Builds the Cassandra-backed image and runs ci/smoke-test-cassandra-container.sh +# against it, on both release architectures, whenever a pull request touches +# the image, its reference stack or the smoke test. +# +# release-image.yml runs the same script against the tagged commit, but only at +# release. A script or stack that fails there, on ubuntu-24.04-arm in +# particular, leaves that tag without a Cassandra image and needs a new tag to +# fix. This runs the same thing on the pull request that changes it, in the +# script's build mode (the image built by the script), so a release dispatch +# meets a script that has already passed on both runners. Nothing here is +# published or needs credentials. + +name: Cassandra image smoke + +on: + pull_request: + paths: + - 'Dockerfile.cassandra' + - 'docker-compose.cassandra.yml' + - 'ci/smoke-test-cassandra-container.sh' + - '.github/workflows/smoke-cassandra-image.yml' + workflow_dispatch: + +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.event_name == 'pull_request' && github.event.pull_request.number || github.run_id }} + cancel-in-progress: ${{ github.event_name == 'pull_request' }} + +env: + # Pinned AWS CLI for runners that lack it (the smoke test needs `aws`), as + # in release-image.yml. + AWSCLI_VERSION: 2.31.6 + AWSCLI_SHA256_X86_64: 45fdcc3003056b3e85c23776636e74208dc2fc16f26278acdbe8eafe3d4e752b + AWSCLI_SHA256_AARCH64: 517dec4ce83fabe7ace6164bb6189cdb686ef816332ccf83e81999e3fb57cad5 + +jobs: + smoke: + strategy: + fail-fast: false + matrix: + include: + - arch: amd64 + runner: ubuntu-latest + - arch: arm64 + runner: ubuntu-24.04-arm # native arm64, no qemu + name: smoke (${{ matrix.arch }}) + runs-on: ${{ matrix.runner }} + timeout-minutes: 60 # a hung Rust build must not burn the 6h default + steps: + - name: Check out + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + + - name: Ensure smoke-test prerequisites + # The AWS CLI is not guaranteed on every runner image. Install is + # pinned to an exact version and verified against a recorded + # checksum: never execute a mutable unverified download. + run: | + set -euo pipefail + if ! command -v aws >/dev/null 2>&1; then + case "$(uname -m)" in + x86_64) EXPECTED="$AWSCLI_SHA256_X86_64"; PKG="awscli-exe-linux-x86_64-${AWSCLI_VERSION}.zip" ;; + aarch64) EXPECTED="$AWSCLI_SHA256_AARCH64"; PKG="awscli-exe-linux-aarch64-${AWSCLI_VERSION}.zip" ;; + *) echo "::error::unsupported architecture $(uname -m)"; exit 1 ;; + esac + curl -fsSL "https://awscli.amazonaws.com/${PKG}" -o /tmp/awscli.zip + echo "${EXPECTED} /tmp/awscli.zip" | sha256sum -c - \ + || { echo "::error::AWS CLI download failed checksum verification"; exit 1; } + unzip -q /tmp/awscli.zip -d /tmp + sudo /tmp/aws/install + fi + for c in docker aws python3 timeout; do + command -v "$c" >/dev/null || { echo "::error::missing $c"; exit 1; } + done + docker compose version + + - name: Build the image and run the smoke test + # Build mode: the script builds the image with `docker build` from + # this checkout, then starts the reference stack against it and + # removes the image afterwards. + env: + VCS_REF: ${{ github.event.pull_request.head.sha || github.sha }} + run: ci/smoke-test-cassandra-container.sh diff --git a/Dockerfile.cassandra b/Dockerfile.cassandra new file mode 100644 index 000000000..9652feabf --- /dev/null +++ b/Dockerfile.cassandra @@ -0,0 +1,76 @@ +# syntax=docker/dockerfile:1.7@sha256:a57df69d0ea827fb7266491f2813635de6f17269be881f696fbfdf2d83dda33e + +# Production ExtendDB image with the Cassandra backend. Cassandra itself is an +# external service: use an Apache Cassandra 4.1+ cluster you operate, or the +# optional docker-compose.cassandra.yml reference stack for local development. + +FROM public.ecr.aws/docker/library/rust:1.88-bookworm@sha256:af306cfa71d987911a781c37b59d7d67d934f49684058f96cf72079c3626bfe0 AS build +ARG VCS_REF +ARG BUILD_DATE +WORKDIR /src +RUN apt-get update \ + && apt-get install -y --no-install-recommends cmake=3.25.1-1 \ + && rm -rf /var/lib/apt/lists/* +COPY Cargo.toml Cargo.lock ./ +COPY crates ./crates +RUN test -n "${VCS_REF}" \ + && test "${VCS_REF}" != unknown \ + && test -n "${BUILD_DATE}" \ + && test "${BUILD_DATE}" != unknown \ + && EXTENDDB_GIT_HASH="${VCS_REF}" \ + EXTENDDB_BUILD_TIME="${BUILD_DATE}" \ + AWS_LC_SYS_USE_SYSTEM=0 \ + cargo build --locked --release \ + -p extenddb --no-default-features --features cassandra + +FROM public.ecr.aws/debian/debian:bookworm-slim@sha256:8eeeb3e089a27d34c398a5580fe479b71a457cba535f59d85e52a303610ec583 AS runtime +ARG VERSION +ARG VCS_REF +ARG BUILD_DATE +ARG DEBIAN_FRONTEND=noninteractive +# The runtime base is explicitly approved for this public OSS distribution. +# Keep both references digest-pinned and update them only through reviewed CRs. +# The base digest is a point-in-time snapshot, and package versions in this +# layer are not pinned: apt-get upgrade installs the bookworm-security updates +# published since the snapshot, so a release build does not ship a package +# Debian has already fixed. --with-new-pkgs installs an update that needs a new +# dependency instead of holding it back. +RUN test -n "${VERSION}" \ + && test -n "${VCS_REF}" \ + && test "${VCS_REF}" != unknown \ + && test -n "${BUILD_DATE}" \ + && test "${BUILD_DATE}" != unknown \ + && apt-get update \ + && apt-get upgrade -y --with-new-pkgs --no-install-recommends \ + && apt-get install -y --no-install-recommends \ + ca-certificates \ + tini=0.19.0-1+b3 \ + && rm -rf /var/lib/apt/lists/* \ + && groupadd --gid 10001 extenddb \ + && useradd --uid 10001 --gid 10001 --create-home \ + --home-dir /var/lib/extenddb --shell /usr/sbin/nologin extenddb \ + && install -d -o 10001 -g 10001 /usr/share/doc/extenddb +# ExtendDB needs no setuid/setgid executable; remove those privilege paths. +RUN find / -xdev -type f -perm /6000 -exec chmod a-s {} + + +COPY --from=build --chmod=0555 /src/target/release/extenddb /usr/local/bin/extenddb +COPY --chmod=0444 LICENSE NOTICE SOFTWARE-LICENSE-NOTICES-CASSANDRA.html /usr/share/doc/extenddb/ + +LABEL org.opencontainers.image.title="ExtendDB Cassandra backend" \ + org.opencontainers.image.description="DynamoDB-compatible API backed by an external Apache Cassandra 4.1+ cluster" \ + org.opencontainers.image.source="https://github.com/ExtendDB/extenddb" \ + org.opencontainers.image.url="https://github.com/ExtendDB/extenddb" \ + org.opencontainers.image.version="${VERSION}" \ + org.opencontainers.image.revision="${VCS_REF}" \ + org.opencontainers.image.created="${BUILD_DATE}" \ + org.opencontainers.image.licenses="Apache-2.0" + +ENV HOME=/var/lib/extenddb +WORKDIR /var/lib/extenddb +USER 10001:10001 +EXPOSE 18443 +STOPSIGNAL SIGTERM +HEALTHCHECK --interval=10s --timeout=5s --start-period=30s --retries=5 \ + CMD ["extenddb", "healthcheck", "--config", "/var/lib/extenddb/extenddb.toml"] +ENTRYPOINT ["/usr/bin/tini", "--", "/usr/local/bin/extenddb"] +CMD ["serve", "--config", "/var/lib/extenddb/extenddb.toml", "--foreground"] diff --git a/README.md b/README.md index c6d931af8..9f5e9f664 100755 --- a/README.md +++ b/README.md @@ -115,9 +115,10 @@ key_path = "/etc/extenddb/tls/key.pem" ## Running in Containers The production images contain ExtendDB only. The storage backend runs as a separate service. See -[`docker/README.md`](docker/README.md) for the PostgreSQL image and [`docker/README-mongodb.md`](docker/README-mongodb.md) for the MongoDB image, -external database requirements, hardened local Compose stacks, smoke tests, and explicit init/migrate/serve -production lifecycle. The SQLite and Cassandra backends have no dedicated container guide yet. +[`docker/README.md`](docker/README.md) for the PostgreSQL image, [`docker/README-mongodb.md`](docker/README-mongodb.md) for the MongoDB image, +and [`docker/README-cassandra.md`](docker/README-cassandra.md) for the Cassandra image (published from 1.0): external database requirements, +hardened local Compose stacks, smoke tests, and explicit init/migrate/serve production lifecycle. The SQLite +backend has no production container image; [`docs/dev-image.md`](docs/dev-image.md) covers the SQLite development image. By default `extenddb serve` daemonizes itself, which doesn't play well with container runtimes and process supervisors that expect PID 1 to stay attached. Pass `--foreground` (alias `--no-daemon`) to keep the process in the foreground and stream logs to stderr instead of syslog: diff --git a/ci/smoke-test-cassandra-container.sh b/ci/smoke-test-cassandra-container.sh new file mode 100755 index 000000000..5247a9b83 --- /dev/null +++ b/ci/smoke-test-cassandra-container.sh @@ -0,0 +1,485 @@ +#!/usr/bin/env bash +# Copyright 2026 ExtendDB contributors +# SPDX-License-Identifier: Apache-2.0 +# +# Build and exercise the production Cassandra-backed ExtendDB image locally. +# Cassandra remains a separate service, as it is for a real deployment. All +# containers, credentials, volumes, and data are disposable. +# +# Set EXTENDDB_CASSANDRA_IMAGE to test a prebuilt image without rebuilding it; +# its version, revision and build date are then read from its OCI labels. + +# -E so the ERR trap below also fires inside functions. +set -Eeuo pipefail +exec < /dev/null + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +PROJECT_DIR="$(cd "$SCRIPT_DIR/.." && pwd)" +cd "$PROJECT_DIR" + +if docker compose version >/dev/null 2>&1; then + COMPOSE=(docker compose) +elif command -v docker-compose >/dev/null 2>&1; then + COMPOSE=(docker-compose) +else + echo "error: Docker Compose v2 (or docker-compose) is required" >&2 + exit 1 +fi + +for command in docker aws python3 timeout; do + if ! command -v "$command" >/dev/null 2>&1; then + echo "error: required command not found: $command" >&2 + exit 1 + fi +done + +PROJECT="extenddb-cassandra-smoke-$$-${RANDOM}" +COMPOSE_FILE=docker-compose.cassandra.yml + +if [[ -n ${EXTENDDB_CASSANDRA_IMAGE:-} ]]; then + PREBUILT_IMAGE=true + EXPECTED_IMAGE=$(docker image inspect "$EXTENDDB_CASSANDRA_IMAGE" --format '{{.Id}}') + EXTENDDB_VERSION=$(docker image inspect "$EXTENDDB_CASSANDRA_IMAGE" \ + --format '{{index .Config.Labels "org.opencontainers.image.version"}}') + VCS_REF=$(docker image inspect "$EXTENDDB_CASSANDRA_IMAGE" \ + --format '{{index .Config.Labels "org.opencontainers.image.revision"}}') + BUILD_DATE=$(docker image inspect "$EXTENDDB_CASSANDRA_IMAGE" \ + --format '{{index .Config.Labels "org.opencontainers.image.created"}}') + export EXTENDDB_VERSION VCS_REF BUILD_DATE + for value in "$EXTENDDB_VERSION" "$VCS_REF" "$BUILD_DATE"; do + if [[ -z "$value" || "$value" == "unknown" ]]; then + echo "error: prebuilt image has incomplete OCI identity metadata" >&2 + exit 1 + fi + done +else + PREBUILT_IMAGE=false + export EXTENDDB_CASSANDRA_IMAGE="extenddb-cassandra:smoke-$$-${RANDOM}" + export EXTENDDB_VERSION="${EXTENDDB_VERSION:-$(grep '^version' Cargo.toml | head -1 | sed 's/.*"\(.*\)".*/\1/')}" + export VCS_REF="${VCS_REF:-$(git rev-parse --short=12 HEAD)}" + export BUILD_DATE="${BUILD_DATE:-$(date -u +%Y-%m-%dT%H:%M:%SZ)}" +fi + +# Fresh random credentials for every run; the compose file refuses to start +# without them. Hex, which also fits init's character set for the +# application role's password. +random_secret() { + python3 -c 'import secrets; print(secrets.token_hex(16))' +} +export CASSANDRA_SUPERUSER_PASSWORD="${CASSANDRA_SUPERUSER_PASSWORD:-$(random_secret)}" +export EXTENDDB_CASSANDRA_APP_PASSWORD="${EXTENDDB_CASSANDRA_APP_PASSWORD:-$(random_secret)}" +export EXTENDDB_ADMIN_USER="${EXTENDDB_ADMIN_USER:-admin}" +export EXTENDDB_ADMIN_PASSWORD="${EXTENDDB_ADMIN_PASSWORD:-$(random_secret)}" +# Port 0 asks Docker to allocate an unused loopback port atomically. +export EXTENDDB_PORT="${EXTENDDB_PORT:-0}" + +TEMP_DIR="$(mktemp -d /tmp/extenddb-cassandra-smoke.XXXXXX)" +KEY_JSON="$TEMP_DIR/access-key.json" +KEY_ENV="$TEMP_DIR/access-key.env" +VERSION_OUTPUT="$TEMP_DIR/version.txt" +GET_JSON="$TEMP_DIR/get-item.json" +touch "$KEY_JSON" "$KEY_ENV" +chmod 600 "$KEY_JSON" "$KEY_ENV" + +cleanup() { + "${COMPOSE[@]}" -f "$COMPOSE_FILE" -p "$PROJECT" down -v --remove-orphans >/dev/null 2>&1 || true + if [[ "$PREBUILT_IMAGE" == "false" ]]; then + docker image rm "$EXTENDDB_CASSANDRA_IMAGE" >/dev/null 2>&1 || true + fi + if command -v shred >/dev/null 2>&1; then + shred -u "$KEY_JSON" "$KEY_ENV" 2>/dev/null || true + fi + rm -rf "$TEMP_DIR" +} +trap cleanup EXIT +# A failed bare assertion would otherwise exit 1 with nothing said and the +# stack already torn down. Name the line and keep the serving container's log. +on_error() { + echo "error: line $1: $2" >&2 + if [[ -n ${APP_CONTAINER:-} ]]; then + docker logs --tail 50 "$APP_CONTAINER" >&2 || true + fi +} +trap 'on_error "$LINENO" "$BASH_COMMAND"' ERR +APP_CONTAINER= + +compose() { + "${COMPOSE[@]}" -f "$COMPOSE_FILE" -p "$PROJECT" "$@" +} + +# The stack must refuse to start without each password, so a default added to +# the compose file later fails here. Checked by rendering the file with one +# variable empty at a time; nothing is started. +echo "=== Verifying the stack requires every password ===" +for variable in CASSANDRA_SUPERUSER_PASSWORD EXTENDDB_CASSANDRA_APP_PASSWORD EXTENDDB_ADMIN_PASSWORD; do + if output=$(env "$variable=" "${COMPOSE[@]}" -f "$COMPOSE_FILE" -p "$PROJECT" config 2>&1); then + echo "error: the compose file renders with $variable empty" >&2 + exit 1 + fi + if ! grep -qF "$variable" <<< "$output"; then + echo "error: compose config failed with $variable empty, but not because of it:" >&2 + echo "$output" >&2 + exit 1 + fi +done + +compose_pull_with_retry() { + local max_attempts=4 + local attempt + + for attempt in $(seq 1 "$max_attempts"); do + if compose pull "$@"; then + return 0 + fi + if (( attempt == max_attempts )); then + echo "error: Compose pull failed after $max_attempts attempts" >&2 + return 1 + fi + local delay=$((5 << (attempt - 1))) + echo "warning: Compose pull failed; retrying in ${delay}s (attempt $((attempt + 1))/$max_attempts)" >&2 + sleep "$delay" + done +} + +wait_for_health() { + local container_id="$1" + local status="" + for _ in $(seq 1 120); do + status=$(docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' "$container_id") + if [[ "$status" == "healthy" ]]; then + return 0 + fi + if [[ "$(docker inspect --format '{{.State.Running}}' "$container_id")" != "true" ]]; then + docker logs "$container_id" >&2 || true + return 1 + fi + sleep 1 + done + echo "error: container did not become healthy (last status: $status)" >&2 + docker logs "$container_id" >&2 || true + return 1 +} + +host_port() { + docker inspect --format '{{(index (index .NetworkSettings.Ports "18443/tcp") 0).HostPort}}' "$1" +} + +# cql USER PASSWORD STATEMENT: run one statement in the Cassandra container. +# The password reaches cqlsh through a 0600 credentials file written from +# stdin, never through a command line, which any user on the host can read. +cql() { + printf '[plain_text_auth]\nusername = %s\npassword = %s\n' "$1" "$2" \ + | docker exec -i "$CASSANDRA_CONTAINER" /bin/sh -ec ' + umask 077 + creds=$(mktemp) + trap '"'"'rm -f "$creds"'"'"' EXIT + cat > "$creds" + cqlsh --credentials "$creds" -e "$1" + ' sh "$3" +} + +# expect_bad_credentials USER PASSWORD: the login must be refused as such, not +# fail for some other reason. +expect_bad_credentials() { + local output + if output=$(cql "$1" "$2" 'SELECT release_version FROM system.local' 2>&1); then + echo "error: Cassandra accepted the login for $1 that it must refuse" >&2 + exit 1 + fi + if ! grep -qF "Bad credentials" <<< "$output"; then + echo "error: login for $1 failed for a reason other than bad credentials:" >&2 + echo "$output" >&2 + exit 1 + fi +} + +check_item() { + aws dynamodb get-item "${AWS_ARGS[@]}" \ + --table-name "$TABLE_NAME" \ + --key '{"pk":{"S":"hello"}}' --consistent-read > "$GET_JSON" + python3 - "$GET_JSON" <<'PY' +import json +import sys + +item = json.load(open(sys.argv[1])).get("Item", {}) +assert item.get("pk", {}).get("S") == "hello", item +assert item.get("msg", {}).get("S") == "world", item +PY +} + +# Compose waits on every dependency condition and has no overall deadline, so +# a bootstrap that never exits would hold the run until the job timeout. +compose_up() { + if ! timeout "${COMPOSE_UP_TIMEOUT:-600}" "${COMPOSE[@]}" -f "$COMPOSE_FILE" -p "$PROJECT" up -d "$@"; then + echo "error: compose up did not complete" >&2 + compose ps -a >&2 || true + for service in cassandra-auth-init extenddb-bootstrap; do + container_id=$(compose ps -a -q "$service" || true) + if [[ -n "$container_id" ]]; then + docker logs "$container_id" >&2 || true + fi + done + return 1 + fi +} + +set_endpoint() { + HOST_PORT=$(host_port "$APP_CONTAINER") + ENDPOINT="https://127.0.0.1:${HOST_PORT}" + AWS_ARGS=(--endpoint-url "$ENDPOINT" --region us-east-1 --no-verify-ssl) +} + +echo "=== Building and starting Cassandra container stack ===" +echo " project: $PROJECT" +echo " image: $EXTENDDB_CASSANDRA_IMAGE" +echo " version: $EXTENDDB_VERSION" +echo " commit: $VCS_REF" +compose_pull_with_retry cassandra +if [[ "$PREBUILT_IMAGE" == "true" ]]; then + echo " mode: prebuilt (build disabled)" + compose_up --no-build +else + echo " mode: build" + # The build is not under COMPOSE_UP_TIMEOUT: a cold release build of the + # image takes longer than any stack start should be allowed to. Built with + # `docker build` rather than `compose build`, which needs buildx 0.17 or + # later; the arguments are the compose file's. + docker build -f Dockerfile.cassandra -t "$EXTENDDB_CASSANDRA_IMAGE" \ + --build-arg "VERSION=$EXTENDDB_VERSION" \ + --build-arg "VCS_REF=$VCS_REF" \ + --build-arg "BUILD_DATE=$BUILD_DATE" . + compose_up --no-build + EXPECTED_IMAGE=$(docker image inspect "$EXTENDDB_CASSANDRA_IMAGE" --format '{{.Id}}') +fi + +APP_CONTAINER=$(compose ps -q extenddb) +INIT_CONTAINER=$(compose ps -a -q extenddb-volume-init) +BOOTSTRAP_CONTAINER=$(compose ps -a -q extenddb-bootstrap) +AUTH_INIT_CONTAINER=$(compose ps -a -q cassandra-auth-init) +CASSANDRA_CONTAINER=$(compose ps -q cassandra) +if [[ -z "$APP_CONTAINER" || -z "$INIT_CONTAINER" || -z "$BOOTSTRAP_CONTAINER" \ + || -z "$AUTH_INIT_CONTAINER" || -z "$CASSANDRA_CONTAINER" ]]; then + echo "error: Compose did not create the expected containers" >&2 + compose ps -a >&2 + exit 1 +fi + +for container_id in "$AUTH_INIT_CONTAINER" "$INIT_CONTAINER" "$BOOTSTRAP_CONTAINER"; do + exit_code=$(docker inspect --format '{{.State.ExitCode}}' "$container_id") + if [[ "$exit_code" != "0" ]]; then + docker logs "$container_id" >&2 || true + echo "error: $(docker inspect --format '{{.Name}}' "$container_id") exited with $exit_code" >&2 + exit 1 + fi +done +docker logs "$BOOTSTRAP_CONTAINER" 2>&1 \ + | grep -F "bootstrap: no config detected; initializing" >/dev/null + +wait_for_health "$APP_CONTAINER" +set_endpoint + +echo "=== Verifying Cassandra authentication and exposure ===" +# The stock superuser password must have been replaced, password +# authentication must be on, the application role init created must log in +# with the password from the environment, and Cassandra must not be published +# on the host. +cql cassandra "$CASSANDRA_SUPERUSER_PASSWORD" 'SELECT release_version FROM system.local' >/dev/null +expect_bad_credentials cassandra cassandra +expect_bad_credentials cassandra "wrong-$(random_secret)" +cql extenddb "$EXTENDDB_CASSANDRA_APP_PASSWORD" 'SELECT release_version FROM system.local' >/dev/null +# CassandraAuthorizer is on, so the application role holds only the grants init +# gave it: it may read the catalog but may not change another role. +cql extenddb "$EXTENDDB_CASSANDRA_APP_PASSWORD" 'SELECT key FROM extenddb_catalog.settings LIMIT 1' >/dev/null +if output=$(cql extenddb "$EXTENDDB_CASSANDRA_APP_PASSWORD" \ + "ALTER ROLE cassandra WITH PASSWORD = 'wrong-$(random_secret)'" 2>&1); then + echo "error: the application role changed the superuser's password" >&2 + exit 1 +fi +if ! grep -qF "Unauthorized" <<< "$output"; then + echo "error: ALTER ROLE as the application role failed for a reason other than authorization:" >&2 + echo "$output" >&2 + exit 1 +fi +[[ "$(docker inspect "$CASSANDRA_CONTAINER" --format '{{len .HostConfig.PortBindings}}')" == "0" ]] +KEYSPACES=$(cql cassandra "$CASSANDRA_SUPERUSER_PASSWORD" 'DESCRIBE KEYSPACES') +tr -s ' ' '\n' <<< "$KEYSPACES" | grep -qx extenddb_catalog + +echo "=== Verifying image identity and hardening ===" +for container_id in "$INIT_CONTAINER" "$BOOTSTRAP_CONTAINER" "$APP_CONTAINER"; do + [[ "$(docker inspect "$container_id" --format '{{.Image}}')" == "$EXPECTED_IMAGE" ]] +done +for container_id in "$BOOTSTRAP_CONTAINER" "$APP_CONTAINER"; do + [[ "$(docker inspect "$container_id" --format '{{.Config.User}}')" == "10001:10001" ]] + [[ "$(docker inspect "$container_id" --format '{{.HostConfig.ReadonlyRootfs}}')" == "true" ]] + [[ "$(docker inspect "$container_id" --format '{{json .HostConfig.CapDrop}}')" == '["ALL"]' ]] + [[ "$(docker inspect "$container_id" --format '{{json .HostConfig.SecurityOpt}}')" == '["no-new-privileges:true"]' ]] +done +[[ "$(docker inspect "$INIT_CONTAINER" --format '{{.Config.User}}')" == "0:0" ]] +[[ "$(docker inspect "$INIT_CONTAINER" --format '{{json .HostConfig.CapAdd}}')" == '["CAP_CHOWN"]' ]] +[[ "$(docker inspect "$APP_CONTAINER" --format '{{(index (index .NetworkSettings.Ports "18443/tcp") 0).HostIp}}')" == "127.0.0.1" ]] + +# The image carries no Cassandra server, client or JVM. (A line starting +# with `!` never stops `sh -e`, so each absence is an explicit exit.) +docker exec "$APP_CONTAINER" /bin/sh -ec ' + for tool in cassandra cqlsh java; do + if command -v "$tool" >/dev/null 2>&1; then + echo "error: the image contains $tool" >&2 + exit 1 + fi + done + test "$(stat -c %u:%g /var/lib/extenddb)" = 10001:10001 + test "$(stat -c %a /var/lib/extenddb/extenddb.toml)" = 600 + test "$(stat -c %a /var/lib/extenddb/.extenddb/tls/key.pem)" = 600 + test -s /usr/share/doc/extenddb/SOFTWARE-LICENSE-NOTICES-CASSANDRA.html +' + +# The serving config holds the application role's password from the +# environment and never the superuser's. Compared in Python with the values +# taken from the environment, so neither reaches a command line. +docker exec "$APP_CONTAINER" cat /var/lib/extenddb/extenddb.toml | python3 -c ' +import os +import sys + +config = sys.stdin.read() +if os.environ["EXTENDDB_CASSANDRA_APP_PASSWORD"] not in config: + sys.exit("error: extenddb.toml does not hold EXTENDDB_CASSANDRA_APP_PASSWORD") +if os.environ["CASSANDRA_SUPERUSER_PASSWORD"] in config: + sys.exit("error: extenddb.toml holds the Cassandra superuser password") +' + +docker run --rm --user 0:0 --entrypoint /bin/sh "$EXTENDDB_CASSANDRA_IMAGE" \ + -ec 'test -z "$(find / -xdev -type f -perm /6000 -print -quit)"' + +docker run --rm --read-only "$EXTENDDB_CASSANDRA_IMAGE" --version \ + | tee "$VERSION_OUTPUT" +grep -Fx "extenddb $EXTENDDB_VERSION" "$VERSION_OUTPUT" >/dev/null +grep -F "commit $VCS_REF" "$VERSION_OUTPUT" >/dev/null +grep -F "built $BUILD_DATE" "$VERSION_OUTPUT" >/dev/null + +echo "=== Provisioning API credentials ===" +export EXTENDDB_PASSWORD="$EXTENDDB_ADMIN_PASSWORD" +docker exec -e EXTENDDB_PASSWORD "$APP_CONTAINER" \ + extenddb manage --user "$EXTENDDB_ADMIN_USER" \ + --config /var/lib/extenddb/extenddb.toml \ + create-account --account-id 123456789012 --account-name cassandra-container-smoke >/dev/null +docker exec -e EXTENDDB_PASSWORD "$APP_CONTAINER" \ + extenddb manage --user "$EXTENDDB_ADMIN_USER" \ + --config /var/lib/extenddb/extenddb.toml \ + create-user --account-id 123456789012 --user-name tester >/dev/null +docker exec -e EXTENDDB_PASSWORD "$APP_CONTAINER" \ + extenddb manage --user "$EXTENDDB_ADMIN_USER" \ + --config /var/lib/extenddb/extenddb.toml \ + put-user-policy --account-id 123456789012 --user-name tester \ + --policy-name dynamodb-full \ + --policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":"dynamodb:*","Resource":"*"}]}' +docker exec -e EXTENDDB_PASSWORD "$APP_CONTAINER" \ + extenddb manage --user "$EXTENDDB_ADMIN_USER" \ + --config /var/lib/extenddb/extenddb.toml \ + create-access-key --account-id 123456789012 --user-name tester > "$KEY_JSON" +unset EXTENDDB_PASSWORD + +python3 - "$KEY_JSON" "$KEY_ENV" <<'PY' +import json +import shlex +import sys + +data = json.load(open(sys.argv[1])) +access_key = data.get("access_key_id") +secret_key = data.get("secret_access_key") +if not access_key or not secret_key: + raise SystemExit(f"unexpected create-access-key fields: {sorted(data)}") +with open(sys.argv[2], "w") as output: + output.write("export AWS_ACCESS_KEY_ID=" + shlex.quote(access_key) + "\n") + output.write("export AWS_SECRET_ACCESS_KEY=" + shlex.quote(secret_key) + "\n") +print(" temporary access key created: " + access_key[:8] + "...") +PY +# shellcheck disable=SC1090 +source "$KEY_ENV" +unset AWS_SESSION_TOKEN AWS_SECURITY_TOKEN +export AWS_DEFAULT_REGION=us-east-1 +export AWS_REGION=us-east-1 +export AWS_EC2_METADATA_DISABLED=true +export AWS_PAGER="" + +TABLE_NAME="cassandra-container-smoke-$(date +%s)" + +aws dynamodb create-table "${AWS_ARGS[@]}" \ + --table-name "$TABLE_NAME" \ + --attribute-definitions AttributeName=pk,AttributeType=S \ + --key-schema AttributeName=pk,KeyType=HASH \ + --billing-mode PAY_PER_REQUEST >/dev/null + +TABLE_STATUS="" +for _ in $(seq 1 120); do + TABLE_STATUS=$(aws dynamodb describe-table "${AWS_ARGS[@]}" \ + --table-name "$TABLE_NAME" --query 'Table.TableStatus' \ + --output text 2>/dev/null || true) + [[ "$TABLE_STATUS" == "ACTIVE" ]] && break + sleep 0.5 +done +[[ "$TABLE_STATUS" == "ACTIVE" ]] + +aws dynamodb put-item "${AWS_ARGS[@]}" \ + --table-name "$TABLE_NAME" \ + --item '{"pk":{"S":"hello"},"msg":{"S":"world"}}' >/dev/null +check_item + +echo "=== Restarting serving container and checking persistence ===" +compose restart extenddb >/dev/null +APP_CONTAINER=$(compose ps -q extenddb) +wait_for_health "$APP_CONTAINER" +set_endpoint +check_item + +echo "=== Verifying graceful SIGTERM shutdown ===" +compose stop -t 20 extenddb >/dev/null +APP_CONTAINER=$(compose ps -a -q extenddb) +[[ "$(docker inspect "$APP_CONTAINER" --format '{{.State.Running}}')" == "false" ]] +[[ "$(docker inspect "$APP_CONTAINER" --format '{{.State.ExitCode}}')" == "0" ]] + +echo "=== Recreating stack and exercising the migrate preview ===" +# Volumes survive `down` without -v, so this restarts Cassandra on its data, +# re-runs cassandra-auth-init against the replaced password, and takes the +# bootstrap's existing-config branch: the migrate preview, as the application +# role, finds the catalog current and exits 0. (Nothing is pending at this +# commit, so `migrate --yes` is not reached.) +compose down --remove-orphans >/dev/null +compose_up --no-build +INIT_CONTAINER=$(compose ps -a -q extenddb-volume-init) +BOOTSTRAP_CONTAINER=$(compose ps -a -q extenddb-bootstrap) +AUTH_INIT_CONTAINER=$(compose ps -a -q cassandra-auth-init) +APP_CONTAINER=$(compose ps -q extenddb) +CASSANDRA_CONTAINER=$(compose ps -q cassandra) +for container_id in "$AUTH_INIT_CONTAINER" "$INIT_CONTAINER" "$BOOTSTRAP_CONTAINER"; do + exit_code=$(docker inspect --format '{{.State.ExitCode}}' "$container_id") + if [[ "$exit_code" != "0" ]]; then + docker logs "$container_id" >&2 || true + echo "error: $(docker inspect --format '{{.Name}}' "$container_id") exited with $exit_code on restart" >&2 + exit 1 + fi +done +docker logs "$AUTH_INIT_CONTAINER" 2>&1 \ + | grep -F "cassandra-auth-init: superuser login works" >/dev/null +docker logs "$BOOTSTRAP_CONTAINER" 2>&1 \ + | grep -F "bootstrap: existing config detected; running migrations" >/dev/null +# The preview reached the catalog and read it, rather than exiting 0 for +# another reason. +docker logs "$BOOTSTRAP_CONTAINER" 2>&1 \ + | grep -F "Up to date: the catalog is at migration" >/dev/null +# The replaced superuser password survived the recreate. +expect_bad_credentials cassandra cassandra +cql cassandra "$CASSANDRA_SUPERUSER_PASSWORD" 'SELECT release_version FROM system.local' >/dev/null +for container_id in "$INIT_CONTAINER" "$BOOTSTRAP_CONTAINER" "$APP_CONTAINER"; do + [[ "$(docker inspect "$container_id" --format '{{.Image}}')" == "$EXPECTED_IMAGE" ]] +done +wait_for_health "$APP_CONTAINER" +set_endpoint +check_item + +echo "=== Cassandra container smoke test passed ===" +echo " health: healthy" +echo " Cassandra: password authentication and CassandraAuthorizer, stock superuser password replaced, not published" +echo " API: CreateTable / PutItem / GetItem" +echo " restart persistence: passed" +echo " graceful SIGTERM: passed" +echo " recreate and migrate preview: passed" diff --git a/ci/smoke-test-container.sh b/ci/smoke-test-container.sh index 32f2fb8b8..3322729ad 100755 --- a/ci/smoke-test-container.sh +++ b/ci/smoke-test-container.sh @@ -189,7 +189,12 @@ done [[ "$(docker inspect "$APP_CONTAINER" --format '{{(index (index .NetworkSettings.Ports "18443/tcp") 0).HostIp}}')" == "127.0.0.1" ]] docker exec "$APP_CONTAINER" /bin/sh -ec ' - ! command -v postgres >/dev/null 2>&1 + for tool in postgres; do + if command -v "$tool" >/dev/null 2>&1; then + echo "error: the image contains $tool" >&2 + exit 1 + fi + done test "$(stat -c %u:%g /var/lib/extenddb)" = 10001:10001 test "$(stat -c %a /var/lib/extenddb/extenddb.toml)" = 600 test "$(stat -c %a /var/lib/extenddb/.extenddb/tls/key.pem)" = 600 diff --git a/ci/smoke-test-mongodb-container.sh b/ci/smoke-test-mongodb-container.sh index 7ba9d8679..2e805202d 100755 --- a/ci/smoke-test-mongodb-container.sh +++ b/ci/smoke-test-mongodb-container.sh @@ -144,8 +144,12 @@ for container_id in "$BOOTSTRAP_CONTAINER" "$APP_CONTAINER"; do done docker exec "$APP_CONTAINER" /bin/sh -ec ' - ! command -v mongod >/dev/null 2>&1 - ! command -v mongosh >/dev/null 2>&1 + for tool in mongod mongosh; do + if command -v "$tool" >/dev/null 2>&1; then + echo "error: the image contains $tool" >&2 + exit 1 + fi + done test "$(stat -c %u:%g /var/lib/extenddb)" = 10001:10001 test "$(stat -c %a /var/lib/extenddb/extenddb.toml)" = 600 test "$(stat -c %a /var/lib/extenddb/.extenddb/tls/key.pem)" = 600 diff --git a/docker-compose.cassandra.yml b/docker-compose.cassandra.yml new file mode 100644 index 000000000..1ce90a03c --- /dev/null +++ b/docker-compose.cassandra.yml @@ -0,0 +1,261 @@ +# Copyright 2026 ExtendDB contributors +# SPDX-License-Identifier: Apache-2.0 +# +# Local development only. This stack runs a single Cassandra node with +# replication factor 1 and must not be used as a production deployment. The +# Cassandra service is separate from ExtendDB so the same image can be used +# with a Cassandra cluster you operate. +# +# Cassandra publishes no port on the host's interfaces; it is reachable from +# this stack's containers (and, on Linux, from the host through the bridge +# network). ExtendDB is published on 127.0.0.1 only. All passwords come from the +# environment and the stack refuses to start without them: +# +# CASSANDRA_SUPERUSER_PASSWORD replaces the stock cassandra/cassandra +# superuser password on first start +# EXTENDDB_CASSANDRA_APP_PASSWORD password of the application role init +# creates and serve logs in as +# EXTENDDB_ADMIN_PASSWORD first ExtendDB administrator + +name: extenddb-cassandra-dev + +x-extenddb-image: &extenddb-image + image: ${EXTENDDB_CASSANDRA_IMAGE:-extenddb-cassandra:dev} + +x-cassandra-image: &cassandra-image + image: ${CASSANDRA_IMAGE:-public.ecr.aws/docker/library/cassandra:4.1@sha256:b877532a79865d0212c860352bb9c3b0be97db9628a662ea815fb90ea4033158} + +x-extenddb-hardening: &extenddb-hardening + read_only: true + tmpfs: + - /tmp:rw,noexec,nosuid,size=16m + cap_drop: + - ALL + security_opt: + - no-new-privileges:true + +services: + cassandra: + <<: *cassandra-image + # The cassandra:4.1 image has no environment variable for the + # authenticator or the authorizer. `extenddb init` needs + # PasswordAuthenticator because it creates its application role WITH + # PASSWORD, and under CassandraAuthorizer it grants that role only the + # permissions ExtendDB needs. Enable both in cassandra.yaml, then hand over + # to the image's own entrypoint. + entrypoint: ["/bin/sh", "-ec"] + command: + - | + sed -i -e 's/^authenticator: .*/authenticator: PasswordAuthenticator/' \ + -e 's/^authorizer: .*/authorizer: CassandraAuthorizer/' \ + /etc/cassandra/cassandra.yaml + # sed exits 0 whether or not a line matched. A cassandra.yaml laid out + # differently (another CASSANDRA_IMAGE) would otherwise start with + # AllowAllAuthorizer and no error. + grep -qx 'authenticator: PasswordAuthenticator' /etc/cassandra/cassandra.yaml \ + || { echo "cassandra: cassandra.yaml has no top-level authenticator line to replace" >&2; exit 1; } + grep -qx 'authorizer: CassandraAuthorizer' /etc/cassandra/cassandra.yaml \ + || { echo "cassandra: cassandra.yaml has no top-level authorizer line to replace" >&2; exit 1; } + exec docker-entrypoint.sh cassandra -f + environment: + # No CASSANDRA_DC: the image then keeps SimpleSnitch, whose datacenter + # is datacenter1, the name init writes into the keyspace definitions. + CASSANDRA_CLUSTER_NAME: extenddb-dev + MAX_HEAP_SIZE: ${CASSANDRA_MAX_HEAP_SIZE:-1024M} + HEAP_NEWSIZE: ${CASSANDRA_HEAP_NEWSIZE:-128M} + healthcheck: + # Native transport up; needs no credentials. cassandra-auth-init waits + # for the authenticated login separately. + test: ["CMD-SHELL", "nodetool statusbinary | grep -qx running"] + interval: 5s + timeout: 10s + retries: 60 + start_period: 30s + volumes: + - cassandra-data:/var/lib/cassandra + # ExtendDB restarts on its own after a reboot and cannot start without + # Cassandra. + restart: unless-stopped + + cassandra-auth-init: + <<: *cassandra-image + depends_on: + cassandra: + condition: service_healthy + environment: + CASSANDRA_SUPERUSER_PASSWORD: ${CASSANDRA_SUPERUSER_PASSWORD:?set CASSANDRA_SUPERUSER_PASSWORD} + # The credentials files below hold the password; keep them off the + # container's writable layer. The image declares VOLUME /var/lib/cassandra; + # a tmpfs there stops each run leaving an empty anonymous volume behind. + tmpfs: + - /tmp:rw,noexec,nosuid,size=1m + - /var/lib/cassandra:rw,noexec,nosuid,size=1m + # Waits until the superuser can log in, and replaces the stock password + # the first time. On later starts the new password already works and + # nothing changes. cqlsh reads every password from a 0600 credentials + # file in this container, never from its command line, which any user on + # the host can read. + entrypoint: ["/bin/sh", "-ec"] + command: + - | + fail() { echo "cassandra-auth-init: $$*" >&2; exit 1; } + [ "$$CASSANDRA_SUPERUSER_PASSWORD" != cassandra ] \ + || fail "CASSANDRA_SUPERUSER_PASSWORD must not be the stock password" + umask 077 + creds() { + printf '[plain_text_auth]\nusername = cassandra\npassword = %s\n' "$$1" > "$$2" + } + creds "$$CASSANDRA_SUPERUSER_PASSWORD" /tmp/new.credentials + # cqlsh parses the file with configparser, which strips surrounding + # whitespace (including \r and non-breaking spaces) and splits on line + # breaks. Refuse a password that would not read back exactly, before + # the stock password is replaced with it. + python3 -c ' + import configparser, os, sys + parser = configparser.RawConfigParser() + try: + parser.read(sys.argv[1], encoding="utf-8") + ok = parser.get("plain_text_auth", "password") == os.environ["CASSANDRA_SUPERUSER_PASSWORD"] + except (configparser.Error, UnicodeDecodeError): + ok = False + sys.exit(0 if ok else 1) + ' /tmp/new.credentials \ + || fail "CASSANDRA_SUPERUSER_PASSWORD must not contain line breaks or start or end with whitespace" + creds cassandra /tmp/stock.credentials + creds "wrong-$$(cat /proc/sys/kernel/random/uuid)" /tmp/wrong.credentials + printf "ALTER ROLE cassandra WITH PASSWORD = '%s';\n" \ + "$$(printf '%s' "$$CASSANDRA_SUPERUSER_PASSWORD" | sed "s/'/''/g")" > /tmp/alter-role.cql + login() { + cqlsh --credentials "$$1" cassandra 9042 \ + -e 'SELECT release_version FROM system.local' 2>&1 + } + # Bounded by the clock, not by passes: each pass can spend two cqlsh + # connect timeouts. + deadline=$$(( $$(date +%s) + 300 )) + rejected_since= + while [ "$$(date +%s)" -lt "$$deadline" ]; do + if out=$$(login /tmp/new.credentials); then + # Fail closed if the authenticator edit did not take: an + # AllowAllAuthenticator node accepts any password. + if wrong=$$(login /tmp/wrong.credentials); then + fail "Cassandra accepted a random password; password authentication is not enabled" + fi + case "$$wrong" in + *"Bad credentials"*) ;; + *) fail "unexpected answer to a wrong password: $$wrong" ;; + esac + echo "cassandra-auth-init: superuser login works" + exit 0 + fi + if stock=$$(login /tmp/stock.credentials); then + # The node caches credentials for a few seconds, so the stock + # login can keep working briefly after the change and the new one + # can fail. A failed ALTER is retried on the next pass. + echo "cassandra-auth-init: replacing the stock superuser password" + cqlsh --credentials /tmp/stock.credentials cassandra 9042 \ + -f /tmp/alter-role.cql || true + rejected_since= + else + case "$$out" in + *"Bad credentials"*) + case "$$stock" in + *"Bad credentials"*) : "$${rejected_since:=$$(date +%s)}" ;; + *) rejected_since= ;; + esac ;; + *) rejected_since= ;; + esac + # Both passwords refused for a minute, well past the node's + # 10-second superuser setup delay: the volume was set up with + # another password. + [ -z "$$rejected_since" ] \ + || [ $$(( $$(date +%s) - rejected_since )) -lt 60 ] \ + || fail "Cassandra refuses CASSANDRA_SUPERUSER_PASSWORD and the stock password. Use the password this stack was first started with, or run 'docker compose -f docker-compose.cassandra.yml down -v' to remove both volumes and start over." + fi + sleep 2 + done + fail "Cassandra did not accept a superuser login within 5 minutes" + restart: "no" + + extenddb-volume-init: + <<: *extenddb-image + build: + context: . + dockerfile: Dockerfile.cassandra + args: + VERSION: ${EXTENDDB_VERSION:-dev} + VCS_REF: ${VCS_REF:-unknown} + BUILD_DATE: ${BUILD_DATE:-unknown} + user: "0:0" + read_only: true + cap_drop: + - ALL + cap_add: + - CHOWN + security_opt: + - no-new-privileges:true + healthcheck: + disable: true + entrypoint: ["/bin/sh", "-ec"] + command: ["chown -R 10001:10001 /var/lib/extenddb"] + volumes: + - extenddb-state:/var/lib/extenddb + restart: "no" + + extenddb-bootstrap: + <<: [*extenddb-image, *extenddb-hardening] + depends_on: + cassandra-auth-init: + condition: service_completed_successfully + extenddb-volume-init: + condition: service_completed_successfully + environment: + # init reads all four. migrate connects as the application role in the + # config and needs none of them. Passed through the environment so they + # stay out of process arguments. + EXTENDDB_CASSANDRA_PASSWORD: ${CASSANDRA_SUPERUSER_PASSWORD:?set CASSANDRA_SUPERUSER_PASSWORD} + EXTENDDB_APP_PASSWORD: ${EXTENDDB_CASSANDRA_APP_PASSWORD:?set EXTENDDB_CASSANDRA_APP_PASSWORD} + EXTENDDB_ADMIN_USER: ${EXTENDDB_ADMIN_USER:-admin} + EXTENDDB_ADMIN_PASSWORD: ${EXTENDDB_ADMIN_PASSWORD:?set EXTENDDB_ADMIN_PASSWORD} + healthcheck: + disable: true + entrypoint: ["/bin/sh", "-ec"] + command: + - | + CONFIG=/var/lib/extenddb/extenddb.toml + if [ -f "$$CONFIG" ]; then + echo "bootstrap: existing config detected; running migrations" + # The preview changes nothing and exits 0 when the catalog is up to + # date, 3 when migrations are pending. Only then run --yes, which + # waits for a running server, so an `up` on a running, current stack + # does not fail. + status=0 + extenddb migrate --config "$$CONFIG" || status=$$? + case "$$status" in + 0) exit 0 ;; + 3) exec extenddb migrate --config "$$CONFIG" --yes ;; + *) exit "$$status" ;; + esac + fi + echo "bootstrap: no config detected; initializing" + exec extenddb init --backend cassandra --config "$$CONFIG" \ + --cassandra-contact-points cassandra:9042 --cassandra-user cassandra \ + --bind-addr 0.0.0.0 --tls-san localhost --tls-san extenddb + volumes: + - extenddb-state:/var/lib/extenddb + restart: "no" + + extenddb: + <<: [*extenddb-image, *extenddb-hardening] + depends_on: + extenddb-bootstrap: + condition: service_completed_successfully + ports: + - "127.0.0.1:${EXTENDDB_PORT:-18443}:18443" + volumes: + - extenddb-state:/var/lib/extenddb + stop_grace_period: 15s + restart: unless-stopped + +volumes: + cassandra-data: + extenddb-state: diff --git a/docker/README-cassandra.md b/docker/README-cassandra.md new file mode 100644 index 000000000..59845e131 --- /dev/null +++ b/docker/README-cassandra.md @@ -0,0 +1,265 @@ +# ExtendDB Cassandra Container + +This directory documents the registry-neutral ExtendDB Cassandra container +image. The image contains the ExtendDB executable compiled with the +`cassandra` backend on a Debian bookworm-slim base. It does **not** contain +Cassandra, `cqlsh` or a Java runtime. + +> **Experimental.** The Cassandra backend is experimental. Read the known +> limitations in [`docs/cassandra-setup.md`](../docs/cassandra-setup.md) and +> validate it against your workload before you rely on it. + +Provide Apache Cassandra 4.1 or newer separately. `docs/cassandra-setup.md` +covers the cluster requirements (password authentication, the `datacenter1` +datacenter name, replication factor) and the backend's known limitations. + +## Image contents and security defaults + +The image: + +* runs as UID/GID `10001:10001`; +* supports a read-only root filesystem; +* uses `tini` to forward SIGTERM; +* publishes only port 18443; +* stores configuration and TLS state under `/var/lib/extenddb`; +* uses `extenddb healthcheck` for liveness; +* contains no Cassandra server or client. + +The healthcheck verifies that ExtendDB is alive. It does not continuously +verify Cassandra availability after startup; use the orchestrator's readiness +checks and Cassandra monitoring for that. `serve` exits if it cannot reach +Cassandra when it starts. + +ExtendDB connects to Cassandra over plain TCP: the driver is built without +TLS, so credentials and item data cross the network in clear text. Run the +ExtendDB container and the Cassandra cluster on a private network, and do not +route Cassandra traffic across an untrusted one. + +## Build locally + +Generate the notices for the Cassandra feature set, then build the image: + +```bash +devtools/generate-software-license-notices --backend cassandra +docker build \ + --file Dockerfile.cassandra \ + --build-arg VERSION="$(grep '^version' Cargo.toml | head -1 | sed 's/.*"\(.*\)".*/\1/')" \ + --build-arg VCS_REF="$(git rev-parse --short=12 HEAD)" \ + --build-arg BUILD_DATE="$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ + -t extenddb-cassandra:dev . +``` + +## Connect to an existing Cassandra cluster + +The cluster must run `PasswordAuthenticator`, and its local datacenter must be +named `datacenter1`; `docs/cassandra-setup.md` explains why. The contact point +must be resolvable and reachable from the ExtendDB container. + +**Run `CassandraAuthorizer`.** Under it, `init` grants the application role +only what ExtendDB needs: SELECT, MODIFY, CREATE and ALTER on the catalog +keyspace, those four and DROP on the default account's keyspace, and CREATE ON +ALL KEYSPACES, which CreateAccount needs and which also lets the role create +tables in any keyspace on the cluster. Under `AllowAllAuthorizer`, Cassandra's +default, `init` grants nothing, because every login role, including the +application role, can then read and drop any keyspace and change any role's +password, the superuser's included; use such a cluster for ExtendDB alone. +Anyone who can read `extenddb.toml` holds the application role's password. +A deployment created before 1.0 under `CassandraAuthorizer` needs the GRANT +statements in `docs/manuals/07-upgrade-manual.md` before it is upgraded. + +Run `extenddb init` once against a writable state volume. It connects as a +Cassandra superuser, creates the catalog keyspace, the default account's +keyspace and an application role (`extenddb` by default), and writes +`/var/lib/extenddb/extenddb.toml` with the application role's credentials. +Pass the passwords through the environment rather than as arguments: + +* `EXTENDDB_CASSANDRA_PASSWORD`: the superuser password `init` connects with + (and `migrate` when it is given `--cassandra-user`). +* `EXTENDDB_APP_PASSWORD`: the password `init` gives the application role when + it creates it. Always set it; without it the role gets the fixed password + `extenddb-local-dev`. It may contain only ASCII letters, digits, space and + ``-_.,!@#$%^&*()+=~` ``. If the role already exists, `init` does not change + or check its password and writes the value you supplied into the config, so + a wrong value surfaces only when `serve` fails to connect. +* `EXTENDDB_ADMIN_USER` and `EXTENDDB_ADMIN_PASSWORD`: the first ExtendDB + administrator. Set both; with only the password set, `init` prints that + password in its output, and so in the container's logs. + +`init` writes only the first of the `--cassandra-contact-points` into +`extenddb.toml`; add the others to `contact_points` there afterwards. The file +is in the volume and the image has no editor, so edit it with `sed` from the +image itself, which runs as the file's owner (UID 10001) and keeps its 0600 +mode: + +```bash +docker run --rm -v extenddb-state:/v --entrypoint sed \ + extenddb/extenddb-cassandra:X.Y.Z -i \ + 's/^contact_points = .*/contact_points = ["cassandra-1.example.internal:9042", "cassandra-2.example.internal:9042", "cassandra-3.example.internal:9042"]/' \ + /v/extenddb.toml +``` + +The examples use a named volume, `extenddb-state`. Docker creates it owned by +the image's `extenddb` user (UID 10001), which runs both commands. To use a +host directory instead, make it owned by UID 10001 first +(`sudo chown 10001:10001 `); `init` otherwise fails with +`Failed to create TLS directory ... Permission denied`. + +```bash +docker run --rm \ + -v extenddb-state:/var/lib/extenddb \ + -e EXTENDDB_CASSANDRA_PASSWORD \ + -e EXTENDDB_APP_PASSWORD \ + -e EXTENDDB_ADMIN_USER=admin \ + -e EXTENDDB_ADMIN_PASSWORD \ + extenddb/extenddb-cassandra:X.Y.Z \ + init --backend cassandra \ + --config /var/lib/extenddb/extenddb.toml \ + --cassandra-contact-points cassandra.example.internal:9042 \ + --cassandra-user \ + --bind-addr 0.0.0.0 \ + --tls-san localhost --tls-san 127.0.0.1 + +docker run -d --name extenddb-cassandra \ + -p 127.0.0.1:18443:18443 \ + -v extenddb-state:/var/lib/extenddb \ + --read-only --tmpfs /tmp:rw,noexec,nosuid,size=16m \ + --cap-drop ALL --security-opt no-new-privileges:true \ + extenddb/extenddb-cassandra:X.Y.Z +``` + +Clients need the self-signed certificate `init` created. Copy it out of the +volume with: + +```bash +docker run --rm -v extenddb-state:/v:ro --entrypoint cat \ + extenddb/extenddb-cassandra:X.Y.Z /v/.extenddb/tls/cert.pem > extenddb-cert.pem +``` + +### Replication factor + +`init` creates its keyspaces with replication factor 1 and has no flag to +change that. On a multi-node cluster, follow "Keyspace replication" in +[`docs/cassandra-setup.md`](../docs/cassandra-setup.md#keyspace-replication): +set `replication_factor` in `extenddb.toml` and restart the serving +container, raise the replication of `system_auth` and of the keyspaces `init` +created, then repair. The config edit, with the volume and image from above: + +```bash +docker run --rm -v extenddb-state:/v --entrypoint sed \ + extenddb/extenddb-cassandra:X.Y.Z -i \ + 's/^replication_factor = .*/replication_factor = 3/' /v/extenddb.toml +docker restart extenddb-cassandra +``` + +### Production + +Run one serving ExtendDB instance per catalog. Every container binds +`0.0.0.0:18443`, from which ExtendDB derives its stream node id, so two +replicas would share one; see "Multi-Instance Considerations" in +[`docs/manuals/11-deployment-guide.md`](../docs/manuals/11-deployment-guide.md). + +Inject credentials through the orchestrator's secret store, pin a version or +digest, and run `init` and `migrate` as separate one-shot containers or Jobs. +The serving container needs only the generated `extenddb.toml` and TLS files. +Do not give it the superuser password. Under `AllowAllAuthorizer` this keeps +the password out of the serving container but does not limit what the +application role can do (see above). + +### Upgrade + +Stop the serving container, then run one migration container with the new +image. `migrate` connects as the application role in `extenddb.toml`. +`migrate --yes` waits up to `--lock-timeout` seconds (60 by default) for every +running server to stop, then exits 6 naming them, and a server does not start +while a migration runs: + +```bash +extenddb migrate --config /var/lib/extenddb/extenddb.toml --yes +``` + +Without `--yes` it lists what is pending and changes nothing; it exits 0 when +the catalog is up to date and 3 when migrations are pending. Pass +`--cassandra-user ` with `EXTENDDB_CASSANDRA_PASSWORD` to run it as +another role. + +## Local reference stack + +For users who want Cassandra and ExtendDB launched together locally, the +repository provides an optional reference stack, `docker-compose.cassandra.yml`. + +This stack is for local development only. It runs one Cassandra node with +replication factor 1 and a 1 GB heap (about 1.6 GB of memory in use), so give +Docker at least 3 GB. Cassandra publishes no port on the host's interfaces; +ExtendDB is published on `127.0.0.1` only. The stack enables +`PasswordAuthenticator` and `CassandraAuthorizer`, replaces the stock +`cassandra` superuser password on first start, and refuses to start until you +set all three passwords: + +```bash +export CASSANDRA_SUPERUSER_PASSWORD='choose-a-password' +export EXTENDDB_CASSANDRA_APP_PASSWORD='choose-another-password' +export EXTENDDB_ADMIN_PASSWORD='choose-a-third-password' +export EXTENDDB_VERSION="$(grep '^version' Cargo.toml | head -1 | sed 's/.*"\(.*\)".*/\1/')" +export VCS_REF="$(git rev-parse --short=12 HEAD)" +export BUILD_DATE="$(date -u +%Y-%m-%dT%H:%M:%SZ)" +docker compose -f docker-compose.cassandra.yml up -d --build +``` + +`EXTENDDB_CASSANDRA_APP_PASSWORD` has the character limits of +`EXTENDDB_APP_PASSWORD` above. `CASSANDRA_SUPERUSER_PASSWORD` must not be +`cassandra`, contain a line break, or start or end with whitespace. + +To run a published image instead of building one, set +`EXTENDDB_CASSANDRA_IMAGE=extenddb/extenddb-cassandra:X.Y.Z` and use +`up -d --no-build`. Other settings you can override: `EXTENDDB_PORT` (default +18443), `EXTENDDB_ADMIN_USER` (default `admin`), `CASSANDRA_MAX_HEAP_SIZE`, +`CASSANDRA_HEAP_NEWSIZE` and `CASSANDRA_IMAGE`. + +Each `up` runs `extenddb init` if the state volume has no config yet. +Otherwise it runs the `extenddb migrate` preview and, only when migrations are +pending, `extenddb migrate --yes`. That waits for a running server, so to move +the stack to a new image that has migrations, run +`docker compose -f docker-compose.cassandra.yml stop extenddb` first; an `up` +with ExtendDB still running fails after 60 s and changes nothing. + +Keep the same `CASSANDRA_SUPERUSER_PASSWORD` across starts: the stock password +is replaced only once, and later starts log in with yours. Cassandra takes +about a minute to accept logins on first start. If a one-shot service fails, +`docker compose ... logs cassandra-auth-init extenddb-bootstrap` shows why. +Compose resolves the three variables for every command run with `-f`, `down`, +`ps` and `logs` included. Rather than exporting them in every shell, put them +in an untracked `.env` file next to the compose file (`.gitignore` already +lists it), with mode 0600: + +``` +CASSANDRA_SUPERUSER_PASSWORD=... +EXTENDDB_CASSANDRA_APP_PASSWORD=... +EXTENDDB_ADMIN_PASSWORD=... +``` + +`down -v` removes both volumes and starts the stack over. + +## Automated smoke test + +The smoke test builds the image, starts the reference stack with random +passwords, verifies Cassandra authentication and that Cassandra is not +published, checks image identity and hardening, performs a DynamoDB API round +trip, and verifies persistence after restarting ExtendDB and after recreating +the stack, where the bootstrap runs the migrate preview against the existing +catalog: + +```bash +ci/smoke-test-cassandra-container.sh +``` + +The `Cassandra image smoke` workflow runs the same script, in build mode, on +every pull request that changes the Dockerfile, the compose file or the +script, on an amd64 and an arm64 runner. `release-image` runs it again against +the image it is about to publish. + +To test an already-built image without rebuilding it: + +```bash +EXTENDDB_CASSANDRA_IMAGE=extenddb-cassandra:dev \ + ci/smoke-test-cassandra-container.sh +``` diff --git a/docker/dockerhub-overview-cassandra.md b/docker/dockerhub-overview-cassandra.md new file mode 100644 index 000000000..1908c5698 --- /dev/null +++ b/docker/dockerhub-overview-cassandra.md @@ -0,0 +1,126 @@ +# ExtendDB with Cassandra + +ExtendDB is a DynamoDB-compatible API server backed by your Apache Cassandra +cluster. This image contains ExtendDB only; Cassandra is a separate service +that you operate. + +**The Cassandra backend is experimental.** Validate it against your workload +before you rely on it; the setup guide below lists its known limitations. + +## Quick start + +The Cassandra backend requires Apache Cassandra 4.1 or newer with +`PasswordAuthenticator` enabled and a local datacenter named `datacenter1`. +Read the +[Cassandra setup guide](https://github.com/ExtendDB/extenddb/blob/main/docs/cassandra-setup.md) +first: it explains these requirements and the backend's known limitations. +Run `CassandraAuthorizer`, under which `init` grants the application role only +the permissions ExtendDB needs. Under `AllowAllAuthorizer`, the default, that +role can read, drop and alter everything in the cluster, so use such a cluster +for ExtendDB alone. + +Pull the image: + +```console +docker pull docker.io/extenddb/extenddb-cassandra:latest +``` + +Initialize the state volume against your cluster. `init` connects as a +Cassandra superuser, creates its keyspaces and an application role, and +writes the configuration. The contact point must be resolvable and reachable +from the container. Pass the passwords through the environment. Always set +`EXTENDDB_APP_PASSWORD` (ASCII letters, digits, space and +``-_.,!@#$%^&*()+=~` `` only): without it the application role gets a fixed, +publicly known password. Docker creates the named volume `extenddb-state` +owned by the image's unprivileged user; a host directory would have to be +owned by UID 10001 first. + +```console +export EXTENDDB_CASSANDRA_PASSWORD='' +export EXTENDDB_APP_PASSWORD='' +export EXTENDDB_ADMIN_PASSWORD='' + +docker run --rm \ + -v extenddb-state:/var/lib/extenddb \ + -e EXTENDDB_CASSANDRA_PASSWORD -e EXTENDDB_APP_PASSWORD \ + -e EXTENDDB_ADMIN_USER=admin -e EXTENDDB_ADMIN_PASSWORD \ + docker.io/extenddb/extenddb-cassandra:latest \ + init --backend cassandra \ + --config /var/lib/extenddb/extenddb.toml \ + --cassandra-contact-points cassandra.example.com:9042 \ + --cassandra-user \ + --bind-addr 0.0.0.0 \ + --tls-san localhost \ + --tls-san 127.0.0.1 +``` + +`init` creates its keyspaces with replication factor 1. On a multi-node +cluster, raise it afterwards as "Keyspace replication" in the setup guide +describes. + +Start the server using the generated configuration: + +```console +docker run --rm --name extenddb-cassandra \ + -p 127.0.0.1:18443:18443 \ + -v extenddb-state:/var/lib/extenddb \ + --read-only \ + --tmpfs /tmp:rw,noexec,nosuid,size=16m \ + --cap-drop=ALL \ + --security-opt=no-new-privileges:true \ + docker.io/extenddb/extenddb-cassandra:latest +``` + +The server listens on `https://127.0.0.1:18443`. Initialization creates a +self-signed certificate; clients need +`/var/lib/extenddb/.extenddb/tls/cert.pem` from the volume. Run one serving +container per catalog. + +## Configuration and security + +The ExtendDB container runs as an unprivileged user and is designed to run +with a read-only root filesystem. The serving container needs only the +generated configuration and TLS files, not the Cassandra superuser password. + +ExtendDB connects to Cassandra over plain TCP; the driver is built without +TLS. Run ExtendDB and Cassandra on a private network. + +For a complete deployment guide, upgrades, and an optional local reference +stack, see +[`docker/README-cassandra.md`](https://github.com/ExtendDB/extenddb/blob/main/docker/README-cassandra.md). + +## Tags and verification + +Pin `X.Y.Z` or a digest for production; a version tag is never overwritten. +`latest` tracks the highest release and only moves forward. Tags of the form +`sha-` are unpromoted build candidates, not releases. Both +`linux/amd64` and `linux/arm64` ship as one multi-architecture index. + +Every published image is signed with ExtendDB's release key, an elliptic-curve +P-256 key held in the AWS Key Management Service. The public key is committed +in the repository as +[`extenddb-signing.pub.pem`](https://github.com/ExtendDB/extenddb/blob/main/extenddb-signing.pub.pem). +Verify an image before deploying it: + +```console +cosign verify --key extenddb-signing.pub.pem \ + docker.io/extenddb/extenddb-cassandra:X.Y.Z +``` + +The same images and their signatures are mirrored by digest to +`ghcr.io/extenddb/extenddb-cassandra` and +`public.ecr.aws/extenddb/extenddb-cassandra`. + +## Note + +ExtendDB is an independent open source project managed by Amazon Web +Services. It is not Amazon DynamoDB and does not contain any DynamoDB source +code. "DynamoDB" is a trademark of Amazon.com, Inc. ExtendDB is a clean-room +implementation that speaks the DynamoDB wire protocol; behavioral differences +from the service are documented in +[Differences from DynamoDB](https://github.com/ExtendDB/extenddb/blob/main/docs/differences-from-dynamodb.md). + +More at [extenddb.org](https://extenddb.org) and +[github.com/ExtendDB/extenddb](https://github.com/ExtendDB/extenddb). Licensed +under Apache-2.0. Apache Cassandra, Cassandra and Apache are trademarks of +The Apache Software Foundation. diff --git a/docs/cassandra-setup.md b/docs/cassandra-setup.md index 3ad0074bf..6b08f4ce7 100644 --- a/docs/cassandra-setup.md +++ b/docs/cassandra-setup.md @@ -21,11 +21,16 @@ These steps start a single-node Cassandra 4.1 container for local development. 1. Start the container: ```bash - docker run -d --name cassandra -p 9042:9042 \ + docker run -d --name cassandra -p 127.0.0.1:9042:9042 \ -e CASSANDRA_CLUSTER_NAME=extenddb-dev \ cassandra:4.1 ``` + The port is published on the loopback interface only. Until step 3 the + node accepts any login, and after it the superuser is on Apache + Cassandra's documented default password; neither should be reachable from + other hosts. + 2. Wait for the node to accept CQL. The port opens before the database is ready, so poll with `cqlsh` until it succeeds: @@ -91,6 +96,17 @@ cargo build --release --no-default-features --features cassandra The binary is single-backend. To run Cassandra and another backend side by side, build one binary per backend. +## Container Image + +Starting with ExtendDB 1.0, the `extenddb/extenddb-cassandra` image is +published to Docker Hub, GHCR and ECR Public. It contains this build of +ExtendDB and no Cassandra server, client or Java runtime; Cassandra stays a +separate service. [`docker/README-cassandra.md`](../docker/README-cassandra.md) covers +running `init`, `serve` and `migrate` with the image, and a local reference +stack, `docker-compose.cassandra.yml`, that starts a password-authenticated +Cassandra node next to ExtendDB. The cluster requirements below apply to the +image unchanged. + ## Initializing ExtendDB with Cassandra Initialize against the authenticated local node from above. Pass the contact @@ -98,14 +114,16 @@ point and the superuser credentials used to create the keyspaces and the application role: ```bash -./target/release/extenddb init \ +EXTENDDB_CASSANDRA_PASSWORD=cassandra ./target/release/extenddb init \ --backend cassandra \ --cassandra-contact-points 127.0.0.1:9042 \ --cassandra-user cassandra \ - --cassandra-pass cassandra \ --config extenddb.toml ``` +`init` reads the superuser password from `EXTENDDB_CASSANDRA_PASSWORD` when +`--cassandra-pass` is not given, which keeps it out of the process list. + The node must run `PasswordAuthenticator` (step 3). `init` always creates its application role with a password, so it fails against an unauthenticated (`AllowAllAuthenticator`) node — there is no credential-free path. @@ -171,12 +189,9 @@ Fields: time. Required against any authenticated node (which `init` requires anyway). - `keyspace_prefix` — prefix for this deployment's keyspaces (default `extenddb`). -- `replication_factor` — keyspace replication factor. `init` always writes `1`, - regardless of the cluster. For a multi-node cluster, edit this to 3 or more - **after** `init`; keyspaces created later (per account) use the config value, - except an account keyspace that `migrate --yes` creates for an account whose - CreateAccount stopped before its keyspace: that one takes the catalog - keyspace's replication. Existing keyspaces are not altered retroactively. +- `replication_factor` — keyspace replication factor. `init` always writes `1` + and creates its own keyspaces with it; see [Keyspace replication](#keyspace-replication) + for a multi-node cluster. - `datacenter` — the local datacenter name for `NetworkTopologyStrategy`. Currently, `init` always writes `datacenter1` and has no flag to change it, and the keyspaces it creates reference `datacenter1`, so the cluster's local datacenter must be @@ -247,12 +262,40 @@ section covers only what is specific to Cassandra. ### Keyspace replication -Set `replication_factor` to 3 or more for a multi-node cluster. `init` always -writes `1`, so edit the config after `init`; keyspaces created afterward (per -account) use the value in the config, except an account keyspace that -`migrate --yes` creates for an account whose CreateAccount stopped before its -keyspace, which takes the catalog keyspace's replication. The `datacenter` must -be `datacenter1`: +Set `replication_factor` to 3 or more for a multi-node cluster. `init` has no +flag for it: it creates the catalog keyspace and the default account's +keyspace with replication factor 1 and writes `replication_factor = 1` to the +config. Keyspaces created later (one per new account) take the value from the +config when `serve` starts, except an account keyspace that `migrate --yes` +creates for an account whose CreateAccount stopped before its keyspace, which +takes the catalog keyspace's replication. On a multi-node cluster, after +`init`: + +1. Set `replication_factor = 3` in `extenddb.toml` (example below), before you + start `serve` or, if it is already running, restart it before you create + accounts. +2. Raise the replication of the keyspaces that already exist: Cassandra's own + `system_auth` keyspace, which holds the roles ExtendDB logs in with and + starts at `SimpleStrategy` with replication factor 1, and every keyspace + whose name starts with the keyspace prefix (`DESCRIBE KEYSPACES` lists + them): + + ```sql + ALTER KEYSPACE system_auth + WITH replication = {'class': 'NetworkTopologyStrategy', 'datacenter1': 3}; + ALTER KEYSPACE extenddb_catalog + WITH replication = {'class': 'NetworkTopologyStrategy', 'datacenter1': 3}; + -- and the same for each extenddb_account_ keyspace + ``` + +3. Run `nodetool repair -full` on each node. + +Use a replication factor no larger than the number of nodes in `datacenter1`. +Logins read `system_auth` at `QUORUM` for the default superuser, so a +`system_auth` replication factor above the node count makes those logins fail +with `Cannot achieve consistency level QUORUM`. + +The `datacenter` must be `datacenter1`: `init` hardcodes that name into the keyspace definitions and provides no flag to change it, so your cluster's local datacenter has to be named `datacenter1` until that is configurable. ExtendDB creates its keyspaces with