Repository navigation
Prepare Record Store 0.2.0 - #106
Merged
Merged
Conversation
Authorization read ?retention and ?legal-hold before looking at the method, while the POST, DELETE and GET handlers act on ?uploads and ?uploadId first. A request could therefore be judged as one operation and performed as another: POST ?uploadId=U&retention completed someone's multipart upload with only s3:PutObjectRetention, DELETE ?uploadId=U&legal-hold aborted one with only s3:PutObjectLegalHold, and neither checked PutObject or DeleteObject, so an explicit Deny on those did not apply. The action is now decided per method in the order the handlers dispatch: GET lists parts before reading a lock document, PUT honours the lock subresources before writing, DELETE aborts an upload before deleting a version, and POST always writes. With only the lock permissions, none of those requests can complete, abort, list or start an upload.
…onour A copy returned before the check that refuses unsupported write headers, and never read the Object Lock headers. So a copy requesting a COMPLIANCE retention or a legal hold was authorized for it, answered 200, and wrote a version with only the bucket default -- a client believed in a lock that did not exist. Server-side encryption, ACL, tagging and website-redirect headers were accepted and dropped the same way, as they already were in 0.1.3. The copy now carries the requested lock to the new version, as PutObject does, and refuses with 501 NotImplemented, writing nothing, what a PUT refuses plus x-amz-tagging-directive and every x-amz-copy-source-* header: a conditional copy whose precondition was ignored could overwrite the object it was meant to protect. CMP-UNSUPPORTED checks the copy cases against the real binary.
A truncated page returned NextUploadIdMarker but not NextKeyMarker, and key-marker was ignored: a client that followed the S3 specification and sent the key marker looped over the first page. boto3's paginator resumes from the upload id alone, which is why the gate only recorded this (RSG-009). A truncated page now names both markers, taken from its last upload. The listing resumes after the (key-marker, upload-id-marker) pair, or after every upload of key-marker when it comes alone; an upload id alone still resumes as before. A marker upload completed or aborted between pages restarts at its key -- a client may see that key's remaining uploads twice, never miss one -- rather than failing the listing. The resume point is also clamped to the prefix: a marker before it used to widen the listing to keys the prefix excludes. COR-PAGINATION now walks the uploads by hand from the markers, with four uploads on one key so a page boundary falls inside a key, and asserts both markers and the key-marker-alone case. Against the previous binaries it fails on each of those. The finding is closed.
axum::serve has no header read timeout, so a client that opened a connection and trickled one header line a second was kept for as long as it liked; 48 such connections were all still open after the whole overload window (RSG-004). Each held a descriptor and a task, and nothing bounded how many one client could open. Both listeners now run an accept loop on hyper-util that serves the router the same way -- HTTP/1.1 and h2c, the caller's address as ConnectInfo, a graceful drain -- with two deadlines: hyper's header read timeout on every HTTP/1 request head, which also closes a kept-alive connection idle that long, and a deadline for a connection's first request, which covers the bytes hyper reads before it knows the protocol and HTTP/2. server.header_read_timeout_seconds (RECORD_STORE_HEADER_READ_TIMEOUT_SECONDS) sets both, default 30 s, range 1-3600. A request body or a response is not limited, so a slow transfer of a large object is unaffected. SEC-SLOW-CLIENTS replaces the advisory PERF-OVERLOAD metric with a blocking contract: against the real binary, 32 trickling and 16 silent connections per listener, and an idle keep-alive, are all closed within the timeout plus 2 s while a well-behaved client is served throughout, no descriptor outlives the attack, and the default is 30 s. Against the previous binaries all 96 are still held when the gate gives up. The finding is closed.
Every redb database was opened with redb's default page cache: 1 GiB, filled lazily as pages are read and written. The audit trail and the storage-event journal grow with every request, so resident memory climbed with the database files until each cache was full (RSG-011). Measured on Linux under the endurance workload in a container limited to 384 MiB, it grew about 1.2 KB per request, reached 374 MiB in 14 minutes and was OOM-killed. The catalog, audit trail and event journal now share storage.metadata_cache_mib (RECORD_STORE_STORAGE_METADATA_CACHE_MIB, default 128, range 8-1048576): half for the catalog, a quarter each for the others. The credential, sharing and lifecycle databases, and tools that open a database briefly, get 16 MiB. Under the same limit and workload the bounded server ran for over an hour and a million requests, levelling off near 230 MiB. What still rose after the caches filled was glibc's allocator, not live data: a heap profile over twelve minutes peaked at 66 MB of allocations, mostly the two bounded caches, with 0.4 MB unreleased at exit, while RSS was near 180 MB. glibc hands contending threads arenas of their own, up to eight per core, and tokio's blocking pool runs dozens. With MALLOC_ARENA_MAX=2 the same run held at half the memory (99 MiB at 30 minutes) at the same throughput, so the image sets it. The workspace forbids unsafe code, so it is set where the process is defined rather than through mallopt; musl builds are unaffected by it. RES-MEMORY makes the property blocking, judged per request so runner speed does not decide it: with an 8 MiB cache, after a 120 s warm-up, resident memory may grow by at most 256 bytes per request, and peak RSS stays under 256 MiB. The previous binaries, which ignore the setting, grow 1189 bytes per request and fail. docs/operations/capacity-planning.md gains a Memory section with the measurements. The finding is closed.
Each catalog change was its own durable transaction through redb's single writer, so concurrent writers waited for each other's commit and fsync in turn (RSG-005). Measured on Linux with the server limited to four CPUs and fresh 4 KiB keys, PUT throughput was 31, 97 and 212 per second at 1, 4 and 16 clients and fell to 173 at 64, where p99 reached 2-2.6 s. One writer thread now commits whatever has queued, applying it in arrival order inside one transaction -- the order the separate transactions would have run in -- and answers each caller only once the commit is durable. A command that fails, such as a refused precondition, abandons the batch, and every command in it is then applied in a transaction of its own in the same order, so a refusal is decided against exactly the state it would have met alone and never fails or partly lands its neighbours. Commands are deterministic and Clone, which is what makes the replay exact. The last repository handle joins the thread, so the database closes with the repository, as the audit trail's writer already does. Under the same measurement, alternating builds twice: 34, 99, 278 and 463 PUT/s at 1, 4, 16 and 64 clients, p99 at 64 down to 0.45-0.62 s. A test checks that 400 concurrent writes share commits, that 40 refusals among them stay alone, and that every acknowledged write survives a reopen; transactions_committed() reports the ratio. The finding is closed.
Nothing in a binary named the commit it came from, and unreleased builds report the previous release's version, so a deployment, a backup or a bug report could not say which code produced it; the gates identified a candidate only by the digests recorded when it was built (RSG-006). A build script embeds RECORD_STORE_BUILD_COMMIT when the build environment provides it -- the image and release builds do, since their build context has no .git -- and otherwise the checkout's HEAD, rebuilding when HEAD moves; with neither it says unknown rather than guessing. The commit appears in `record-store version` (a second line, or `--json version`), in GET /api/v1/system/info, in the start-up log line and as record_store_commit in a backup manifest. `record-store --version` keeps its one-line form, which release.yml and the packages compare with the tag. ART-IDENTITY now requires the CLI and the running server to name the candidate's commit, which also makes a release fail if an image is ever built without it. The Dockerfile's builder moves from rust:1.98 to the 1.97.1 image rust-toolchain.toml pins: the file is in the build context, so rustup downloaded 1.97.1 on every uncached build and the image digest did not pin the compiler. The finding is closed.
Four gaps in the new backup and restore, found in review: - A restore killed after moving metadata and system records into place but before the payloads left a directory that start-up refuses, but that backup copied into a complete-looking backup with no objects. backup now refuses a data directory holding an unfinished restore (exit 2). - The wrong credential master key was refused only when payloads were encrypted. The key, or the root secret where none is set, also seals service-account secrets, share links and webhook secrets, so a plaintext deployment restored under another key started and then could not authenticate its own accounts. A backup now records a domain-separated, one-way reference to that material, and restore refuses a mismatch before writing anything, in both modes; the refusal names no key. - The manifest recorded the binary's catalog schema rather than the catalog's, so a pre-upgrade backup of a 0.1.3 directory said schema 6. It is now read from the copied catalog without migrating it (schema 4 in that case), falling back to the binary's for a catalog that needs repair. verify also refuses a backup whose storage format is newer than this release reads, as it did for the backup format and schema, instead of restoring it into a directory that then fails start-up. - A configuration that did not load exited 1 with no JSON from check-config, doctor, backup and restore, while the documented code is 2. It is now 2, with a JSON error under --json; 1 is documented as an unexpected failure. REC-BACKUP now asserts the wrong-key refusal with encryption off as well as on, and that a half-restored directory is refused as a backup source.
Since publication records are written under a temporary name and renamed into place, a write, sync or rename that failed returned without removing the .publish.partial file. A multipart completion retries under the object id it reserved, so after a transient failure such as a full disk every retry was refused as already existing until the next restart cleaned the file up. The partial file is now guarded from its creation until the rename, and removed on any failure; it is never touched when another writer created it.
- A management API that answered 503 -- starting, or a check it depends on failing -- was shown as "Record Store is unreachable". It now has its own "not ready" screen that points at the readiness endpoint and the logs. - An upload refused for quota (507) was reported as "outcome unknown: the server may have stored this file", because every 5xx is treated as uncertain. A quota refusal is decided inside the commit that would have stored the object, so when the body is the API's own error envelope it is now reported as failed, with the API's message. The same status from an intermediary stays uncertain. - Opening Metrics from Overview never showed the server's history: Overview's cached reading reached the window first, and history was only ever seeded into an empty one, because server-clock and browser-clock samples cannot be mixed. The history endpoint now also returns the server's clock (`now`); the console moves the server's readings onto the browser's clock by the difference and places them beneath its own. A server without `now` is handled as before.
Fixes found reviewing the chart, the packages and the release workflow before their first release: - Every `helm upgrade` to a new chart version would have been refused: the volumeClaimTemplates carried the chart and app version labels, and a StatefulSet's claim templates are immutable. They now carry only the selector labels, and chart-check.sh renders two chart versions and fails if the claim templates differ. - The chart used the volume root as the data directory. local-path and many NFS provisioners create it world-writable, which the server refuses to store data in, so the pod would crash-loop -- invisible to CI, whose smoke test deployed the `latest` image, 0.1.3, which predates that check. The data directory is now data/ inside the volume, created by the server, and CI builds both images from the commit and loads them into kind, so the chart is proven against the server it ships with. - The NetworkPolicy admitted 0.0.0.0/0 to the management port for kubelet probes, which CNIs that match ipBlock against pod addresses read as every pod. The rule is gone; mainstream CNIs admit node-local probes without one, and apiFrom takes node addresses where a CNI needs them. With the console disabled and no apiFrom there is no management rule at all, rather than an empty `from` that would admit every source, and chart-check.sh asserts it. - The chart is published only after the release decision, so a blocked release leaves no chart pointing at it. - A pre-release tag would have failed the package jobs, which looked for 0.2.0-rc.1 in file names that nfpm spells 0.2.0~rc.1. - The documented package check-config always failed on missing root credentials, which live in the environment file only systemd reads. The command now sources it, as the release workflow's install test does. - The upgrade example used --reuse-values, which drops defaults a new chart adds, and the extraEnv example named a variable nothing reads.
…lease Start-up now refuses three states 0.1.3 accepted -- a world-writable data directory, a temporary directory on another filesystem, an unfinished restore -- so an upgrade could stop at the start of the new version with the old one already stopped and backed up. The guide adds `doctor` with the new image against the stopped deployment's volumes before the start, and CMP-UPGRADE-CONTAINER runs it as written against the published 0.1.3 image's data. The guide also says that 0.1.2 and earlier must pass through 0.1.3, which converts the databases to the file format this release reads, and that the first start migrates the schema from 4 to 6, after which 0.1.3 cannot open the directory.
The compatibility reference describes CopyObject's lock headers and refused headers and ListMultipartUploads' markers; CMP-UNSUPPORTED's guarantee names the copy cases it checks. The release README lists the negative controls demonstrated against the previous binaries for this matrix version -- COR-PAGINATION's multipart markers, SEC-SLOW-CLIENTS and RES-MEMORY -- and the findings table says no finding is open.
A minor release: Object Lock, coordinated backup and restore, a tamper-evident audit trail, proof bundles, Debian and RPM packages and a Helm chart, and stricter S3 request handling. It changes the on-disk format (schema 4 to 6, redb 4), cannot be rolled back by swapping the binary, and refuses some requests and deployments 0.1.3 accepted, which is why it is a minor bump. The workspace, the console, both lockfiles, the chart's version and appVersion, the Compose defaults and every current-version example move to 0.2.0. References that are history stay: which releases have no provenance, how 0.1.3 refuses upgraded data, the release that migrates a v2 database, and 0.1.3 as the release CMP-UPGRADE upgrades from. The attestation note now names 0.2.0, the first release built with it, instead of a 0.1.4 that will not exist. CHANGELOG.md's 0.2.0 section opens with what an upgrade requires and what it breaks, corrects the schema step to 4 to 6, and adds what was missing: backup and restore, doctor and the start-up checks, admission control, the crash-safe audit and event guarantees, the listener timeout, the bounded cache, the build commit, and the fixes of this release. The redb 4 notes that had been added to the already-published 0.1.3 section -- one of them calling 0.1.3 a rollback target for data it cannot open -- are moved here and 0.1.3's section is restored to what it published.
While a Find was shown the upload control was withdrawn, since results span the bucket and are not a folder, but the list still accepted dropped files. They were uploaded to the bucket root, with the overwrite check made against the find results rather than the root's own listing, so on an unversioned bucket a root object could be replaced without the confirmation that exists for exactly that. The drop target now follows the same rule as the control.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release preparation for 0.2.0. It closes every open release finding (RSG-004, -005, -006, -009, -011) and fixes defects found while reviewing this release's changes. Each commit is self-contained and builds on its own.
Candidate evaluation: commit
3cff577(this branch's head) at matrix2026.09.25-1is READY SO FAR, with 25/25 runnable gates passing: https://github.com/danielmarv/record-store/actions/runs/36176693912. The run was on a fork, on branchcandidate/0.2.0. ART-PACKAGE is deferred; it only runs inrelease.ymlafter the tag.Findings closed
storage.metadata_cache_mib(default 128) bounds the cache; the image setsMALLOC_ARENA_MAX=2(glibc arena fragmentation)axum::serveoffers no header timeoutserver.header_read_timeout_seconds(default 30)NextKeyMarkermissing,key-markerignoredbuild.rsembeds it; shown inversion,system/info, logs and backup manifestsOther defects fixed
POST ?uploadId&retention/DELETE ?uploadId&legal-holdcompleted or aborted uploads with only the lock permissions. Authorization now follows the handlers' dispatch order..publish.partialfile blocked retries until restart.helm upgradewould have been refused (versioned labels on the immutable volumeClaimTemplates).data/inside the volume).latest.check-configcommand could not work.Release preparation
release-version.sh v0.2.0passes.After merge
git tag -s v0.2.0 -m "Record Store v0.2.0". Ifmainmoves before the merge, re-run the candidate on the merge commit first.release.ymlruns ART-PACKAGE (provenance, SBOMs, arm64/amd64 smoke tests, .deb/.rpm install, chart publish) and makes the final decision.Not re-verified in this branch's final form: the 90-minute memory soaks (still running when local testing stopped; flat at about 90 MiB at 17 min). The stale
candidate/0.2.0branch on this repository (b8f2b19) is superseded and can be deleted.