From 85dde63c5c9e848af98401656742adff5cbe20af Mon Sep 17 00:00:00 2001 From: Louis Haftmann <30736553+LouisHaftmann@users.noreply.github.com> Date: Sun, 13 Sep 2026 15:21:11 +0200 Subject: [PATCH] docs(cache-server): EAGER_MERGE and server-side merge notes per storage driver Refs falcondev-oss/github-actions-cache-server#267 Created with AI. Verified by a human. Co-Authored-By: Claude Fable 5.1 --- packages/gha-cache-server/getting-started.md | 14 +++++++++++++ .../storage-drivers/file-system.md | 4 ++++ .../storage-drivers/google-cloud-storage.md | 16 +++++++------- .../gha-cache-server/storage-drivers/s3.md | 21 +++++++++++++++++-- 4 files changed, 46 insertions(+), 9 deletions(-) diff --git a/packages/gha-cache-server/getting-started.md b/packages/gha-cache-server/getting-started.md index b62a7d0..5f7288c 100644 --- a/packages/gha-cache-server/getting-started.md +++ b/packages/gha-cache-server/getting-started.md @@ -113,6 +113,20 @@ The actions runner needs to be able to reach the storage provider directly to us ::: +#### `EAGER_MERGE` + +- Default: `false` + +Cache data arrives in parts. By default the server merges them into one object on the first download, so with `ENABLE_DIRECT_DOWNLOADS` the first restore of every entry is still proxied through the server. With `EAGER_MERGE=true` the merge starts right after the upload completes, and the first restore can be a direct download. + +On S3 and GCS the merge happens inside the bucket, without the data passing through the server. On S3 this requires every part but the last to be at least 5 MiB (the default upload chunk size of `actions/cache` is 32 MiB; buildx uploads 1 MiB blocks and is merged by streaming instead). On the filesystem driver the server streams the parts into the merged file at upload time. + +::: warning + +Every entry occupies twice its size until the parts cleanup job removes the parts, including entries that are never restored. On S3, set an `AbortIncompleteMultipartUpload` lifecycle rule on the bucket so a server killed mid-merge does not leave billed multipart uploads behind. + +::: + #### `DEFAULT_ACTIONS_RESULTS_URL` - Default: `https://results-receiver.actions.githubusercontent.com` diff --git a/packages/gha-cache-server/storage-drivers/file-system.md b/packages/gha-cache-server/storage-drivers/file-system.md index e922833..8558620 100644 --- a/packages/gha-cache-server/storage-drivers/file-system.md +++ b/packages/gha-cache-server/storage-drivers/file-system.md @@ -18,3 +18,7 @@ Driver: `filesystem` The path to the filesystem storage location. The folder will be created if it does not exist. The cache server process needs read and write access to this path (and permission to create it). No other permissions are required for this driver. + +## Eager merge + +The filesystem driver has no direct downloads, so [`EAGER_MERGE`](/getting-started#eager-merge) gains little here. It streams the parts into the merged file at upload time and doubles disk use per entry until the parts cleanup job runs. Leave it off unless you want the first restore to read a single file. diff --git a/packages/gha-cache-server/storage-drivers/google-cloud-storage.md b/packages/gha-cache-server/storage-drivers/google-cloud-storage.md index 78c9c03..321a4c3 100644 --- a/packages/gha-cache-server/storage-drivers/google-cloud-storage.md +++ b/packages/gha-cache-server/storage-drivers/google-cloud-storage.md @@ -64,14 +64,16 @@ The API endpoint for GCS. All cache objects live under the `gh-actions-cache/` prefix in the bucket. The service account needs these IAM permissions on the bucket: -| Permission | Used for | -| ----------------------- | ------------------------------------------- | -| `storage.buckets.get` | Startup bucket check | -| `storage.objects.get` | Downloading cache entries, existence checks | -| `storage.objects.list` | Listing/counting cache objects | -| `storage.objects.create`| Uploading cache entries | -| `storage.objects.delete`| Cache cleanup and abandoned uploads | +| Permission | Used for | +| ------------------------ | ------------------------------------------- | +| `storage.buckets.get` | Startup bucket check | +| `storage.objects.get` | Downloading cache entries, existence checks | +| `storage.objects.list` | Listing/counting cache objects | +| `storage.objects.create` | Uploading cache entries, merging parts | +| `storage.objects.delete` | Cache cleanup and abandoned uploads | The predefined [`roles/storage.objectAdmin`](https://cloud.google.com/storage/docs/access-control/iam-roles) role covers all the object permissions but **not** `storage.buckets.get`. Either grant `roles/storage.objectAdmin` together with `roles/storage.legacyBucketReader`, or create a custom role with the five permissions above. If you enable direct downloads (`ENABLE_DIRECT_DOWNLOADS`) without a `STORAGE_GCS_SERVICE_ACCOUNT_KEY` (i.e. using Application Default Credentials), signing download URLs additionally requires the `iam.serviceAccounts.signBlob` permission (granted by `roles/iam.serviceAccountTokenCreator`). This is not needed when a service account key file is provided, since URLs are then signed locally. + +With [`EAGER_MERGE`](/getting-started#eager-merge) the server merges parts inside the bucket using `compose`, covered by the permissions above. The merged object is a composite object; it downloads like any other object. diff --git a/packages/gha-cache-server/storage-drivers/s3.md b/packages/gha-cache-server/storage-drivers/s3.md index 5ea06e7..8d8c5f6 100644 --- a/packages/gha-cache-server/storage-drivers/s3.md +++ b/packages/gha-cache-server/storage-drivers/s3.md @@ -143,8 +143,8 @@ The cache server only ever touches objects under the `gh-actions-cache/` prefix | ------------------------- | ---------------------------------------------------- | | `s3:ListBucket` | Startup bucket check, listing/counting cache objects | | `s3:GetObject` | Downloading cache entries, existence checks | -| `s3:PutObject` | Uploading cache entries (multipart) | -| `s3:AbortMultipartUpload` | Cleaning up failed multipart uploads | +| `s3:PutObject` | Uploading cache entries (multipart), merging parts | +| `s3:AbortMultipartUpload` | Cleaning up failed multipart uploads and merges | | `s3:DeleteObject` | Cache cleanup and abandoned uploads | A minimal policy, scoped to the bucket and prefix (replace `YOUR_BUCKET`): @@ -169,6 +169,23 @@ A minimal policy, scoped to the bucket and prefix (replace `YOUR_BUCKET`): } ``` +## Eager merge + +With [`EAGER_MERGE`](/getting-started#eager-merge) the server merges parts inside the bucket using `UploadPartCopy`, covered by the permissions above. A server killed mid-merge leaves an incomplete multipart upload that S3 keeps and bills. Add a lifecycle rule that aborts them: + +```json +{ + "Rules": [ + { + "ID": "abort-incomplete-merges", + "Status": "Enabled", + "Filter": { "Prefix": "gh-actions-cache/" }, + "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 1 } + } + ] +} +``` + ## Troubleshooting ### Checksum errors with S3-compatible storage (Garage, MinIO)