ci: Derive EC2 artifacts from ECS build by akash1810 · Pull Request #16821 · guardian/dotcom-rendering

akash1810 · 2026-09-30T12:38:09Z

Note

It might be easiest to review commit by commit.

What does this change?

This change updates the build such that we call webpack once to produce assets. This webpack call is performed within Production.dockerfile (the ECS build). The assets are reused for the EC2 infrastructure¹. This change also means the static assets produced in the ECS build are deployed to the CDN, resolving a regression - see also #16808.

As a result of these changes, the GitHub hosted Docker image is no longer used and is therefore removed to simplify the build.

How has this change been tested?

Artifact comparison

Due to the inclusion of hashes in the filenames, the artifacts won't exactly match, so can't declare this a no-op change. However, we can compare the files uploaded to Riff-Raff for this branch to the main branch. The volume of files are the same - 696 total files.

Deploy to CODE

Note

In between these steps, I've purged content in Fastly and also deleted files in S3 via:

aws s3 rm s3://aws-frontend-static/CODE/frontend-static/assets/ --recursive --profile frontend

Before

In CODE, tag-page-rendering is being served entirely by ECS. Therefore, if we visit https://m.code.dev-theguardian.com/tone/minutebyminute when main is deployed, we see requests to load JS from assets-code.guim.co.uk/assets/ fail (reproducing the scenario described in #16808).

Failing to load assets on main (tag-page-rendering from ECS)

If we then deploy a change to switch tag-page-rendering CODE to be served by EC2 (#16792) and make the same request, assets are successfully loaded:

Success loading assets on main (tag-page-rendering from EC2)

After

If we then deploy this branch to CODE and visited https://m.code.dev-theguardian.com/tone/minutebyminute, assets successfully load from assets-code.guim.co.uk/assets/:

Success loading assets on this branch (tag-page-rendering from ECS)

Closes #16180.
Closes #16808.
Closes #16455.

This reuse for EC2 isn't new. See also https://github.com/guardian/dotcom-rendering/pull/16821#discussion_r4146596114. ↩

github-actions · 2026-09-30T12:39:29Z

🚀 Image pushed to AWS ECR

Image digest: sha256:9193922ab7be429e9749be7daa32dc51c5f20527f2e63e24e8b0cf38ef1f794a

🐛 Run the image locally

The following can be used to run the image locally:

# Refer to image using the immutable digest. Find alternatives below.
IMAGE_IDENTIFIER="@sha256:9193922ab7be429e9749be7daa32dc51c5f20527f2e63e24e8b0cf38ef1f794a"

# Refer to image using branch tag
# IMAGE_IDENTIFIER=":branch-aa-remove-github-container"

# Refer to image using build tag
# IMAGE_IDENTIFIER=":build-31884"

# Set environment variables for the AWS CLI
AWS_PROFILE="<A_PROFILE_FROM_JANUS>"
AWS_DEFAULT_REGION="eu-west-1"

IMAGE_ACCOUNT_ID=$(aws ssm get-parameter --name /organisation/accounts/artifacts --query "Parameter.Value" --output text)
REGISTRY="${IMAGE_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com"
IMAGE="${REGISTRY}/guardian/dotcom-rendering${IMAGE_IDENTIFIER}"

# Login to AWS ECR https://docs.aws.amazon.com/AmazonECR/latest/userguide/registry_auth.html
aws ecr get-login-password | docker login --username AWS --password-stdin $REGISTRY

# Pull the image
docker pull $IMAGE

# Run the image. You'll likely need to set additional flags. See https://docs.docker.com/reference/cli/docker/container/run.
docker run $IMAGE

From guardian/actions-publish-image.

github-actions · 2026-09-30T13:21:00Z

Deploy build 31884 of `dotcom:rendering-all` to CODE

All deployment options

From guardian/actions-riff-raff.

github-actions · 2026-09-30T15:02:10Z

Deploy build 3516 of `dotcom:ab-testing` to CODE

All deployment options

From guardian/actions-riff-raff.

akash1810 · 2026-09-30T15:12:04Z

+      - name: Set up Node environment
+        uses: ./.github/actions/setup-node-env
+
+      # TODO abstract this into a reusable GitHub Action e.g. https://github.com/guardian/actions-publish-image


Along with #16811 and guardian/commercial#2745 we've done this three times, so it's time to refactor! This'll be done in a separate PR.

guardian/actions-ecr#36.

akash1810 · 2026-09-30T15:21:48Z

@@ -1,2 +1,2 @@
-// this should get overwritten in CI (see .github/workflows/container.yml)
+// this should get overwritten in CI (see .github/workflows/container-production.yml)


dotcom-rendering/.github/workflows/container-production.yml

Lines 36 to 38 in a598f34

- name: Add commit hash for PRout

working-directory: dotcom-rendering

run: echo 'export const GIT_COMMIT_HASH = "${{ needs.facts.outputs.commitSha }}";' > src/server/prout.ts

Copilot

Copilot review overview

🟡 Changes recommended

Repository checkout lacks required permissions, and removing the GHCR image breaks a documented external consumer.

Review effort: Balanced
Findings: 1 · 2

Open (3)

Missing contents read permission blocks repository checkout · New
Removing container job breaks Commercial E2E image consumption · New
Removing Node version synchronization check risks runtime drift · New

What changed in this PR

Consolidates deployment packaging around the production ECR image and removes the separate GHCR container build.

Changes:

Extracts deployment artifacts from the production image.
Simplifies RiffRaff bundle construction.
Removes the legacy GHCR container pipeline.

File	Description
`.github/workflows/cicd.yml`	Routes publishing through the production image.
`.github/workflows/publish.yml`	Pulls ECR artifacts and constructs deployment bundles.
`.github/workflows/container.yml`	Removes the legacy GHCR workflow.
`dotcom-rendering/Containerfile`	Removes the legacy CI container definition.
`dotcom-rendering/makefile`	Decouples bundle construction from build and synth.
`dotcom-rendering/scripts/check-node-versions.mjs`	Removes the deleted Containerfile version check.
`dotcom-rendering/src/server/prout.ts`	Updates the commit-hash workflow reference.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

akash1810 · 2026-09-30T15:50:00Z

+      - name: CDK synth
        env:
          IMAGE_DIGEST: ${{ inputs.production-image-digest }}
        working-directory: dotcom-rendering
+        run: make cdk-synth


Performing CDK synth as a discrete step makes it more explicit where the IMAGE_DIGEST environment variable is used.

The synthed files were previously in the GitHub hosted Docker image. They do not form part of the ECR hosted image so we synth here. Consequently, this workflow also now clones includes actions/checkout.

akash1810 · 2026-09-30T15:52:53Z

 	@pnpm cdk diff --path-metadata false --version-reporting false --profile frontend

-riffraff-bundle: clean-dist build cdk-synth
+riffraff-bundle:


This make task now does one thing - shuffle files around into a structure suitable for creating EC2 tarballs.

akash1810 · 2026-09-30T15:53:55Z

-        id: create-container
-        run: echo "CONTAINER_ID=$(docker create ${{ inputs.container-image }})" >> "$GITHUB_OUTPUT"
-      - name: Move container files into GHA Workspace
-        run: docker cp ${{ steps.create-container.outputs.CONTAINER_ID }}:/opt/app/dotcom-rendering/dotcom-rendering/. ${{ github.workspace }}


This is the main change. Previously we sourced files from within the GitHub hosted Docker image. Now we source files from the ECR hosted Docker image.

akash1810 · 2026-09-30T15:55:47Z

+      - name: CDK synth
        env:
          IMAGE_DIGEST: ${{ inputs.production-image-digest }}
        working-directory: dotcom-rendering
+        run: make cdk-synth


The synthed files were previously in the GitHub hosted Docker image. They do not form part of the ECR hosted image so we synth here. Consequently, this workflow also now clones includes actions/checkout.

akash1810 · 2026-09-30T16:06:24Z

+        working-directory: dotcom-rendering
+        run: |
+          CONTAINER_ID=$(docker create ${{ steps.ecr-pull.outputs.IMAGE }})
+          docker cp $CONTAINER_ID:/app/. dist


Copy files into dist as that is build-riffraff-bundle.mjs expects them to be¹.

Footnotes

build-riffraff-bundle.mjs is called via make riffraff-bundle. ↩

github-actions · 2026-10-01T07:59:45Z

Hello 👋! When you're ready to run Chromatic, please apply the run_chromatic label to this PR.

You will need to reapply the label each time you want to run Chromatic.

Click here to see the Chromatic project.

AshCorr

Nice work!

Do we have any statistics on CI build time? As I recall the previous GHCR container build was a bit slow.

akash1810 · 2026-10-01T08:37:10Z

Do we have any statistics on CI build time? As I recall the previous GHCR container build was a bit slow.

Strangely, the difference is marginal at -2s.

Build summary on this branch

Build summary on branch dependabot/npm_and_yarn/webpack-8db84ccbe1

PR #16827.

AshCorr · 2026-10-01T08:42:27Z

Build summary on branch dependabot/npm_and_yarn/webpack-8db84ccbe1

Ah I think when we used the GHCR image for Playwright it was built in series with the production image, so I guess the speed gains were already realized when you merged your last PR.

akash1810 · 2026-10-01T08:55:12Z

Ah I think when we used the GHCR image for Playwright it was built in series with the production image, so I guess the speed gains were already realized when you merged your last PR.

Good point! Looking at the build of #15853 which hasn't been rebased to include the changes in #16811, the build time decreased by ~2.5mins.

arelra

Great work 👏

See also #16808.

To improve readability.

This file was used to build the image published to GitHub Packages. It isn't needed anymore, so can be deleted.

gu-prout · 2026-10-05T09:07:36Z

Seen on PROD (merged by @akash1810 7 minutes and 57 seconds ago) Please check your changes!

akash1810 · 2026-10-06T11:11:52Z

Fastly metrics don't seem to be negatively impacted by this change, in that the number of 403s hasn't changed much since deployment (highlighted in the screenshot below).

akash1810 · 2026-10-08T09:07:34Z

Confirming the volume/pattern of 403 responses remains unchanged since this change:

This change routes all traffic for tag-page-rendering in PROD to ECS. Previous attempts at this (#16774) were reverted (#16776) due to an issue with assets on the CDN (see also #16808). The root cause of that issue has now been addressed (#16821) so let's try again!

* feat(tag-page-rendering): Serve 100% of PROD traffic from ECS This change routes all traffic for tag-page-rendering in PROD to ECS. Previous attempts at this (guardian#16774) were reverted (guardian#16776) due to an issue with assets on the CDN (see also guardian#16808). The root cause of that issue has now been addressed (guardian#16821) so let's try again! * feat(tag-page-rendering): Increase minimum tasks to 12 At 3 tasks and 100% traffic, ECS latecy averaged 300ms. This is quite high compared to EC2 which averages around 40ms. This change over-provisions to 12 tasks in an attempt to mitigate high latency. Co-authored-by: Connor O'Malley<connoromalleyatwork@users.noreply.github.com>

akash1810 force-pushed the aa/remove-github-container branch 5 times, most recently from 287d303 to ef4a307 Compare September 30, 2026 13:17

akash1810 force-pushed the aa/remove-github-container branch 4 times, most recently from 6b08435 to d46a3a7 Compare September 30, 2026 14:58

akash1810 added the maintenance Departmental tracking: maintenance work, not a fix or a feature label Sep 30, 2026

akash1810 force-pushed the aa/remove-github-container branch from d46a3a7 to 93e968d Compare September 30, 2026 14:59

akash1810 force-pushed the aa/remove-github-container branch from 0343ab2 to 5ed2cf5 Compare September 30, 2026 15:19

akash1810 commented Sep 30, 2026

View reviewed changes

akash1810 requested a balanced review from Copilot September 30, 2026 15:26

Copilot started reviewing on behalf of akash1810 September 30, 2026 15:26

View session

Copilot AI reviewed Sep 30, 2026

View reviewed changes

Comment thread .github/workflows/publish.yml

Comment thread .github/workflows/cicd.yml

Comment thread dotcom-rendering/scripts/check-node-versions.mjs

akash1810 mentioned this pull request Sep 30, 2026

chore: Update check-node-versions.mjs to check Production.dockerfile #16824

Merged

akash1810 force-pushed the aa/remove-github-container branch from 5ed2cf5 to 76690ed Compare September 30, 2026 15:47

akash1810 commented Sep 30, 2026

View reviewed changes

akash1810 changed the title ~~wip~~ ci: Derive EC2 artifacts from ECS build Sep 30, 2026

akash1810 commented Sep 30, 2026

View reviewed changes

akash1810 added the run_chromatic Runs chromatic when label is applied label Sep 30, 2026

github-actions Bot removed the run_chromatic Runs chromatic when label is applied label Sep 30, 2026

akash1810 marked this pull request as ready for review October 1, 2026 07:59

akash1810 requested a review from a team as a code owner October 1, 2026 07:59

AshCorr approved these changes Oct 1, 2026

View reviewed changes

akash1810 requested a review from a team October 1, 2026 14:17

arelra approved these changes Oct 2, 2026

View reviewed changes

akash1810 added 4 commits October 2, 2026 15:40

ci: Derive EC2 artifacts from ECS build

b6f4045

See also #16808.

ci: Perform CDK synth and file suffling independently

3eedf02

To improve readability.

ci: Stop producing Docker image for GitHub Packages

c9136ea

chore: Remove unused Containerfile

9c4aa43

This file was used to build the image published to GitHub Packages. It isn't needed anymore, so can be deleted.

akash1810 force-pushed the aa/remove-github-container branch from 76690ed to 9c4aa43 Compare October 2, 2026 14:41

akash1810 added the run_chromatic Runs chromatic when label is applied label Oct 5, 2026

github-actions Bot removed the run_chromatic Runs chromatic when label is applied label Oct 5, 2026

akash1810 added this pull request to the merge queue Oct 5, 2026

Merged via the queue into main with commit da129cf Oct 5, 2026
36 checks passed

akash1810 deleted the aa/remove-github-container branch October 5, 2026 08:59

gu-prout Bot added Pending-on-PROD Seen-on-PROD and removed Pending-on-PROD labels Oct 5, 2026

akash1810 mentioned this pull request Oct 9, 2026

Serve 100% of tag-page-rendering PROD from ECS #16908

Merged

		@@ -1,2 +1,2 @@
		// this should get overwritten in CI (see .github/workflows/container.yml)
		// this should get overwritten in CI (see .github/workflows/container-production.yml)

	- name: Add commit hash for PRout
	working-directory: dotcom-rendering
	run: echo 'export const GIT_COMMIT_HASH = "${{ needs.facts.outputs.commitSha }}";' > src/server/prout.ts

Conversation

akash1810 commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this change?

How has this change been tested?

Artifact comparison

Deploy to CODE

Before

After

Footnotes

Uh oh!

github-actions Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Image pushed to AWS ECR

Uh oh!

github-actions Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

akash1810 Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Footnotes

Uh oh!

github-actions Bot commented Oct 1, 2026

Uh oh!

AshCorr left a comment

Choose a reason for hiding this comment

Uh oh!

akash1810 commented Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AshCorr commented Oct 1, 2026

Uh oh!

akash1810 commented Oct 1, 2026

Uh oh!

arelra left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gu-prout Bot commented Oct 5, 2026

Uh oh!

akash1810 commented Oct 6, 2026

Uh oh!

akash1810 commented Oct 8, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

akash1810 commented Sep 30, 2026 •

edited

Loading

github-actions Bot commented Sep 30, 2026 •

edited

Loading

github-actions Bot commented Sep 30, 2026 •

edited

Loading

github-actions Bot commented Sep 30, 2026 •

edited

Loading

akash1810 Sep 30, 2026 •

edited

Loading

akash1810 commented Oct 1, 2026 •

edited

Loading