Skip to content

Serve 100% of tag-page-rendering PROD from ECS - #16908

Merged
akash1810 merged 2 commits into
mainfrom
tag-page-rendering-ecs-100-try-2
Oct 9, 2026
Merged

akash1810 merged 2 commits into
mainfrom
tag-page-rendering-ecs-100-try-2

Conversation

@akash1810

@akash1810 akash1810 commented Oct 9, 2026 •

Copy link
Copy Markdown
Member

What does this change?

This change routes all traffic for tag-page-rendering in PROD to ECS. Previous attempts at this (#16774) were reverted (#16776) due to an issue with assets on the CDN (see also #16808). The root cause of that issue has now been addressed (#16821) so let's try again!

This change also increases the minimum tasks to 12. At 3 tasks and 100% traffic, ECS latency averaged 300ms. This is quite high compared to EC2 which averages around 40ms. This change over-provisions to 12 tasks in an attempt to mitigate high latency; we'll tune this once we have data for how its performing.

How has this change been tested?

Not possible, as this changes PROD infrastructure.

After deploying, things to check are:


Co-authored by: @connoromalleyatwork.

@akash1810
akash1810 requested a review from a team as a code owner October 9, 2026 11:20
@akash1810 akash1810 added the feature Departmental tracking: work on a new feature label Oct 9, 2026
@akash1810 akash1810 changed the title Tag page rendering ecs 100 try 2 Serve 100% of tag-page-rendering PROD from ECS Oct 9, 2026
@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

This change routes all traffic for tag-page-rendering in PROD to ECS.

Previous attempts at this (#16774) were reverted (#16776)
due to an issue with assets on the CDN (see also #16808).

The root cause of that issue has now been addressed (#16821)
so let's try again!
At 3 tasks and 100% traffic, ECS latecy averaged 300ms.
This is quite high compared to EC2 which averages around 40ms.

This change over-provisions to 12 tasks in an attempt to mitigate high latency.

Co-authored-by: Connor O'Malley<connoromalleyatwork@users.noreply.github.com>
@akash1810
akash1810 force-pushed the tag-page-rendering-ecs-100-try-2 branch from 3670585 to 082bc93 Compare October 9, 2026 13:01
@github-actions

github-actions Bot commented Oct 9, 2026

Copy link
Copy Markdown

🚀 Image pushed to AWS ECR

Image digest: sha256:345db3e723a3fab88f92e76545e54b76fb92cdada65be814b8239255614ea3f5

🐛 Run the image locally

The following can be used to run the image locally:

# Refer to image using the immutable digest. Find alternatives below.
IMAGE_IDENTIFIER="@sha256:345db3e723a3fab88f92e76545e54b76fb92cdada65be814b8239255614ea3f5"

# Refer to image using branch tag
# IMAGE_IDENTIFIER=":branch-tag-page-rendering-ecs-100-try-2"

# Refer to image using build tag
# IMAGE_IDENTIFIER=":build-32222"

# Set environment variables for the AWS CLI
AWS_PROFILE="<A_PROFILE_FROM_JANUS>"
AWS_DEFAULT_REGION="eu-west-1"

IMAGE_ACCOUNT_ID=$(aws ssm get-parameter --name /organisation/accounts/artifacts --query "Parameter.Value" --output text)
REGISTRY="${IMAGE_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com"
IMAGE="${REGISTRY}/guardian/dotcom-rendering/dotcom-rendering${IMAGE_IDENTIFIER}"

# Login to AWS ECR https://docs.aws.amazon.com/AmazonECR/latest/userguide/registry_auth.html
aws ecr get-login-password | docker login --username AWS --password-stdin $REGISTRY

# Pull the image
docker pull $IMAGE

# Run the image. You'll likely need to set additional flags. See https://docs.docker.com/reference/cli/docker/container/run.
docker run $IMAGE

From guardian/actions-ecr.

@akash1810
akash1810 added this pull request to the merge queue Oct 9, 2026
Merged via the queue into main with commit 1608a8c Oct 9, 2026
32 checks passed
@akash1810
akash1810 deleted the tag-page-rendering-ecs-100-try-2 branch October 9, 2026 13:16
@gu-prout

gu-prout Bot commented Oct 9, 2026

Copy link
Copy Markdown

Seen on PROD (merged by @akash1810 6 minutes and 43 seconds ago) Please check your changes!

@akash1810

Copy link
Copy Markdown
Member Author

Initial metrics show latency has creeped up a touch, from 40ms to 70ms:

image

This is still within the thresholds of expectations of:

github-merge-queue Bot pushed a commit that referenced this pull request Oct 12, 2026
Since #16746, ECS has been serving 100% of CODE.
Since #16908, ECS has been serving 100% of PROD.

This change scales down the EC2 infrastructure of these apps to 0 instances.
This has two benefits:
- Cost saving
- Reducing "noise" from logs by removing EC2 launch configuration logs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature Departmental tracking: work on a new feature Seen-on-PROD

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants