CI/CD: Improve deployment security

[kartpop]

Is your feature request related to a problem?
Our CI/CD pipelines have vulnerabilities that could lead to deployment incidents, similar to what occurred with the Glific team. Addressing these gaps is crucial to ensure deployment success and minimize failures.

Describe the solution you'd like

• Ensure that Staging deploys wait for a successful CI run by updating deploy-staging.yml with workflow_run and adding branch protection on main.
• Integrate a Discord webhook step to notify about staging and production deploy failures.
• Implement health checks for deployments by enabling the ECS deployment circuit breaker and polling the rolloutState.
• Add Docker image builds in CI for every PR to catch issues earlier.
• Set up ECS staging rehearsals for every merge to main by scaling the staging ECS service and verifying it, with notifications on failures.

Original issue

Our CI CD pipelines can be hardened. There are gaps that need fixing, which will prevent from us from having similar deployment incidents like what Glific team shared recently.
More details here

1. Deploys wait for CI to be green

Staging deploy currently races CI instead of waiting for it. Change deploy-staging.yml to trigger only after a successful CI run (workflow_run), add branch protection on main with required checks, and make create-release.yml confirm the tagged commit passed CI before building.

2. Discord notifications on deploy failures

Add a Discord webhook step (if: failure()) to the staging and production deploy workflows. Today a red deploy on main is only visible if someone opens the Actions tab — and Sentry can't catch failures that happen before the app boots.

3. Verify deployments actually became healthy

• Enable the ECS deployment circuit breaker on our services.
• After update-service, poll the deployment's rolloutState until COMPLETED (pass) or FAILED / timeout (fail). A plain curl isn't enough — the old task can answer with 200 while the new one is failing.
• Bake the git SHA into the image and return it from the health endpoint, so we can confirm the new code is the one responding.
• On the EC2 staging box, wait for the container healthcheck after docker compose up -d instead of reporting success when containers start.

4. Build the Docker image in CI

on every PR CI never builds the Dockerfile or validates the compose files, so breakage there is only discovered during a deploy. Add docker build ./backend and docker compose config checks to CI.

5. ECS staging rehearsal on every merge to main

Staging runs on EC2 (to save costs), so the production ECS deploy path is never tested before a release. After the EC2 staging deploy completes (serial, not parallel — the EC2 path owns migrations), scale the staging ECS service from 0 to 1 with the new image, verify with the rolloutState poll, then scale back to 0 (if: always()). One-time prep: remove autoscaling on the staging service and enable its circuit breaker. A failed rehearsal pings Discord but does not roll back EC2 staging.

[Ayush8923]

Need to check these 2 as well:

• rolloutState polling — after the two aws ecs update-service calls, the workflow currently exits green immediately. ECS then swaps tasks in the background; if the new task crashes on boot, the Actions run still shows. The issue asks for a poll loop after update-service: describe the service's deployment until rolloutState is COMPLETED (pass) or FAILED/timeout (fail). This is the ECS equivalent of what the PR did do for the EC2 SSM command.
• SHA in the health endpoint — bake the git SHA into the image and return it from /health, so you can confirm the new code is what's responding. This is why the issue says "a plain curl isn't enough — the old task can answer 200 while the new one is failing." Applies to prod primarily, but benefits staging too.

[Ayush8923]

This is the Relevent PR of this issue: #1158