[feiyunzhou]

WHY?

Nightly nactl update runs fail intermittently. foxops pushes the branch and
asks GitLab to open an MR in the same breath; GitLab answers from a branch cache
that can lag the push and rejects it with Source branch "<x>" does not exist.
foxops did not retry. Logging the resulting error then took ~52s, which pushed
the reply past the 60s reverse-proxy timeout, so the caller saw a misleading 504
instead of the real GitLab error.

WHAT?

• Retry the merge request POST for up to 10s while GitLab still reports the
  pushed source branch as missing. Safe to retry: GitLab only sends this message
  when it created nothing.
• Raise MergeRequestCreationFailedError carrying GitLab's own message, instead
  of httpx's Client error '400 Bad Request' for url ....
• Render plain tracebacks. structlog's ConsoleRenderer silently switches to
  rich's formatter when rich is installed, and that formatter reprs every
  frame's local variables.

ADDITIONAL NOTES

The traceback was rendered twice per failed request — once in catch_all, once
more after Starlette re-raises and uvicorn logs it. Rendering is synchronous on
the event loop, so it stalled every other in-flight request too, and the first
render happens before the error response is sent.

Not touched: the mergeability-check loop in merge_request() can still spend up
to 25s of the same 60s budget. Separate concern.

[ermirry]

Found one timing issue with the retry deadline. All six added tests pass locally.

[ermirry]

[P2] Enforce the retry deadline during HTTP requests

stop_after_delay(10) checks elapsed time after a failed attempt; it does not interrupt an in-flight POST. Each retry inherits the client's 120-second HTTP timeout, so a quick missing-branch response followed by a slow retry can exceed both the stated 10-second budget and the proxy's 60-second timeout, producing another 504. Apply a deadline covering retry waits and HTTP requests.

Reproduced with a 0.1-second budget: a slow second attempt kept the call running for 0.62 seconds.

[feiyunzhou]

Thanks for catching this! Fixed

I give the POST request itself a timeout, same value as the 10s retry budget. So even GitLab answer very slow, this one request cannot run more than 10s (before it use client's default 120s).

Also I simplify the retry logic: now reuse httpx.HTTPStatusError (from raise_for_status()) directly as the retry signal, not need custom exception class anymore. One small function decide: if error is "branch not visible yet" then retry it, if other error then fail immediately, not retry.

[feiyunzhou]

Tested this change on a real environment, not only unit tests.

What I did:

• Built the image from the latest branch head (86e9bdd) locally, as multi-arch (amd64 + arm64).
• Deployed it to a dev control plane and checked in the pod that the new code is what runs.
• Created 3 test incarnations with Terraform, then triggered an update on all 3.

Result:

• All 3 updates created the merge request with a single POST (HTTP 201). No retry was needed.
• Automerge was triggered for all 3. No errors or tracebacks in the foxops logs.

Limitation:

• I could not reproduce the GitLab "source branch does not exist" race on a real GitLab, so the retry path (and the give-up log) is covered by the unit tests only.