Skip to content

nginx: re-resolve service addresses so it survives docker compose restart - #13

Merged
ArtyomSavchenko merged 2 commits into
mainfrom
fix-nginx-restart
Sep 29, 2026
Merged

ArtyomSavchenko merged 2 commits into
mainfrom
fix-nginx-restart

Conversation

@ArtyomSavchenko

@ArtyomSavchenko ArtyomSavchenko commented Sep 29, 2026 •

Copy link
Copy Markdown
Member

After docker compose restart nginx could return 502 (or proxy to the wrong service): it resolved service names once at startup, and restarted containers can come back with different IPs. Seen in CI: nginx sent /_accounts to 172.18.0.15, which by then was fulltext; account was at 172.18.0.3.

Fix (.huly.nginx)

  • resolver 127.0.0.11 valid=10s ipv6=off; — Docker's DNS on huly_net
  • upstreams go through a variable (set $upstream http://account:3000; proxy_pass $upstream;), which makes nginx resolve them per request (cached 10s)
  • set is placed before rewrite ... break (break skips later rewrite-module directives, including set)
  • prefixes are stripped with rewrite ^/_x/?(.*)$ /$1 break; so /_accounts still maps to / as with the old proxy_pass http://account:3000/;
  • commented-out optional services (print, aibot, love, pulse, telegram, github, export) follow the same pattern, so uncommenting them keeps working

Checked

  • Old and new config side by side (local nginx, stub services): identical upstream and path for /, /files?…, /_accounts, /_accounts/api/…?token=…, /_collaborator/…, /_transactor/<token>, /eyJ…, /_rekoni/…, /_stats/…?a=b&c=d, GET and POST
  • Changing a service's DNS answer: new config follows the new IP within 10s, old config keeps the startup one
  • CI: new step Services recreated behind a running nginx recreates account and transactor (new IPs) and re-runs smoke.sh --verify. With the old config the restart step failed again on this PR (ede6e6b); with the fix two runs in a row are green.

🤖 Generated with Claude Code

https://claude.ai/code/session_012bF628vAWVF7d1gqS2bQwG


Generated by Claude Code

ArtyomSavchenko and others added 2 commits September 29, 2026 10:48
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012bF628vAWVF7d1gqS2bQwG
Signed-off-by: Artem Savchenko <armisav@gmail.com>
nginx looked up service names once at startup. After 'docker compose
restart' containers can come back with different IPs, and nginx kept
proxying to the old ones: 502 (or another service) until nginx itself was
restarted.

Use Docker's DNS (127.0.0.11, valid=10s) and put upstreams in a variable so
proxy_pass resolves them per request. 'set' goes before 'rewrite ... break',
which would otherwise skip it, and prefixes are stripped with
'^/_x/?(.*)$ -> /$1' so /_accounts still maps to / as before. Routing of
all paths and query strings is unchanged (checked old vs new side by side).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012bF628vAWVF7d1gqS2bQwG
Signed-off-by: Artem Savchenko <armisav@gmail.com>
@ArtyomSavchenko
ArtyomSavchenko merged commit 7589fa1 into main Sep 29, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant