Repository navigation
Give the first credential fetch 30s and keep AWS error causes - #199
Conversation
The default MySQL connection factory tested a monitor's credentials func with a 500ms deadline. The first call to an AWS source (Secrets Manager or IAM auth) might have to fetch AWS credentials first, e.g. EKS IRSA AssumeRoleWithWebIdentity plus a cross-account AssumeRole, which takes seconds, so it failed with "request canceled, context deadline exceeded" and the caller retried. Make now waits up to DefaultCredentialTimeout (30s), configurable with the new WithCredentialTimeout option (zero disables it). Reloads by the hotswap driver are unchanged: they use the connect context. aws.Secret.GetSecretPayload, aws.Secret.GetSecret, and blip.DefaultPasswordSecretParser formatted errors with %s, so callers couldn't use errors.As to detect, e.g., *aws.RequestCanceledError. They now wrap with %w; messages are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Follow-ups from review of the credential timeout change: - WithCredentialTimeout with a negative value now makes Make return an error, like timeout-connect, instead of silently disabling the bound. Zero still disables it. Move the context setup into credentialContext and test its deadline directly, since a Make test can't tell a disabled timeout from the 30s default. - aws.ConfigFactory.Make dropped the IMDS error when region auto-detection failed. Return it wrapped in regionError, which keeps the same message and unwraps to the cause. - Docs: the monitor stop section still said credentials take up to 500ms, and the release note didn't mention that startup can wait for plans.table credentials. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Follow-ups from a second review of the credential timeout change: - Start the credential timeout right before the first call to the credentials func, as WithCredentialTimeout documents. Since the last commit it also covered f.Credentials, which loads the AWS config and can auto-detect the region. - WithCredentialTimeout panics on a negative value, like time.NewTicker, instead of every Make returning an error and monitors retrying forever. - Zero in the factory field now means DefaultCredentialTimeout, so a factory struct literal gets the default instead of no timeout. WithCredentialTimeout(0) still disables it. - Release note: say that a slow first fetch used to make loading plans.table fail, so Blip exited at startup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- WithCredentialTimeout and the release note: zero disables the timeout, so a hung credentials source can block a monitor stop or Blip startup indefinitely. The note said stops take "up to this long". - WithCredentialTimeout: validate a value from user config before passing it, since a negative value panics. - Config docs: the 30s credential wait in the monitor-stop section is the default, not a fixed limit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zyin-squareup
left a comment
There was a problem hiding this comment.
🤖 Review findings - created by zyin-squareup's code review agent - for /pull/199, 3eacb18.
Verdict: ship — 1 finding: 0 blocking, 0 important, 1 nice-to-have, 0 suggestions.
This PR gives dbconn factory.Make 30s instead of 500ms for its first credential fetch, with a WithCredentialTimeout option to change it. It also keeps AWS and JSON error causes by switching %s to %w and wrapping the IMDS region error. The main risk is the longer blocking window during monitor startup and stop. That trade-off is disclosed, and I found nothing that holds a lock or depends on the old bound. The only finding is a docs understatement.
Assumptions & unknowns
- Reviewed head
3eacb18bc19a7416f17b4def16707513c456e4d2(4 commits) against a freshorigin/main. The head was re-checked before posting. - Local
go build,go vetandgo testcould not run because this session's sandbox denied them. Validation here means source inspection plus the green CIbuildjob. - I ran no mutation or pin checks. Test-pinning claims below come from reading the tests.
Nice-to-have
[NICE-TO-HAVE] Monitor-stop docs understate the startup wait when plans come from a table (CONFIRMED)
File:
blip/docs/content/config/config-file.md
Line 209 in 3eacb18
Failure scenario: A monitor uses a Secrets Manager or IAM auth password and
plans.table, and it is stopped during startup. The stop can wait about 30s (the first Make), then another 30s (the Make for plan loading), then 5s (the query). The docs give plan loading "up to 5 seconds, connecting included."Evidence:
monitor/monitor.go:301(m.makeDB()) is followed bymonitor/monitor.go:323(m.planLoader.LoadMonitor(m.cfg, m.dbMaker)). The only stop checks are in the retry-sleep branches.LoadMonitorcallsdbMaker.Make(mon)again (plan/loader.go:198), which runs a freshf.credentialContext()fetch (dbconn/factory.go:162).- The 5s covers only
ReadTable's query context (plan/loader.go:490). - The base had the same wording, but the gap was 500ms. This PR makes it 30s.
Impact is limited. The second slow fetch happens only when the first one was slow but succeeded, and the loader stops monitors in the background with a monitor-stop-timeout event.
Suggested fix: Add to the plan-loading clause: "plus one more credentials call (up to the credential timeout) when plans are in a table."
General suggestions
- The PR already lists two follow-ups that would remove most of this wait: caching
aws.ConfigFactoryper region (eachMakecurrently rebuilds the credentials cache) and a context-awareDbFactory.
The one thing that could have broken, verified
The longer Make timeout could have blocked something that holds a lock or depends on the old 500ms bound. Verified by inspection that this doesn't happen:
Makeruns in the monitor'srunLoopgoroutine withoutrunMux, which is taken only to assignm.db(monitor/monitor.go:304-309).- The loader stops monitors in the background.
credentialFuncgets the new context only atdbconn/factory.go:162-164.Repo.ReloadDSNstill uses the driver's connect context.
Verified correct
- Error messages are unchanged:
%s→%winaws/secret.go:36,52andblip.go:124produces the same text, andregionError.Error()returns the old IMDS string exactly. Nothing in the repo matches on or type-asserts these errors. - Wrapping changes no control flow: the
DeadlineExceeded/Canceledchecks inmonitor/engine.goandstatus/handle collector errors, notMakeerrors.setErrlogs with the monitor ID and retries. - Timeout semantics hold:
- A zero-value
factory{}gets 30s. WithCredentialTimeout(0)maps to the -1 sentinel, which gives a context with no deadline.- A negative value panics when the option is built, as documented.
- A zero-value
- Credential sources: static, file and my.cnf ignore the context. Only IAM auth and Secrets Manager see the 30s.
- Tests, read but not mutation-checked:
TestMakeCredentialTimeoutDefault(700ms provider) fails against a 500ms deadline.TestMakeCredentialTimeoutOption(2s provider, 50ms option) fails if the option is ignored. Both enter throughNewConnFactory(...).Make. - Release notes match the code.
- Categories that don't apply: concurrency beyond the lock check above, security, schemas and migrations.
- CI:
build(go build ./...+go test ./...), DCO, Semgrep and zizmor all pass at this head.
This review was generated by Claude Code (claude-opus-5-5). Local tests were not run (sandbox denied go commands); CI build and tests passed.
When a monitor's plans are in a table, startup calls Make again to load them, which makes another credentials call with its own timeout. So a stop during startup can wait up to twice the credential timeout. The monitor-stop section of the config docs and the release note counted only one call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Ian Oberst <ioberst@block.xyz>
zyin-squareup
left a comment
There was a problem hiding this comment.
🤖 Re-review by zyin-squareup's code review agent - /pull/199, 5665fd7.
Verdict: ship - the only change since the last review is docs, and it fixes the prior finding; 2 new nice-to-haves.
Prior findings: monitor-stop docs understated the wait when plans are in a table - fixed (release note and config-file.md:209 now count the second Make from LoadMonitor)
New findings:
- [NICE-TO-HAVE] Local auto-detect waits up to 30s per probe and isn't mentioned in the docs (monitor/loader.go:668).
- Trigger: no monitors are configured and top-level
aws.iam-authoraws.password-secretis set, soApplyDefaultscopies it into each localhost probe. - Each probe's
Makecan now block 30s (was 500ms) on a hung source, users × (sockets+1) times, at synchronous startup. The release note names onlyplans.table.
- Trigger: no monitors are configured and top-level
- [NICE-TO-HAVE] No test covers the region-error wrapping at aws/aws.go:43.
TestRegionErrorWrapsIMDSErrorconstructsregionErrordirectly, so changing line 43 toregionError{}still passesgo test ./aws/ ..- The production code is correct by inspection.
Validation:go vetis clean andgo test ./aws/ .passes. In./dbconn/, the new tests pass; onlyTestConnectfails, because it needs MySQL on :33800.
Pin checks: reverting the default to 500ms and the three%w→%ssites each made a test fail; dropping the region cause failed none. CI is green.
TestRegionErrorWrapsIMDSError built regionError directly, so dropping the cause in ConfigFactory.Make passed every test. Make the IMDS client options a package variable so a test can point Region at a local IMDS that returns 404, then check that Make keeps the old message and that errors.As finds the IMDS response error. Release note: local auto-detect copies top-level aws.iam-auth and aws.password-secret into each probe, so it can wait up to the credential timeout for each user and socket it tries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Ian Oberst <ioberst@block.xyz>
zyin-squareup
left a comment
There was a problem hiding this comment.
🤖 Re-review by zyin-squareup's code review agent - /pull/199, bf8c71f.
Verdict: ship - the delta since 5665fd7 is a real entry-point test for the region error plus a release-note line, and both close the open findings.
Prior findings:
- Monitor-stop docs understated the wait when plans are in a table: fixed
- Local auto-detect wait was missing from the docs: fixed
- No test pinned region-error wrapping in
ConfigFactory.Make: fixed
New findings: none
Validation:go vetclean.go test ./aws/ .andgo test -run 'Credential|Timeout' ./dbconn/pass. Pin check: makingMakereturnregionError{}with the cause dropped failsTestConfigFactoryWrapsRegionError(killed). CI (build, DCO, Semgrep, zizmor) is green.
Why
The default MySQL connection factory (
dbconn.NewConnFactory) fetches a monitor's credentials once inMake, before it builds the DSN. It gave that first fetch only 500ms.With AWS Secrets Manager or IAM auth, the first fetch may also have to get AWS credentials. On EKS that can be IRSA (
AssumeRoleWithWebIdentity) plus a cross-accountAssumeRole, which takes 5 to 11 seconds. So the firstMakeoften failed withoperation error Secrets Manager: GetSecretValue, get identity: get credentials: request canceled, context deadline exceeded. Monitors retried, so this was mostly log noise. But loading shared plans fromplans.tablefailed, and Blip exited at startup.Callers also couldn't tell this timeout from a real failure.
aws.Secret.GetSecretPayloadformatted the SDK error with%s, soerrors.Ascouldn't find*aws.RequestCanceledError.What
Makenow waits up todbconn.DefaultCredentialTimeout(30s) for the first credentials. The newdbconn.WithCredentialTimeoutoption changes it.How
ConnFactoryOption, likeWithPasswordSecretParser.Maketakes no context, and adding one would change theblip.DbFactoryinterface.time.NewTicker. A factory struct literal gets the default.Repo.ReloadDSN) don't change. They already use the driver's connect context.%sbecame%winGetSecretPayload,GetSecret, andblip.DefaultPasswordSecretParser.aws.ConfigFactory.Makedropped the IMDS error whenregion: autofailed. It now returns it wrapped in a type that keeps the old message.Risk
Low for healthy setups: they see fewer false startup errors. When AWS is slow or down,
Makenow blocks up to 30s instead of 500ms. So a monitor stop during startup can take that long, and Blip startup can wait that long forplans.table. The release notes and the monitor-stop section ofconfig-file.mddescribe this. Error messages didn't change, so callers that match on them still work.Testing
I put back the 500ms deadline and the
%sformatting and checked that the new timeout anderrors.Astests fail on that code.Bigger picture
Two follow-ups are out of scope here:
aws.ConfigFactorybuilds a new AWS config, with an empty credentials cache, on everyMake. So every monitor repeats the slow first fetch. Caching the config per region would make only the first one slow. Its 200ms IMDS deadline forregion: autois also short.Makecan't be canceled. A context-aware factory interface would let a monitor stop, or Blip startup, give up the wait.Generated with Claude Code