Repository navigation
feat(load): retry a load while another load holds the table - #335
Conversation
Loads into one table are serialised server-side. A second writer gets a 409 whose code is RESOURCE_LOCKED and whose Retry-After says how long to hold off. The CLI printed the error and exited 1, so concurrent appends from several processes mostly failed (7 of 8 in Tenki's run). `databases load` and `databases tables load` now retry that 409, honouring Retry-After, for up to --retry-timeout seconds (default 60; 0 sends once). Matched on the error code, never the message, so a permanent CONFLICT is still fatal. The spinner says when it is waiting and for how long.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| spinner.set_message(format!( | ||
| "Loading table... another load holds the table, retrying in {}s", | ||
| wait.as_secs() | ||
| )); |
There was a problem hiding this comment.
nit: The spinner keeps showing "retrying in Ns" after the sleep ends and during the next attempt (not blocking).
Fix: reset the spinner message to "Loading table..." after the sleep. One way is to pass a second callback, or to set the message before each post_raw_with_headers call. Without the reset, the user sees a stale countdown while the retry request runs.
| .and_then(|v| v.to_str().ok()) | ||
| .and_then(|v| v.trim().parse::<u64>().ok()) | ||
| .map(std::time::Duration::from_secs) | ||
| .unwrap_or(LOCK_RETRY_FALLBACK); |
There was a problem hiding this comment.
nit: A Retry-After: 0 makes post_load_with_retry send requests in a tight loop until the deadline (not blocking).
Fix: set a minimum wait, for example .max(Duration::from_secs(1)), on the parsed value. RFC 9110 allows Retry-After: 0. With a 60s budget and no minimum, one CLI process can send hundreds of POSTs to a locked table. The load_retries_resource_locked_until_it_succeeds test uses "0", so the test needs a different header value or a test-only minimum.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
1 participant
Why
Loads into one table are serialised server-side. A second writer gets a 409 with code
RESOURCE_LOCKEDand aRetry-Afterheader. The CLI printed the error and exited 1, so concurrent appends from several processes mostly failed. In Tenki's integration run, 7 of 8 concurrentdatabases load --appendcalls exited 1.What
databases loadanddatabases tables loadretry aRESOURCE_LOCKED409, honouringRetry-After(fallback 5s), for up to--retry-timeoutseconds. Default 60;0sends exactly once.CONFLICT(idempotency-key reuse, catalog scoped elsewhere) is still fatal.Api::post_raw_with_headersadded so the load path can readRetry-After;post_rawdelegates to it.Tests
lock_retry_after_matches_resource_locked_only: code match, header parse, fallback, and the non-retry cases.load_retries_resource_locked_until_it_succeeds: two 409s then a 200 against mockito.load_retry_timeout_zero_sends_once_and_returns_the_409.