From b85855c528c22d64dc2f5da197b8a778ec02f05a Mon Sep 17 00:00:00 2001 From: Floating-Y <118035379+Floating-Y@users.noreply.github.com> Date: Tue, 13 Oct 2026 11:09:14 +0800 Subject: [PATCH 1/3] Record why source checkpoints need bounded continuation Signed-off-by: Floating-Y <118035379+Floating-Y@users.noreply.github.com> --- ...ce-checkpoint-follows-a-complete-window.md | 134 ++++++++++++++++++ docs/decisions/README.md | 1 + 2 files changed, 135 insertions(+) create mode 100644 docs/decisions/0067-a-source-checkpoint-follows-a-complete-window.md diff --git a/docs/decisions/0067-a-source-checkpoint-follows-a-complete-window.md b/docs/decisions/0067-a-source-checkpoint-follows-a-complete-window.md new file mode 100644 index 000000000..466f95d5d --- /dev/null +++ b/docs/decisions/0067-a-source-checkpoint-follows-a-complete-window.md @@ -0,0 +1,134 @@ +# A source checkpoint follows a complete window + +Proposed 2026-10-13 (#1212); open: maintainer discussion, continuation storage and implementation. + +Discussion: [bounded continuation proposal and database reproduction](https://github.com/deeplethe/utopia/issues/1212#issuecomment-6137283600). + +## Context + +[Issue #1212](https://github.com/deeplethe/utopia/issues/1212) remains present on +`dev` at `259931ac702623351c2ed4e6487474790e57a6b0`. Custom, GitHub and Jira +consume at most 200 records and nevertheless finish the run as `ok`. +`last_successful_sync_start` then treats that run's start as a covered lower +bound. The unconsumed records can disappear from subsequent incremental reads. + +[PR #794](https://github.com/deeplethe/utopia/pull/794) made failed attempts and +updates during a run safe; it explicitly left the batch limit unresolved. +[PR #1061](https://github.com/deeplethe/utopia/pull/1061) requires each selected +GitHub issue to retain its complete discussion when its document is replaced. + +Keeping the old lower bound alone does not make progress: the next attempt +selects the same first 200 records. Charging only changed documents against +the limit helps custom responses but still cannot reach beyond GitHub's ten +100-record pages or Jira's ten 50-record pages. It also makes GitHub replay +unbounded: checking content requires reading the previous issues' discussions. + +## Proposed decision, pending discussion + +Add one private `sources.sync_state JSONB` column. Do not add it to the +serialized `Source` model or place it in user-editable source configuration. +It contains a completed `checkpoint` and an optional active window: + +- The fixed `since` used to open that window. +- The first batch's database start time, `window_started_at`. +- The provider's next page or `startAt`, the fetched page awaiting processing, + its consumption position, and whether the listing is exhausted. +- The source configuration identity, so a changed endpoint, repository or + project invalidates continuation and starts a safe new scan. + +Custom has no pagination or source-update timestamp in its existing protocol. +Preserve the first response's `items` and `deleted` snapshot and consume that +snapshot in batches. `doc_time` remains document time; it never selects a +continuation position or advances the checkpoint. A custom endpoint must still +return the complete requested window under its existing contract. + +GitHub and Jira fetch and persist one listing page at a time. Keep the existing +200-record processing budget, the ten-listing-page budget and the existing +per-issue discussion budget. Persist a fetched page before ingesting its items. +Save consumption progress at batch boundaries; a crash may replay the current +batch. Reuse `ingest_item` and the existing transactional document-version and +processing-job writes to make that replay harmless. + +Use GitHub's `sort=created&direction=asc` and Jira's +`ORDER BY created ASC, key ASC`. Sorting by update time moves already visited +members behind later offsets when they change, potentially skipping an older +unvisited member. Fixed creation order avoids that movement for ordinary +updates. A record entering the incremental set during the window may appear +before the saved offset, but its update belongs to the next window as well. + +An unfinished batch must retain the previous successful checkpoint and report +its committed counts. The smallest compatible status option uses the existing +`failed` status with an explicit batch-limit message, without raising an error +alert or making the queue retry a completed batch. A separate `partial` status +would express this better but changes the public status contract, database +checks, web types and translations. This choice requires discussion before +implementation. + +Only after the snapshot and listing are exhausted does a transaction set +`checkpoint = window_started_at` and clear the active window. The checkpoint +uses the first batch's start, not the last batch's start or finish. Updates +during any of the intervening batches therefore remain eligible next time. +Sync-run audit rows retain their actual per-batch start times. The successful +lower bound no longer depends on the retention of the latest 50 audit rows. + +## Pagination limitations to resolve + +An offset alone is insufficient when an upstream deletion, project transfer +or permission change removes a member before the saved page. Keep a page +boundary identity and re-read the boundary when resuming; if it has drifted, +restart from the fixed old `since` and replay through existing deduplication. +This must be tested before claiming coverage. It is not an upstream atomic +snapshot. GitHub also does not expose a second sorting key for equal creation +times; the resumed boundary must be checked rather than assuming stable ties. + +The existing GitHub discussion reader silently stops at ten full pages. Source +listing continuation must not turn a truncated discussion into a replacement +document or a complete success. Explicitly detect this condition and retain +the item for retry; supporting continuation inside unusually large individual +discussions needs its own bounded state, rather than removing that budget. + +Custom snapshots may be large. Saving progress once per batch avoids rewriting +the entire JSON response per item, but storage size and configuration access +checks still need review. There is no claim that the current custom protocol +can recover records the endpoint itself silently omits. + +## Upgrade and recovery + +Old `ok` rows cannot distinguish a complete window from a truncated one. +Initialize the new state for these three source kinds without a successful +lower bound, causing one bounded, resumable full recovery scan after upgrade. +Do not delete documents, versions, jobs or audit history. Existing identity and +content deduplication preserves unchanged documents during recovery. After that +scan, resume normal incremental reads; this is not permanent full polling. + +Before this change ships, an operator can rewind the successful run history for +an affected source only while its scheduler and worker are idle. That merely +re-exposes the lost window: the old binary still stops at the first 200 records, +so rewinding alone is not a complete recovery procedure. + +## Rejected approaches + +- Marking a capped attempt successful: advances over unread content. +- Keeping `since` but repeatedly taking the same prefix: does not finish. +- Counting only writes while retaining the fixed first ten pages: stalls on a + larger result set and increases GitHub detail requests without a bound. +- Advancing from maximum `doc_time`: custom has no source-update clock, and ties + can span a batch or page boundary. +- Raising or removing the limits, or disabling incremental sync permanently: + changes the resource contract instead of implementing continuation. +- Hiding progress in `config`, error strings or job payload history: exposes or + overloads unrelated state, and makes retention and configuration changes + determine correctness. + +## Acceptance + +Reuse the existing wiremock and database fixtures with required PostgreSQL and +pgvector. Cover 201 records, several batches, more than each reader's ten-page +limit, a previously imported prefix, equal update timestamps, failure after +committed items, and changes during multiple batches. Assert actual `since`, +JQL and page parameters, completed checkpoints, resource budgets, stable +document identities, version counts and processing-job counts. Exercise every +changed provider branch, including boundary drift and crash replay. + +This record is a proposal. No continuation schema or production implementation +has been written or represented as accepted. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index b4d4490dc..f82d381f1 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -112,6 +112,7 @@ By domain; the domains are the files of [../design/](../design/README.md). **Sta | 0052 | [Document content is a read contract over the retained ledger](0052-document-content-is-a-read-contract.md) | Proposed | | | 0054 | [A source may push statements in the open contract](0054-a-source-may-push-statements-in-the-open-contract.md) | Implemented | | | 0065 | [The OCR reader chooses its provider](0065-readers-choose-their-provider.md) | Implemented | | +| 0067 | [A source checkpoint follows a complete window](0067-a-source-checkpoint-follows-a-complete-window.md) | Proposed | | ### [lakehouse-and-actions](../design/lakehouse-and-actions.md) From 19083fff16a4a44792065e4d49422a1c3b54809d Mon Sep 17 00:00:00 2001 From: Floating-Y <118035379+Floating-Y@users.noreply.github.com> Date: Mon, 19 Oct 2026 16:06:43 +0800 Subject: [PATCH 2/3] Record why source syncs stop at timestamp boundaries Signed-off-by: Floating-Y <118035379+Floating-Y@users.noreply.github.com> --- ...ce-checkpoint-follows-a-complete-window.md | 134 ------------------ ...checkpoint-ends-at-a-timestamp-boundary.md | 132 +++++++++++++++++ docs/decisions/README.md | 2 +- 3 files changed, 133 insertions(+), 135 deletions(-) delete mode 100644 docs/decisions/0067-a-source-checkpoint-follows-a-complete-window.md create mode 100644 docs/decisions/0072-a-source-checkpoint-ends-at-a-timestamp-boundary.md diff --git a/docs/decisions/0067-a-source-checkpoint-follows-a-complete-window.md b/docs/decisions/0067-a-source-checkpoint-follows-a-complete-window.md deleted file mode 100644 index 466f95d5d..000000000 --- a/docs/decisions/0067-a-source-checkpoint-follows-a-complete-window.md +++ /dev/null @@ -1,134 +0,0 @@ -# A source checkpoint follows a complete window - -Proposed 2026-10-13 (#1212); open: maintainer discussion, continuation storage and implementation. - -Discussion: [bounded continuation proposal and database reproduction](https://github.com/deeplethe/utopia/issues/1212#issuecomment-6137283600). - -## Context - -[Issue #1212](https://github.com/deeplethe/utopia/issues/1212) remains present on -`dev` at `259931ac702623351c2ed4e6487474790e57a6b0`. Custom, GitHub and Jira -consume at most 200 records and nevertheless finish the run as `ok`. -`last_successful_sync_start` then treats that run's start as a covered lower -bound. The unconsumed records can disappear from subsequent incremental reads. - -[PR #794](https://github.com/deeplethe/utopia/pull/794) made failed attempts and -updates during a run safe; it explicitly left the batch limit unresolved. -[PR #1061](https://github.com/deeplethe/utopia/pull/1061) requires each selected -GitHub issue to retain its complete discussion when its document is replaced. - -Keeping the old lower bound alone does not make progress: the next attempt -selects the same first 200 records. Charging only changed documents against -the limit helps custom responses but still cannot reach beyond GitHub's ten -100-record pages or Jira's ten 50-record pages. It also makes GitHub replay -unbounded: checking content requires reading the previous issues' discussions. - -## Proposed decision, pending discussion - -Add one private `sources.sync_state JSONB` column. Do not add it to the -serialized `Source` model or place it in user-editable source configuration. -It contains a completed `checkpoint` and an optional active window: - -- The fixed `since` used to open that window. -- The first batch's database start time, `window_started_at`. -- The provider's next page or `startAt`, the fetched page awaiting processing, - its consumption position, and whether the listing is exhausted. -- The source configuration identity, so a changed endpoint, repository or - project invalidates continuation and starts a safe new scan. - -Custom has no pagination or source-update timestamp in its existing protocol. -Preserve the first response's `items` and `deleted` snapshot and consume that -snapshot in batches. `doc_time` remains document time; it never selects a -continuation position or advances the checkpoint. A custom endpoint must still -return the complete requested window under its existing contract. - -GitHub and Jira fetch and persist one listing page at a time. Keep the existing -200-record processing budget, the ten-listing-page budget and the existing -per-issue discussion budget. Persist a fetched page before ingesting its items. -Save consumption progress at batch boundaries; a crash may replay the current -batch. Reuse `ingest_item` and the existing transactional document-version and -processing-job writes to make that replay harmless. - -Use GitHub's `sort=created&direction=asc` and Jira's -`ORDER BY created ASC, key ASC`. Sorting by update time moves already visited -members behind later offsets when they change, potentially skipping an older -unvisited member. Fixed creation order avoids that movement for ordinary -updates. A record entering the incremental set during the window may appear -before the saved offset, but its update belongs to the next window as well. - -An unfinished batch must retain the previous successful checkpoint and report -its committed counts. The smallest compatible status option uses the existing -`failed` status with an explicit batch-limit message, without raising an error -alert or making the queue retry a completed batch. A separate `partial` status -would express this better but changes the public status contract, database -checks, web types and translations. This choice requires discussion before -implementation. - -Only after the snapshot and listing are exhausted does a transaction set -`checkpoint = window_started_at` and clear the active window. The checkpoint -uses the first batch's start, not the last batch's start or finish. Updates -during any of the intervening batches therefore remain eligible next time. -Sync-run audit rows retain their actual per-batch start times. The successful -lower bound no longer depends on the retention of the latest 50 audit rows. - -## Pagination limitations to resolve - -An offset alone is insufficient when an upstream deletion, project transfer -or permission change removes a member before the saved page. Keep a page -boundary identity and re-read the boundary when resuming; if it has drifted, -restart from the fixed old `since` and replay through existing deduplication. -This must be tested before claiming coverage. It is not an upstream atomic -snapshot. GitHub also does not expose a second sorting key for equal creation -times; the resumed boundary must be checked rather than assuming stable ties. - -The existing GitHub discussion reader silently stops at ten full pages. Source -listing continuation must not turn a truncated discussion into a replacement -document or a complete success. Explicitly detect this condition and retain -the item for retry; supporting continuation inside unusually large individual -discussions needs its own bounded state, rather than removing that budget. - -Custom snapshots may be large. Saving progress once per batch avoids rewriting -the entire JSON response per item, but storage size and configuration access -checks still need review. There is no claim that the current custom protocol -can recover records the endpoint itself silently omits. - -## Upgrade and recovery - -Old `ok` rows cannot distinguish a complete window from a truncated one. -Initialize the new state for these three source kinds without a successful -lower bound, causing one bounded, resumable full recovery scan after upgrade. -Do not delete documents, versions, jobs or audit history. Existing identity and -content deduplication preserves unchanged documents during recovery. After that -scan, resume normal incremental reads; this is not permanent full polling. - -Before this change ships, an operator can rewind the successful run history for -an affected source only while its scheduler and worker are idle. That merely -re-exposes the lost window: the old binary still stops at the first 200 records, -so rewinding alone is not a complete recovery procedure. - -## Rejected approaches - -- Marking a capped attempt successful: advances over unread content. -- Keeping `since` but repeatedly taking the same prefix: does not finish. -- Counting only writes while retaining the fixed first ten pages: stalls on a - larger result set and increases GitHub detail requests without a bound. -- Advancing from maximum `doc_time`: custom has no source-update clock, and ties - can span a batch or page boundary. -- Raising or removing the limits, or disabling incremental sync permanently: - changes the resource contract instead of implementing continuation. -- Hiding progress in `config`, error strings or job payload history: exposes or - overloads unrelated state, and makes retention and configuration changes - determine correctness. - -## Acceptance - -Reuse the existing wiremock and database fixtures with required PostgreSQL and -pgvector. Cover 201 records, several batches, more than each reader's ten-page -limit, a previously imported prefix, equal update timestamps, failure after -committed items, and changes during multiple batches. Assert actual `since`, -JQL and page parameters, completed checkpoints, resource budgets, stable -document identities, version counts and processing-job counts. Exercise every -changed provider branch, including boundary drift and crash replay. - -This record is a proposal. No continuation schema or production implementation -has been written or represented as accepted. diff --git a/docs/decisions/0072-a-source-checkpoint-ends-at-a-timestamp-boundary.md b/docs/decisions/0072-a-source-checkpoint-ends-at-a-timestamp-boundary.md new file mode 100644 index 000000000..7438883ee --- /dev/null +++ b/docs/decisions/0072-a-source-checkpoint-ends-at-a-timestamp-boundary.md @@ -0,0 +1,132 @@ +# A source checkpoint ends at a timestamp boundary + +Proposed 2026-10-19 (#1212, #1213); open: implementation and regression coverage. + +## Context + +Source syncs can stop after 200 items and still record success. The next run +uses that successful run's start time as its lower bound, so it can skip items +that were never imported. [Issue #1212](https://github.com/deeplethe/utopia/issues/1212) +reproduces this with a custom response containing 201 items. + +A checkpoint must describe the data consumed. For GitHub and Jira, an item's +source update time provides that position, but a timestamp cannot identify a +position inside a group of items that share the same time. + +The [reader probes](https://github.com/deeplethe/utopia/issues/1212#issuecomment-6183423449) +show why that distinction matters. With one shared update time, GitHub's +1,001 items span eleven 100-item pages; Jira's 501 items span eleven 50-item +pages. A fixed ten-page limit repeatedly returns the same prefix. Keeping the +timestamp repeats that prefix; advancing past it loses the unread tail. + +## Decision + +Use a timestamp watermark: the latest source update time whose entire group +has been consumed. **A GitHub or Jira run stops only at a timestamp boundary.** +This follows the [maintainer's decision](https://github.com/deeplethe/utopia/issues/1212#issuecomment-6261482074). + +### GitHub and Jira + +Read items in ascending update-time order. Count the existing 200-item +processing budget and ten-page listing budget as before. When either budget +is exhausted inside a group, continue reading and processing until that group +is complete. Stop before consuming the next timestamp group, then save the +completed group's time as the checkpoint. Reading ahead to identify the next +group does not make that group consumed. + +For example, if item 200 belongs to a group ending at item 240, consume all +240 items and leave the next group for the next run. The 1,001-item GitHub +and 501-item Jira groups above must each arrive in full on the first run. + +The next run starts logically after the completed group. Map that boundary +to each provider's query semantics: GitHub's `since` is exclusive, while the +current Jira query uses `>=` and formats its lower bound to a minute. Query +overlap caused by provider precision must neither skip newer items nor trap +the run in an already consumed prefix. Test the actual query parameters and +returned timestamps rather than assuming the providers share one boundary. + +Store the completed timestamp on the source, independently of retained sync +audit rows. Do not derive it from a run's start or finish time, or advance it +to the clock time merely because a query is empty. A failed run leaves the +previous checkpoint intact. Retry already committed items through the existing +document, version and processing-job deduplication. + +There is no persisted page, offset, fetched response or continuation state. +An unusually large timestamp group makes one run longer. Do not add a separate +group cap or a group-size alert; use the existing rate-limit backoff. The +exception applies to source listing and processing, not permission to replace +a GitHub document with a truncated comments or events history. + +### Custom + +The endpoint already returns the whole requested response. Walk its items in +order, charging only `Created` and `Updated` outcomes against the 200-item +budget. `Unchanged` items do not consume that budget. Mark the batch incomplete +only when processing stops with items left, and retain the previous checkpoint. +Retries can then pass the unchanged prefix and reach the remaining items. + +Once the whole response, including its deletions, is processed successfully, +advance the checkpoint to that run's start time, preserving the existing +protection for changes during the run. Keep the committed counts visible for +incomplete runs. No source-update field, pagination token or saved response is +added to the custom protocol; `doc_time` remains document time. + +The custom fix is being reviewed in +[PR #1246](https://github.com/deeplethe/utopia/pull/1246). It does not resolve +the GitHub and Jira parts of #1212. + +## Recovery and tradeoffs + +Old successful runs cannot prove that their windows were complete. On upgrade, +clear the checkpoint once for these three source kinds and recover from the +beginning using the same rules. Preserve documents, versions, jobs and audit +history; deduplication keeps unchanged content from creating new versions. +Resume incremental reads afterward. + +This design trades a hard per-run bound for progress through a timestamp group. +It does not make a provider's paginated result an atomic snapshot. Concurrent +updates and failures still need coverage before the implementation can claim +that checkpoints never cross unread data. + +## Rejected approaches + +- **Keep the old checkpoint but always take the first 200 items.** Retries + never reach the tail. This is why custom must count changed items only. +- **Use a timestamp with an unconditional ten-page limit.** The equal-time + examples above still stall. Completing the group removes that limitation. +- **Persist continuation state or a full response snapshot.** The original + proposal stored pages and consumption positions in `sources.sync_state`. + Finishing each timestamp group avoids that storage and recovery machinery. +- **Resume from a saved offset, even in creation order.** Deletions, transfers + and permission changes can move page boundaries. Keeping boundary identities + and restarting scans adds complexity the chosen design does not need. +- **Advance to the run's start time after partial work, or use custom + `doc_time`.** Neither proves that the remaining source changes were consumed. + +Revision 2026-10-19: the original draft required hard budgets and persisted +continuation. The maintainer accepted longer runs for equal-time groups, so +this revision replaces that design and moves the record from reserved 0067 +to 0072. The suggested 0071 was subsequently taken by +[PR #1319](https://github.com/deeplethe/utopia/pull/1319). + +## Acceptance + +Reuse the real readers and existing HTTP/database fixtures. The implementation +must demonstrate: + +- GitHub's 1,001-item and Jira's 501-item equal-time groups are fully imported + in the first run, beyond ten pages. +- Exhausting either budget completes the current group but leaves the next + group for the next run, with the correct persisted checkpoint and queries. +- GitHub's exclusive boundary and Jira's minute-level query overlap preserve + progress without omissions or duplicate versions and processing jobs. +- A failed page fetch, detail fetch or write cannot advance the checkpoint; + retry recovers the whole group. Updates during pagination remain reachable. +- Custom's 201 changed items finish on the second run; 450 finish over three + capped runs. Cover an unchanged prefix and no custom update field. +- The one-time recovery scan preserves existing data and returns to incremental + sync. GitHub replacements retain complete discussion content. + +Database assertions must run with PostgreSQL/pgvector and +`UTOPIA_TEST_REQUIRE_DB=1`. This ADR and its index are the only changes in #1213; +the production fix and these regression tests follow separately. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index f82d381f1..921e43c0e 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -112,7 +112,7 @@ By domain; the domains are the files of [../design/](../design/README.md). **Sta | 0052 | [Document content is a read contract over the retained ledger](0052-document-content-is-a-read-contract.md) | Proposed | | | 0054 | [A source may push statements in the open contract](0054-a-source-may-push-statements-in-the-open-contract.md) | Implemented | | | 0065 | [The OCR reader chooses its provider](0065-readers-choose-their-provider.md) | Implemented | | -| 0067 | [A source checkpoint follows a complete window](0067-a-source-checkpoint-follows-a-complete-window.md) | Proposed | | +| 0072 | [A source checkpoint ends at a timestamp boundary](0072-a-source-checkpoint-ends-at-a-timestamp-boundary.md) | Proposed | | ### [lakehouse-and-actions](../design/lakehouse-and-actions.md) From 075604d7f4b766cb94974c5e82c7829d0396f1db Mon Sep 17 00:00:00 2001 From: Floating-Y <118035379+Floating-Y@users.noreply.github.com> Date: Tue, 20 Oct 2026 10:37:00 +0800 Subject: [PATCH 3/3] Clarify how source syncs advance past a completed timestamp group Signed-off-by: Floating-Y <118035379+Floating-Y@users.noreply.github.com> --- ...checkpoint-ends-at-a-timestamp-boundary.md | 41 +++++++++---------- 1 file changed, 19 insertions(+), 22 deletions(-) diff --git a/docs/decisions/0072-a-source-checkpoint-ends-at-a-timestamp-boundary.md b/docs/decisions/0072-a-source-checkpoint-ends-at-a-timestamp-boundary.md index 7438883ee..d0f0bbbc9 100644 --- a/docs/decisions/0072-a-source-checkpoint-ends-at-a-timestamp-boundary.md +++ b/docs/decisions/0072-a-source-checkpoint-ends-at-a-timestamp-boundary.md @@ -27,23 +27,24 @@ This follows the [maintainer's decision](https://github.com/deeplethe/utopia/iss ### GitHub and Jira -Read items in ascending update-time order. Count the existing 200-item -processing budget and ten-page listing budget as before. When either budget -is exhausted inside a group, continue reading and processing until that group -is complete. Stop before consuming the next timestamp group, then save the -completed group's time as the checkpoint. Reading ahead to identify the next -group does not make that group consumed. +Read items in ascending update-time order. Skip items at or before the +checkpoint before counting either budget. When the 200-item processing budget +or ten-page listing budget is exhausted inside a group, continue reading and +processing until that group is complete. Stop before consuming the next +timestamp group, then save the completed group's time as the checkpoint. +Reading ahead to identify the next group does not make that group consumed. For example, if item 200 belongs to a group ending at item 240, consume all 240 items and leave the next group for the next run. The 1,001-item GitHub and 501-item Jira groups above must each arrive in full on the first run. -The next run starts logically after the completed group. Map that boundary -to each provider's query semantics: GitHub's `since` is exclusive, while the -current Jira query uses `>=` and formats its lower bound to a minute. Query -overlap caused by provider precision must neither skip newer items nor trap -the run in an already consumed prefix. Test the actual query parameters and -returned timestamps rather than assuming the providers share one boundary. +The next run consumes only items updated strictly after the checkpoint. +GitHub's `since` is inclusive in practice, and Jira's `updated >=` bound is +rounded to the minute, so either provider can return items already consumed. +Skip every returned item at or before the checkpoint; these items count against +neither the 200-item processing budget nor the ten-page listing budget. +Pages containing only skipped items therefore cannot exhaust the listing +budget. For GitHub, `since` can be the checkpoint plus one second. Store the completed timestamp on the source, independently of retained sync audit rows. Do not derive it from a run's start or finish time, or advance it @@ -103,12 +104,6 @@ that checkpoints never cross unread data. - **Advance to the run's start time after partial work, or use custom `doc_time`.** Neither proves that the remaining source changes were consumed. -Revision 2026-10-19: the original draft required hard budgets and persisted -continuation. The maintainer accepted longer runs for equal-time groups, so -this revision replaces that design and moves the record from reserved 0067 -to 0072. The suggested 0071 was subsequently taken by -[PR #1319](https://github.com/deeplethe/utopia/pull/1319). - ## Acceptance Reuse the real readers and existing HTTP/database fixtures. The implementation @@ -118,8 +113,11 @@ must demonstrate: in the first run, beyond ten pages. - Exhausting either budget completes the current group but leaves the next group for the next run, with the correct persisted checkpoint and queries. -- GitHub's exclusive boundary and Jira's minute-level query overlap preserve - progress without omissions or duplicate versions and processing jobs. +- After importing the 1,001-item group at time T, the next run skips any + returned items at or before T without charging either budget, imports a + later group and advances the checkpoint past T. +- GitHub's inclusive boundary and Jira's minute-rounded query bound preserve + this progress without omissions or duplicate versions and processing jobs. - A failed page fetch, detail fetch or write cannot advance the checkpoint; retry recovers the whole group. Updates during pagination remain reachable. - Custom's 201 changed items finish on the second run; 450 finish over three @@ -128,5 +126,4 @@ must demonstrate: sync. GitHub replacements retain complete discussion content. Database assertions must run with PostgreSQL/pgvector and -`UTOPIA_TEST_REQUIRE_DB=1`. This ADR and its index are the only changes in #1213; -the production fix and these regression tests follow separately. +`UTOPIA_TEST_REQUIRE_DB=1`.