Skip to content

v0.49.0: Qualify escalation safety, recovery and operating limits #12

Open
Open
v0.49.0: Qualify escalation safety, recovery and operating limits#12

Description

@grove
grove
opened on Oct 17, 2026

Owner acceptance amendment — 2026-10-28: The frozen S1 candidate throughput floor is 40 cases/minute; 60/minute remains the offered workload and target. This supersedes the 2026-10-25 45/minute and 2026-10-22 59/minute floors for throughput only. The exception applies only when throughput is the sole raw harness failure, the raw exit remains visible, all 300 admissions and complete source/case/receipt vectors are present, zero future receipts exist, the paired control passes and every other correctness, latency, resource, permission and supervision gate passes. This does not approve the operating envelope, final pg-react stack, restore/clone or release.

Owner acceptance amendment — 2026-10-25: The requester has superseded the prior 59 cases/minute minimum with 45 cases/minute for the frozen S1 candidate throughput-only exception. 60/minute remains the offered workload and target. Original raw exit/failure evidence, 300 admissions, complete source/case/receipt vectors, zero future receipts, due-burst p95 <=150s/max <=180s, storage <10 GiB, permissions, supervision, exact-image/current-source paired control and all other qualification gates remain mandatory. This amendment does not sign the separate React operating envelope or authorize release. See React PR #14 policy commit.

2026-10-22: owner-approved S1 throughput acceptance amendment

The requester explicitly authorized replacing the binary 60/minute candidate minimum with 59 cases/minute, while retaining 60/minute as the offered workload and performance target. A measured 59–<60/minute result is acceptable only if the frozen workload has 300 admissions, complete case/source and receipt vectors, no future receipts, and throughput is the sole raw harness failure; the original raw exit/failure remains visible. Less than 59/min or any unrelated correctness, latency, memory/storage, supervision or authorization failure is not accepted. Full current-source image and approved 10-CPU ARM64 (or separately approved equivalent) paired requalification is still necessary. Old runs cannot be attributed to new source commits. This narrow policy approval does not supply the full signed React v0.49 operating envelope or approve restore/clone or production activation.

Part of #8.

Release dependencies:

Outcome

Qualify the final managed-escalation stack against all T49.01-T49.12 cases and common gates C01-C12, and document the exact supported operating profile and recovery procedures. Parent #8 remains open until implementation, performance, owner inputs and qualification are complete.

The previously measured local React candidate passed 2,609 unit tests and 38 focused runtime executions but delivered 14.83 cases/minute, below the now-approved 40/minute floor; its exact control deadlocked on admission 3, leaving that local pair incomplete. Historical upstream S1 runs #40789839577 on head 2b9f4fb30847f0a4cbed1ca5e5ea1e4276fb9e18 and #40824898024 on head ddd45afa01ba49438793edb88061307a8e94c086 passed the 40 floor at 44.689881605 and 45.0132284876 cases/minute respectively, with complete vectors and zero future receipts. PR #1232 has since advanced to 822dd6227f1e7de6e35bea13993163ab738a4f16. Current S1 run #40836093714 reports PASS_OWNER_APPROVED_FLOOR: 44.6131296619 cases/minute versus corrected control 23.8066196538, 300 admissions, 1,200 source rows, 300 case vectors, zero future receipts, due p95/max 120.882/126.348 seconds, and 3,398,332,113 conservative storage bytes. Raw candidate exit 1 remains visible against the 60/minute target. The workflow qualified a 4-CPU ARM64 profile, but this issue requires approved 10-CPU ARM64 or a separately approved equivalent; reconcile that before treating the result as full qualification. Current CI run #40836093812 failed the restore lifecycle exact-action assertion (REINITIALIZE observed, DIFFERENTIAL expected) in both the sensitive E2E and packaged receipt E2E, and failed test_multi_cycle_stddev_algebraic against the exact oracle. A focused local restricted-role audit on React revision 90c7e17bb060a85166b6830394d1f0f0a7c7089a passed React recovery but blocked pg-trickle capture recovery with PGT_EXT_GRAPH_INVALID because stored relation OID 41149 did not exist; failed recovery left state unchanged and release_qualified=false. That local image uses pg-trickle 0.108.3, not the current PR head. Current-head source review of PR #1232 at 822dd6227f1e7de6e35bea13993163ab738a4f16 confirms recover_capture_instance() still delegates to adopt_database_instance(), which reads stored pgt_relid and resolves it through StreamTableMeta::get_by_relid(); the current PR does not fix this stale-OID path. The prior-head pause failure #40789839749 remains historical. Both failing tests passed in preceding CI #40824898084 on head ddd45afa01ba49438793edb88061307a8e94c086 (lifecycle at 520/528; stddev at 310/538). The ddd45 to 822dd diff changes snapshot restore but excludes src/dvm/operators/aggregate.rs and tests/e2e_multi_cycle_tests.rs, so the stddev failure cause is unisolated. These results do not qualify the final React stack. Linked upstream issue #1168 is closed and its description now records the 40/minute floor.

Owner inputs and operating envelope

  • The requester delegated selection of the separate v0.49 supported operating envelope on 2026-10-29. The intended limits are 300 admissions over five minutes at 60/minute offered plus up to 1,000 simultaneously due cases (up to 1,300 distinct if the sets do not overlap); one-second polling; at most 32 claims per cycle; one next level per evaluation through level 2; delivery p95 <=150 seconds and maximum <=180 seconds; dispatch and backlog drain <=180 seconds; outage <=120 seconds and recovery <=180 seconds; peak sampled memory <4 GiB; WAL <=4 GiB; <=10,000,000 total row inserts, updates and deletes per run; supervised storage <10 GiB; no duplicate, unauthorized, stale-binding or preactivation actions; and request-key, canonical-body and applied-level receipt retention for the occurrence lifetime plus at least 90 days with coherent backups; retain associated retry/audit evidence with each request and receipt. The 40/minute value is only the narrow qualification floor, not a production rate guarantee. Named production cohort and workload-specific service policies remain v0.50. These are selected limits to qualify, not achieved measurements or approval to activate or release; exact-stack qualification remains open.
  • Throughput target remains 60/minute at the unchanged offered workload; the owner-approved 40/minute acceptance floor is limited to the sole throughput-rate failure with complete otherwise-passing evidence. The 150/180-second due-burst p95/maximum limits and below-10-GiB run budget apply to the selected envelope above. This throughput-only amendment accepts no other measured shortfall; it is separate from the supported envelope.
  • Obtain the remaining escalation decision/receipt fixtures and the approved next-level, occurrence/applied-level and catch-up semantics through implementation issue v0.49.0: Complete managed bounded escalation #11. Outage/recovery qualification remains an explicit blocker.
  • Freeze and record the exact joint artifacts, public capability/API inventory, roles/bindings and actual adjacent upgrade predecessor. Recheck compatibility admission after artifact or capability changes.

Qualification work

  • Retain deterministic before/equal/after time vectors and real managed-worker idle-case delivery without business-source edits or manual refresh/time calls. Verify exact requests, receipts, controls, deadlines and complete case coverage.
  • Cover cross-key per-occurrence/level uniqueness, genuine recurrence, retries, crash/uncertain-commit replay, atomic rollback, concurrent claims, human-change races, pause/replacement barriers and fail-closed domain outcomes.
  • Verify bounded backlog/catch-up under the supported mixed workload and a real outage across deadlines. Record every case's progress, due/accepted times, queue behavior and resource costs; missed polls must not generate repeated actions.
  • Qualify fresh installation, populated adjacent upgrade and injected upgrade failure. Preserve request codecs, attempted work, deadlines, bindings and applied-level state; upgrades must not silently enable escalation.
  • Qualify restart and coherent whole-stack restore, including occurrence mappings, applied-level history, requests and matching MDM receipts, followed by deliberate capability/binding recheck and bounded resumption.
  • Qualify logical and physical clone startup with copied overdue state and effects blocked until an environment-specific operator activation check succeeds. Copied enabled flags and database OIDs alone are insufficient.
  • Run least-privilege, denied/cross-scope and hostile-search-path cases, optional-adapter/MDM-absent core compatibility, ordinary API/stale-review compatibility, and complete read-only comparison coverage with no state mutation.
  • Demonstrate the full public lifecycle from ambiguous case through queue/deadline control, idle escalation, human decision and separate publication/closure. Automation must not record the human decision or treat delivery as resolution.

Completion criteria

  • The signed numeric profile and all owner fixtures are recorded; no required owner input remains pending.
  • Throughput issue perf(refresh): meet the S1 graph-refresh throughput gate pg-trickle#1168 and implementation issue v0.49.0: Complete managed bounded escalation #11 meet their completion criteria on the final source/artifact set.
  • T49.01-T49.12 and C01-C12 have explicit expected/actual evidence with no required skips or omissions. Assertions compare complete rows, canonical bytes, receipts and durable-state vectors, with dynamically assigned values checked against public authoritative sources.
  • Unauthorized, repeated, stale-binding and preactivation clone controls are zero. Timing, admission, backlog and resource measurements satisfy the signed limits.
  • Upgrade/failure recovery, restart/outage catch-up, coherent restore and logical/physical clone isolation pass on the intended release artifacts.
  • Operator documentation covers the qualified temporal API, limits, least-privilege setup, pause/activation, timer behavior, retention and recovery. Release notes and executable examples match the evidence.
  • Independent review covers the exact release candidate with no open critical correctness or authorization finding. Record findings and dispositions using ordinary review practices.
  • Machine-readable evidence binds source, environment, commands, logs, failures and expected/actual results to the exact artifacts intended for publication. Tests of earlier images are not substituted for changed candidates.

References and boundaries

Authoritative specification: https://github.com/trickle-labs/pg-react/blob/843256fb465de69f172fb61e66c5f76d981f0670/plans/v0.49.0.md, especially sections 5, 7 and 8 and the referenced common release gates.

Existing local evidence: pg-react work/r53-capped-deep-join-20261017/results.txt, validation-summary.json, source-manifest.json and candidate/control qualification observations. Preserve failures and historical files; unchanged candidate images are not rerun merely to regenerate a review or acceptance claim. This issue uses ordinary test/review practices and does not invoke Promise-to-Proof workflows.

Use the existing runtime and release machinery and preserve the parent scope and 30-person-day budget, or record an explicit approved re-estimation/split. This tracker does not authorize production activation, publication or merge.

Previous pg-trickle PR #1232 head 28321 status — 2026-10-29

At this earlier 09:03 UTC snapshot, PR #1232 was at head 28321dbceb67d5a4f894792230b5f45651c9302e, CI had 15 checks passed and 12 pending, and S1 had no candidate result yet. The prior 44.6131296619 cases/minute result belongs to head 822dd6227f1e7de6e35bea13993163ab738a4f16; it is not transferable. The current head changes snapshot restore and the aggregate implementation plus multi-cycle test, so those older restore/stddev failures remain historical; completed evidence for head 28321 is below, and the newer PR head is tracked separately at the end.

Exact-head source inspection at 28321dbceb67d5a4f894792230b5f45651c9302e confirms recover_capture_instance() still delegates to adopt_database_instance(), which reads stored pgt_relid values and resolves them through StreamTableMeta::get_by_relid(). The exact-head S1 profile gate labels a 4-CPU ARM64 runner QUALIFIED_PROFILE because it enforces only a 4-CPU minimum; the final enforcement step is skipped when that gate passes. This does not satisfy the issue’s 10-CPU ARM64 requirement, and no equivalent approval is recorded. The local React restore audit remains limited to pg-trickle 0.108.3 and reported stale relation OID 41149; it is not evidence for the new PR artifact.

The exact-head test_pg_dump_restore_recreates_oid_shifted_dag() calls recover_capture_instance() after logical restore but does not create an output-delta consumer. register_output_delta_consumer() is the only writer of pgt_output_delta_logs, while adopt_database_instance() reads st.pgt_relid only for rows in that log table. With no log row, this restore test never reaches the stale-OID lookup. Exact-stack restore evidence therefore needs an output-delta consumer present before dump/restore.

For head 28321dbceb67d5a4f894792230b5f45651c9302e, exact-head CI #40845740212 completed successfully with 28 checks passed and no failures. The exact-image S1 #40845740073 completed with candidate PASS_OWNER_APPROVED_FLOOR: 45.4096859597 cases/minute against a 24.0137302505 corrected control. The run records 300 admissions, 1,200 source rows, 300 case vectors, zero future receipts, due p95/max 120.151098/125.563445 seconds, 3,418,323,740 conservative storage bytes, and 1,911,705,600 peak-memory bytes. The candidate raw harness exit 1 is retained against the 60/minute target; the 40/minute exception applies because the result is at least 40 and other checked candidate gates passed. The ARM64 candidate image digest is sha256:2a4042a8ac334ba4924b0987c626d5cf5a2b609cad4911a0554b8c5d7d9afc14, built from commit 28321dbceb67d5a4f894792230b5f45651c9302e with MDM eef7937180d504e62ce1076074bbeed8cef5d5a7 and React f19d9b2c5f3d947705ea26aa1619a989f08ae1dc. That result passes only the throughput exception; the 10-CPU-or-approved-equivalent profile and final-stack recovery gates remain open.

Current pg-trickle PR #1232 status — 2026-10-29

PR #1232 merged into main on 2026-10-29 at 11:01 UTC as merge commit 75f2d1f590ea2aa029b18b8ee91332f719b14057, from head 44d4b33fba3e9ededec24718c02b7b43e0961d9a. Final checks show 28 passed, zero failed, and 18 skipped.

Exact-image S1 run #40853118615 completed with candidate PASS_OWNER_APPROVED_FLOOR: 44.883911 cases/minute against corrected control 23.687264/minute. The run recorded 300 admissions, 1,200 source rows, 300 case vectors and zero future receipts; due p95/max were 87.614/93.084 seconds; conservative storage was 3,297,604,592 bytes and peak memory 1,834,786,816 bytes. The raw candidate exit 1 against the 60/minute target is retained; throughput is the only failed candidate observation, and the 40/minute floor is met. Candidate image digest: sha256:f169306d316341e4c525ac0f1f46f23af47d53f6e663cd91d137597005389d92; corrected control image digest: sha256:7d813ec5b71a80a1bb3b114142513ab799bfc50e078db1625a59363a3d8a33f0.

The profile gate recorded QUALIFIED_PROFILE at 4 Docker CPUs with a 4-CPU minimum. This does not satisfy the required 10-CPU ARM64 profile or an approved equivalent. The workflow selects ubuntu-24.04-arm; GitHub documents the standard public ARM64 runner as 4 CPUs and offers 16-CPU Linux ARM64 larger runners, configured by organization or enterprise owners and granted to repositories through runner groups (standard runner specs, larger-runner specs and access, larger-runner management). The repository-scoped self-hosted runner inventory is empty. The current GitHub context cannot inspect organization runner groups (the organization endpoint returns HTTP 403 for this identity), so access to a larger-runner group remains unknown and needs an organization administrator to verify. A 16-CPU ARM64 runner is a viable profile if the organization configures and grants this repository access.

The merged diff since 28321dbceb67d5a4f894792230b5f45651c9302e changes only tests/e2e_wal_receipt_recovery_tests.rs; src/api/recovery.rs, src/api/output_delta.rs and tests/e2e_pg_dump_tests.rs are unchanged. Adoption still resolves stored output-delta pgt_relid values, while the logical-restore test has no output-delta consumer and does not reach that lookup. Exact-stack stale-OID recovery remains unqualified; the 10-CPU profile and remaining #12 release gates remain open.

2026-10-29 output-delta restore follow-up

Follow-up pg-trickle PR #1288, at head 030466d8c996e04536f144f356edbb067d618eac, addresses stale output-delta stream and payload relation OIDs after logical restore. The producer's clean-image targeted restore E2E passed (1 passed, 3 skipped), and just lint passed. Current PR checks are 25 passed, 0 failed, 18 skipped, with none pending; the PR is open with review requested and no review recorded.

This is candidate producer evidence, not final joint-stack qualification. The prior React audit used pg-trickle 0.108.3. Recovery must still be qualified downstream with the merged producer on the intended React artifacts; the 10-CPU-or-approved-equivalent profile and other #12 release gates remain open.

The separate local React restore probe currently seeds HYSTERESIS only as an internally created REMOVED rule and labels its scope recovery mechanics only. It verifies OID rebinding but does not exercise an active temporal rule after restore. This observation is from uncommitted local changes based on React 90c7e17; it is a coverage gap, not a demonstrated defect or merged-artifact result.

Owner-delegated supported-envelope selection — 2026-10-29

The requester delegated numerical selection of a separate v0.49 operating envelope. Reuse the approved qualification profile's numeric limits as the intended support ceilings, applied together to a mixed workload of 300 admissions over five minutes at 60/minute offered plus up to 1,000 simultaneously due cases (at most 1,300 distinct cases if the sets do not overlap). Poll every second, claim no more than 32 cases per cycle, and permit at most one next level per evaluation through level 2. Due-burst latency is p95 <=150 seconds and maximum <=180 seconds; detection/dispatch and backlog drain are each <=180 seconds. The supported outage is <=120 seconds and recovery <=180 seconds after recovery begins.

Per combined qualification run, peak sampled memory must remain below 4 GiB, WAL at or below 4 GiB, row writes at or below 10,000,000 and supervised storage below 10 GiB. Duplicate and unauthorized controls remain zero. Preserve original deadlines; restore and clone effects require explicit activation. Retain request keys, canonical bodies and applied-level receipts throughout each occurrence and for at least 90 days after it ends, with coherent backups across that horizon. The 40/minute value remains only the narrow throughput-only qualification floor; it is not a continuous production arrival or service guarantee.

These are the limits to qualify, not evidence of achieved capacity or authorization for production activation or release. The exact joint stack, mixed-backlog, resource, outage/recovery and restore/clone gates remain open.

Activity

  1. grove commented on Oct 22, 2026

    @grove
    grove
    on Oct 22, 2026 – with ChatGPT Codex Connector
    ContributorAuthor

    2026-10-22 — current v0.49 joint-source handoff, strict acceptance still blocked

    React draft PR #14 now includes MDM stale-runtime repair adoption and accepted focused real single/two-case outage/catch-up and public bounded-claim fixtures. React dependent PR #20 was merged into the feature branch as 6a5500899f99a38040d91cd9545709e24032a710 with exact-head green checks, proving two independent pending episodes are claimed through separate public max_items=1 calls, not full managed mixed-workload fairness. Draft PR #21 now verifies the installed worker function chain for pg_react.batch_size → least(batch_limit,100) → pgreact.claim(max_items), plus raw counter intervals. Earlier CI failed because it incorrectly inferred a single worker cycle from adjacent heartbeat times. That invalid test inference was corrected at 46b98a958898169297d7ece308764250f1501eaf; current exact-head CI is pending. Heartbeat intervals without cycle IDs are not per-cycle measurements.

    Independent pg-trickle performance gate #1168 is NOT satisfied. Newer pg-trickle PR #1191 retains a 10-CPU ARM64 candidate measured at 59.3637431605094 cases/min for 300 admissions, with raw benchmark exit 1. The unchanged minimum is 60/min. Its previously green S1 workflow #39731566295 ran on only 4 CPUs, explicitly skipped live rate qualification and accepted the historical 59.36 via an erroneous PASS_WITH_RATE_TOLERANCE code path. Dependent pg-trickle PR #1213 now removes that <60 tolerance and adds regression checks. Historical source/image/evidence remains retained unchanged. Current exact-source candidate speed >=60 and a complete corrected control on approved 10-CPU ARM64 remain required.

    Further blocking work remains the owner-signed numeric operating envelope, complete T49.01–T49.12/C01–C12 qualification on final artifacts, mixed-backlog/admission fairness and real multi-case outage budgets, coherent whole-stack restore, logical/physical clone activation isolation, documentation and release review. Do not close React #11/#12/#8 or approve PR #14 main merge/release based on code-review approval alone. No source/results threshold relaxation, no production activation.

  2. grove commented on Oct 25, 2026

    @grove
    grove
    on Oct 25, 2026 – with ChatGPT Codex Connector
    ContributorAuthor

    2026-10-25 dependency check after upstream merges

    Confirmed upstream change: pg-trickle #1191 was squash-merged into main as c7e257e551d6f6bacb84ecfe6f5daf7637c53673. The PR-head 232eaef913e4395315351d0a2d071d921e1021f4 had green PR correctness, S1 exact-image qualification, unit/integration, sensitive E2E and ASan checks. Its S1 record reports 54.6673/min on 10-CPU ARM64 and 47.1893/min on a 4-CPU runner, judged against that PR's 45/min floor. These are not evidence that this issue's independent >=59/min React v0.49 acceptance floor passed.

    Other observed dependencies: MDM #21 has merged (merge commit 5fdee89b589da8fb3ab86157cc716a2aaff8bdf8). React #14 remains open and ready for review at 9abb0f41ed20c1ee4e078d84d7ec9f8e3ced77dc; CI, pinned producer compatibility and MDM stale-runtime repair preview runs at that exact head are green. Preview success is not automatic release-pin adoption or complete qualification.

    Still open; no release or merge inference for React: pg-trickle #1168 remains open, with its distinct >=59/min requirement, original 60/min offered load, paired control, exact source/image identity and complete vectors. This issue also needs the signed v0.49 numeric operating envelope, full T49.01–T49.12/C01–C12 evidence, restore/clone isolation, outage, recurring-level safety and independent review.

    Next evidence-bearing step: align the React #14 pinned producer/MDM candidate with the merged upstream artifacts under explicit source/archive/shared-library manifests, retain the old pin compatibility proof, then run the frozen 300-admission joint-stack candidate/control and remaining recovery gates on those exact images. Do not transplant #1191's 45/min acceptance policy into React's 59/min gate or close #1168/#12 from green upstream CI alone.

  3. grove commented on Oct 26, 2026

    @grove
    grove
    on Oct 26, 2026
    ContributorAuthor

    PR #26, commit 2e36f59453c33b5d52828c801dbaa4d8d68dde29, fixes actual missing autonomous occurrence enrollment, with a failing-before/passing-after real MDM integration regression. See its gap matrix and evidence manifest.

    Passed locally: repeated fresh-database autonomous recurrence/cold restart with exact durable vectors, final non-superuser inspection assertions, 8 Rust unit tests, formatting, clippy, documentation/workflow audits, API inventory and source/assembly/evidence checksum verification. Original occurrence receipts remain intact; no duplicate control was observed. Approved profile, workload, pins and activation defaults are unchanged.

    Required failures remain visible: the unchanged full joint audit stopped in upstream MDM bootstrap at /tests/e2e.sql:3544, with server logs recording a backend killed by signal 9. Two original-image attempts and a separately identified lower-memory diagnostic also failed there. The complete core suite passed correctness/concurrency/DDL-lock-order/M54/M34/M31 authorization and RLS, then the unchanged M59 benchmark rejected this aarch64 Docker host; subsequent upgrade stages were not reached. Raw exits, logs and available observations are archived, not reclassified. The focused lane uses the original memory settings and does not replace either full suite. PR native packaged CI remains pending.

    Keep #12, #11, parent #8 and pg-trickle #1168 open. Both claimed-work transaction orders (including the MDM pause-barrier question), logical restore/clean physical clone and same-OID activation guarantees, complete operator diagnostic coverage, exact-current paired 300-admission qualification, 1,000-case burst/fairness, numeric resource/recovery/retention limits and independent review remain. #25 pending-work/physical recovery evidence is preserved. This is an implementation improvement, not complete v0.49 qualification, throughput acceptance or production readiness; remaining work is not established as qualification-only.

  4. grove commented on Oct 26, 2026

    @grove
    grove
    on Oct 26, 2026
    ContributorAuthor

    Two confirmed implementation defects are addressed by reviewable linked PRs:

    • MDM #22, 40e9cb7b9018627e2ecb17ca387017950d438c31: original MDM acknowledged an authorized pause while a real claimed React delivery waited on its case row, then applied ESCALATE after that acknowledgment. The shared repair serializes state transitions on the binding and takes delivery's binding lock before its separate fresh eligibility read. The independent SQL regression fails on the original binary and passes on the repair; frozen generated SQL remains byte-identical.
    • React #27, 1f9f39254e12bb07721e9e42918c9f993a93ef4d: an isolated current-main checkout with fixed MDM reproduced a retained BINDING_PAUSED hold blocking genuinely fresh work after authorized resumption. The repair keeps old request/receipt/attempt/hold bytes intact, permits only fresh work passing current eligibility, and distinguishes the old held pause from the new application in public inspection.

    The real native typed-callback regression passes seven named gates: delivery-first pause barrier, pause-first runtime freshness, committed pause before claim, human deadline/level edits in both claim orders, binding replacement lowering limits in both claim orders, fresh resumption and cross-key applied-level denial. Complete canonical request/receipt and case/source/work/attempt vectors, authorized reader diagnostics and healthy later managed cycles are asserted. Independent fresh-database repeats passed. The merged autonomous-recurrence/cold-restart lane still passes.

    MDM formatting, all-target clippy/Rust tests, cargo-pgrx test command, security/upgrade checks and frozen SQL comparison passed locally; React's 8 Rust tests, formatting/clippy, documentation/workflow audits and actionlint passed. The unchanged full local joint audit again stopped at upstream /tests/e2e.sql:3544, with a backend killed by signal 9. Its raw failure and both failing-before defects are retained in the React evidence manifest and MDM evidence.

    Native packaged CI remains pending. The new preview pins the exact MDM repair but does not adopt it as the release pin; the original pinned compatibility workflow, workload/profile limits and activation defaults are unchanged. Keep #11/#12/#8 open. Exact merged dependency adoption, broader uncertain-commit/overlapping replacement and privilege-revocation cases, logical/clean physical clone and same-OID activation guarantees, remaining diagnostics, paired workload/resource/recovery/retention qualification and independent review remain. No release, merge or production activation claim; remaining work is not established as qualification-only.

  5. grove commented on Oct 26, 2026

    @grove
    grove
    on Oct 26, 2026
    ContributorAuthor

    Merged MDM adoption and recovery follow-up

    Draft React PR #28, head b8e90d6c6f8dc1869bd781e1526dfd16756c8122, selects merged MDM 061705d17dae52697f6e7371b7840722a6ad4bde for the candidate and retains the original 6ad23ca11cb956f264e93e46a6351d32ea4a1ca8 compatibility lane separately. GitHub reports tree 95896f0286830fb873672676d6ac1b8e2c6baa9a for both the merged MDM commit and previously tested 40e9cb7b9018627e2ecb17ca387017950d438c31. Tree equality and the earlier green preview are not qualification of the merged artifacts. Exact-stack CI is pending; profile and pg-trickle pin/checksum are unchanged. Local format, 8 Rust tests, clippy, workflow syntax, docs and whitespace checks passed; actionlint was unavailable.

    Recovery remains an implementation question, not merely a qualification task. The existing pending physical recovery test pauses the binding before pg_basebackup and starts with the managed database list disabled; its empty runtime-table assertion does not establish safety of a clean filesystem clone with copied active runtime rows and matching database and role OIDs. A real same-OID, copied-enabled-state regression and any resulting repair are still needed; logical restore, full durable-history and bounded-resumption evidence remain open. No production activation or release is authorized by this update.

    Keep #11, #12, #8 and pg-trickle #1168 open. Final exact-artifact compatibility, privileges, race and recovery gates, paired 300-admission performance and resources under the approved profile, operating envelope, retention and independent review remain required.

  6. grove commented on Oct 26, 2026

    @grove
    grove
    on Oct 26, 2026
    ContributorAuthor

    Follow-up diagnostic: on the local pgreact:claimed-candidate PostgreSQL image, an unlogged row containing an enabled flag and the current database/role OIDs survived a clean Docker stop/start, and both OIDs still matched. This is a generic persistence probe, not a clone or MDM delivery regression, and does not by itself prove an exploitable delivery. It reinforces why the clean filesystem clone with copied runtime rows must be tested end to end before classifying recovery work as qualification-only.

  7. grove commented on Oct 27, 2026

    @grove
    grove
    on Oct 27, 2026
    ContributorAuthor

    Confirmed clean-clone preactivation delivery

    Draft PR #29, caf3838b7a1894eb6f83cc02a6051edda284df57, contains the real-MDM clean-filesystem-clone regression and retained before/after vectors and raw logs. Two independent runs against React main 96e07090be0da636445b5a27fd8d5e39cc5476f1, merged MDM 061705d17dae52697f6e7371b7840722a6ad4bde and pg-trickle c7e257e551d6f6bacb84ecfe6f5daf7637c53673 exited 1: the copied server started with the binding enabled and database/worker OIDs unchanged, and the real background worker applied an overdue ESCALATE before explicit activation. Both runs preserved one previously applied receipt and observed a second APPLIED_CONTROL. The manifest records archive, image and packaged-library hashes; the image ran amd64 under aarch64 emulation.

    This confirms an implementation defect. The draft does not fix it or qualify release. A repair must gate pending admission as well as final MDM delivery, preserve the retained occurrence/request/receipt history, recheck bindings and capabilities on authorized activation, and resume within the existing limits without changing deadlines. Logical restore, postactivation and failure/race tests remain outstanding. The earlier full-audit SIGKILL and harness failures remain in the raw archive. Keep #11/#12/#8 and pg-trickle #1168 open.

  8. grove commented on Oct 27, 2026

    @grove
    grove
    on Oct 27, 2026
    ContributorAuthor

    PR #29 now also isolates the producer boundary at d762d1940442b154504be57a20c921eb4dacdc9e. In two independent clean physical copies, with pg_react.databases empty, the original enabled binding and copied database/role OIDs, an authorized worker-role call to mdm_steward.submit_policy_intent returned APPLIED_CONTROL before reconciliation. The prior receipt remained present; complete before/after vectors and raw exits are in the PR evidence archive.

    This is a separate real MDM delivery-path defect: preventing only the React managed worker from claiming work cannot close it. Both MDM delivery authorization and React pending-work admission need fail-closed recovery behavior and explicit activation. The draft still contains reproductions only, not a fix; logical restore, activation, bounded resumption and release qualification remain open.

  9. grove commented on Oct 27, 2026

    @grove
    grove
    on Oct 27, 2026
    ContributorAuthor

    Real logical restore failure

    A disposable pg_dump -Fc and pg_restore --exit-on-error of the real two-occurrence pending-work fixture both completed on merged React 96e07090be0da636445b5a27fd8d5e39cc5476f1, merged MDM 061705d17dae52697f6e7371b7840722a6ad4bde and the unchanged pg-trickle pin. The exact recovery-state assertion then failed: MDM case and receipt vectors remained, while both React work rows and the prior React attempt were absent from the restored database. The database OID changed from 34156 to 42756. This is data loss, not a passing preactivation guard or evidence that no pending effect can occur. The focused failing regression and raw observations are being preserved on PR #29.

    An uncommitted physical-clone prototype has passed a focused preactivation, existing-receipt replay and authorized bounded resumption scenario, but it changes restart behavior and requires further safety, upgrade, logical restore, compatibility and current-artifact checks. No repair or release is claimed.

  10. grove commented on Oct 27, 2026

    @grove
    grove
    on Oct 27, 2026
    ContributorAuthor

    MDM draft PR #23, 330c2c36de4e7f360143c7a68a5688e7c1f05583, proposes binding policy activation to a fresh in-memory postmaster nonce. New requests on copied or restarted state fail before a receipt until explicit reconciliation; existing matching receipts replay. This requires pg_mdm preload and reactivation after every PostgreSQL restart. A dirty-source joint prototype passed the focused physical-clone preactivation, prior-receipt replay, unauthorized and stale-version rejection, producer-setting rejection and bounded pending-work resumption; its exact-commit native CI and populated upgrade remain pending.

    Independent logical-restore data loss is recorded in React draft PR #30, 4adf6d2dcc108bd2153804abb1033b370bc29ae1: two pg_dump/pg_restore runs lost React work and attempts even though MDM cases and receipts remained. This is not repaired by the activation candidate. Keep qualification and parent issues open. No release or production activation is claimed.

  11. grove commented on Oct 27, 2026

    @grove
    grove
    on Oct 27, 2026
    ContributorAuthor

    Recovery update: React draft PR #29 now at 00b9a2dfbc10866d448333581eee7c38508d1078 depends on MDM draft PR #23 at 74b5936d8cd6f1c21553913c3de2739eb20fb975. A new pre-fix regression showed reconciliation accepted an adapter escalation limit inconsistent with retained MDM policy. The candidate rechecks binding identity, roles, limits and producer capabilities; two dirty-source physical-clone probes rejected the mismatch, retained complete histories and resumed bounded pending work. A claimed-work pause/edit/replacement/cross-key matrix and autonomous cold-restart recurrence also passed locally. The archived evidence identifies the tested image and source files; it is not exact-current-head release qualification.

    React draft PR #30 separately demonstrates real logical restore loses React extension-owned work and attempt state while MDM history survives. This defect remains unresolved. MDM preload and explicit reconciliation after every restart are required by the candidate; all managed jobs in an affected database wait for enabled bindings to be reconciled. Native CI, full joint upgrade/restore, exact artifact gates, mixed workload, paired performance, operating envelope and independent review remain open. No throughput, release or production acceptance is claimed.

  12. grove commented on Oct 27, 2026

    @grove
    grove
    on Oct 27, 2026
    ContributorAuthor

    CI repair update: MDM PR #23 now at e8b1dff81f7375bc0c5a5d4eb1db311bce1c7043 upgrades the E2E foundation to 0.14.2 and revokes PUBLIC execution of the new internal activation-epoch function, granting it to the configured helper. The original local failure and raw logs are retained. Helper configuration must be rerun after upgrade.

    React PR #29 now at 4cc522c7f45a625ac013ddd7780701238bab615f preloads MDM for all candidate audit modes and adds a full joint candidate audit. The previous focused candidate CI passed on React 00b9a2dfbc10866d448333581eee7c38508d1078 and MDM 74b5936d8cd6f1c21553913c3de2739eb20fb975; new exact-head checks are pending. A local full-audit PostgreSQL connection loss remains archived without a claimed cause. Older MDM compatibility and preview lanes remain separate and fail closed because those revisions lack the activation epoch; they are not waived. Logical restore and all remaining release gates remain open.

  13. grove commented on Oct 27, 2026

    @grove
    grove
    on Oct 27, 2026
    ContributorAuthor

    PR #29 CI update: candidate workflow 40495668338 passed both repetitions of clean clone, direct delivery, claimed-work and autonomous recurrence against React 4cc522c7f45a625ac013ddd7780701238bab615f and MDM e8b1dff81f7375bc0c5a5d4eb1db311bce1c7043. Its full audit failed after a postmaster restart because it expected delivery before explicit reconciliation; that is a test setup failure, not qualification. The correction verifies the worker is blocked and histories unchanged before an authorized reconcile. The core-only compatibility test also failed to start because it was asked to preload unavailable pg_mdm; that setup is corrected.

    The local MDM E2E run on 49c00e965aa7e1ade37cd89e9b1795e16ef4b3ee reached incremental_qualification.sql:438 and exited 1 with a PostgreSQL deadlock. The raw log is retained in the React PR evidence; the cause is not yet established and no performance acceptance is claimed. Original-pin and other older MDM lanes remain separately visible and fail closed without the activation epoch.

  14. grove commented on Oct 27, 2026

    @grove
    grove
    on Oct 27, 2026
    ContributorAuthor

    React PR #29 now at c39d378b176f07046e8efe7cc06fd4faf0725011 retains the full diagnostic and prior failures. A local core-only compatibility run passed with pg_mdm.so removed. A local full joint run passed the human-edit, recurrence, pause, restart and physical-restore reconciliation/replay checks after the restored postmaster received its required preloads. Its test-source SHA, image and all observations are archived. The local producer was d8bb8eb5d45c5c9982e173e88914037433a1ee05, not the release pin, so this is diagnostic evidence only; exact-head pinned-stack CI remains required. Original-pin and earlier preview lanes remain separate and fail closed without the activation epoch. The MDM local incremental qualification deadlock and logical-restore defect remain open.

  15. grove commented on Oct 27, 2026

    @grove
    grove
    on Oct 27, 2026
    ContributorAuthor

    Candidate CI run 40523163879 on React PR head c39d378b176f07046e8efe7cc06fd4faf0725011 and MDM e8b1dff81f7375bc0c5a5d4eb1db311bce1c7043 passed both repetitions of physical clone, direct delivery, claimed-work and autonomous recurrence. The full audit exited 1 because it read the previous blocked worker status immediately after authorized reconciliation. PR #29 now waits for ready and a heartbeat, checks complete state, and retains the raw CI failure at head 83209c211393326f01eacc180781e1146bc728da. Current-head CI remains required; old MDM lanes without the epoch remain separately visible and unwaived. No release or throughput acceptance is claimed.

  16. 6 remaining items

  17. grove commented on Oct 28, 2026

    @grove
    grove
    on Oct 28, 2026
    ContributorAuthor

    PR #31 candidate CI 40655163798 passed at PR head 6eb4a2c09e5d0fff6846d9b41467de500d83a900; the actual React checkout was synthetic merge 4dc89c3af891017b1e89df7c42bf5f1a4fa03f25 (archive SHA-256 31b012981093d5f59433cab7aef3a000a693a6af1eb129f60a1659087892db7f). It used merged MDM ad2597f72ee657cb946c5bbd7ae5eb2bfd1ac1b3 (archive d5ef46335ac1e3a1c8255548f388e3b6949dbe8bfb7377fa8c790788754544aa) and unchanged pg-trickle source c7e257e551d6f6bacb84ecfe6f5daf7637c53673 (archive d6233d49cee80d33ac30626e0af7932a9698279233b8fceef3447275192abc27).

    Producer image sha256:05577a553f4f25fcbfad3ea68df3c373069b8bd3af8d7b54db49e113aa84de75, joint image sha256:17b52a0e70e191ec43d4c9ba75e57431c35c18c00215bc8b8243b4c7f9cd78a9; packaged SHA-256s: MDM 98b55d51d05e11bbdc26b240165c93f1a6e5dc2908ff11c1a14f339cd75e7ebb, pg-trickle 3f83ffdc01756e81b04785d173c7e8307327fc6bf09e019f8cc521dcb234fa58, React 34a46bbf37c3dcc327d156ae24663af44ca89f8bec0d634dd65f357b7711a4f5. Both repeated physical clone, direct-delivery, claimed-work and recurrence modes and their validation commands exited 0. The full audit exited 0 with 51 recorded passing checks and explicitly reported QUALIFICATION_INCOMPLETE. The downloadable CI artifact includes configuration, commands, exits, full observations and build logs. Historical nonblocking lanes do not qualify their underlying failures.

    The logical-restore defect in merged #30 remains unrepaired; the logical mode was not run in this CI. This is a correctness blocker, not qualification-only work. A future merged React artifact and all remaining #11/#12 gates need their own evidence; green PR checks do not accept performance or authorize production.

  18. grove commented on Oct 28, 2026

    @grove
    grove
    on Oct 28, 2026
    ContributorAuthor

    Repeated real logical-restore regression on the selected sources: React 6eb4a2c09e5d0fff6846d9b41467de500d83a900 (archive SHA-256 2db2f22446b0cbb7972a81e9ddf39e1ee7465ea179a65f263faef6ffcf766225), merged MDM ad2597f72ee657cb946c5bbd7ae5eb2bfd1ac1b3 (archive d5ef46335ac1e3a1c8255548f388e3b6949dbe8bfb7377fa8c790788754544aa), unchanged pg-trickle c7e257e551d6f6bacb84ecfe6f5daf7637c53673 (archive d6233d49cee80d33ac30626e0af7932a9698279233b8fceef3447275192abc27). Both fresh-container executions of audit.py IMAGE OUTPUT --logical-restore exited 1 with logical restore did not preserve the complete recovery state. Each completed actual pg_dump/pg_restore, with empty React extension dump configuration: two source rules, rule versions, activations, lifecycle events, and work items restored as zero, and one React attempt restored as zero. Complete MDM case, receipt, adapter request and attempt vectors matched. The assertion stops before the preactivation and subsequent activation/resumption checks; those remain unproven for logical restore.

    This is a local ARM64 diagnostic, not the earlier x86 CI image: joint image sha256:cc8d28e763b7ad5687b2038e5c87c9ffa649e4413c40386afc84add5ff8370d1; libraries SHA-256 MDM 02d7cb4db49051c588b1cabb220b63fb8fd28260e5ed0041b6acf0b533a50573, pg-trickle 9113108b8fc2eb1f68373c194c74763d008df6d22deba4f45f9c90044bb6b327, React 80f43edcd4d14bac6ad6e910652094bf8cfbfb835ba8515f98967972926894b1. The stock React builder has no ARM64 manifest (build exit 1); the local build used a separately identified ARM64 builder. An earlier emulated producer build was stopped (exit 130); the native producer and alternative React builds exited 0. The raw logs, complete observations, diagnostic Dockerfile and identity/command/exit manifest are retained locally in /private/tmp/pgreact-pr31-logical-arm64-evidence.tar.gz (SHA-256 e26aff27cdab4bb7d08155d197140bbf4bd531b71a391feaa240c42f2b81e6a1); this temporary path is not a durable GitHub artifact.

    No repair is claimed. The existing pgreact.rebuild_transient_metadata() validates and rebuilds some object references but cannot recreate omitted state; portable identity and ownership revalidation must also be addressed before activation. All exact-final-artifact qualification, workload and operating-envelope gates remain open.

  19. grove commented on Oct 28, 2026

    @grove
    grove
    on Oct 28, 2026
    ContributorAuthor

    PR #31 merged at c07e9d650c60977372236745c95e08cb300a728e; its green preview tested a synthetic merge, not this merge artifact.

    Draft PR #32, test commit 39e56203b1c53a4913b1c6ad29c2a0be757a526a, strengthens the logical-restore regression without weakening its full-state assertion. On two fresh local ARM64 source-matched diagnostic runs, complete MDM/adapter history matched, the binding guard passed, and two restricted managed cycles each returned {state: blocked, reason: MDM activation required, pending_jobs: 0, processed_jobs: 0} without changing the observed restored state. Both runs exited 1 at the original complete React recovery-state comparison because the dump lost pending React work. These empty-agenda cycles do not prove safety with retained overdue work, automatic worker startup, or authorized recovery/resumption.

    Complete local logs and observations, including intermediate failures, are retained at /private/tmp/pgreact-logical-guard-evidence.tar.gz (SHA-256 14e89c474ea6f06d694f18e242e2307021ad8539f03e6bdb41be6e573ef30728). This temporary path is not a durable GitHub artifact. Python compile initially exited 1 due to the protected default cache; the temporary-cache retry and diff check passed. The diagnostic image and source identities are recorded in the archive and previous issue update. No logical-restore fix, exact-merged-artifact qualification, performance acceptance or production activation is claimed.

  20. grove commented on Oct 28, 2026

    @grove
    grove
    on Oct 28, 2026
    ContributorAuthor

    PR #32 head 39e56203b1c53a4913b1c6ad29c2a0be757a526a is green. Its candidate run 40662881399 actually checked out synthetic React merge af0fbbe42f1c00426659b15cd93b9ed3fb84d5c8, archive SHA-256 0beb257fd50ae04c56979579de8a3ee3737d3c5203842037dcaf8b2199b5cb52, MDM ad2597f72ee657cb946c5bbd7ae5eb2bfd1ac1b3, and unchanged pinned pg-trickle. Joint image sha256:030b599faa1b82b74fa3adc4b888f08c389ae8eb7e6f7c951b15243def224102; packaged libraries MDM 98b55d51d05e11bbdc26b240165c93f1a6e5dc2908ff11c1a14f339cd75e7ebb, pg-trickle 3f83ffdc01756e81b04785d173c7e8307327fc6bf09e019f8cc521dcb234fa58, React 34a46bbf37c3dcc327d156ae24663af44ca89f8bec0d634dd65f357b7711a4f5. Repeated focused tests and the full audit exited 0; full audit recorded 51 checks and QUALIFICATION_INCOMPLETE. It did not run logical restore.

    An exact-merged-main candidate run 40684900145 is queued at c07e9d650c60977372236745c95e08cb300a728e. A separate logical activation diagnostic 40685124814 is queued at probe commit f95eabba7f1dafeb9cbf3467e6628c0e3de04086 on probe/v049-logical-activation; it attempts authorized reconciliation after actual logical restore, retains the full state comparison, exits nonzero on failure, and uploads logs and observations. It is an investigation, not a proposed repair.

    An initial local activation probe exited 1 before the test started because Docker reported its internal metadata filesystem read-only; raw log /private/tmp/pgreact-logical-activation-probe-first.log. No other containers or volumes were changed. Logical restore remains unrepaired, and activation/resumption with retained work remains unproven. Green CI does not mean performance or release acceptance.

  21. grove commented on Oct 28, 2026

    @grove
    grove
    on Oct 28, 2026
    ContributorAuthor

    PR #32 merged at 90c7e17bb060a85166b6830394d1f0f0a7c7089a. The focused candidate workflow passed on prior merged main c07e9d650c60977372236745c95e08cb300a728e in run 40684900145, React archive SHA-256 dafd8e9c99078003af67756bfc8f80d02f6501fa712ce840b2a00c8c1727547d, MDM merge ad2597f72ee657cb946c5bbd7ae5eb2bfd1ac1b3, MDM archive d5ef46335ac1e3a1c8255548f388e3b6949dbe8bfb7377fa8c790788754544aa, unchanged pg-trickle source c7e257e551d6f6bacb84ecfe6f5daf7637c53673 and archive d6233d49cee80d33ac30626e0af7932a9698279233b8fceef3447275192abc27. Producer image sha256:4553efcc15e7c77330b9cee4e0ce0e4d6901f762b94d4169fd0aedfeafa98ff4, joint image sha256:1465d62f46549e6cba60154320ee19a928fd0a38047ba05a8388f4a6eefbc9e0. Packaged SHA-256: MDM 98b55d51d05e11bbdc26b240165c93f1a6e5dc2908ff11c1a14f339cd75e7ebb, pg-trickle 3f83ffdc01756e81b04785d173c7e8307327fc6bf09e019f8cc521dcb234fa58, React 34a46bbf37c3dcc327d156ae24663af44ca89f8bec0d634dd65f357b7711a4f5. All repeated focused executions and validation exits and full-audit exit were 0; full audit recorded 51 passing checks and QUALIFICATION_INCOMPLETE. Logical restore and release-scale qualification were not part of this candidate run.

    A candidate run on new main 40689383405 is queued; the separate logical activation probe 40685124814 remains in progress. The logical-restore state-loss defect is still open. No performance acceptance or production activation is claimed.

  22. grove commented on Oct 29, 2026

    @grove
    grove
    on Oct 29, 2026
    ContributorAuthor

    Focused missing-dispatcher recovery regression (2026-10-29): a local pre-fix run showed pgreact.rebuild_transient_metadata() reporting 2 rebuilt rules and 0 blocked when a stored consequence dispatcher identity did not resolve. The local guard now blocks that rule and writes a RECONCILING barrier. The regression passes on fresh 0.46.1 install and 0.46.0→0.46.1 upgrade; restoring the identity allows rebuild and barrier clearance. The React logical-restore regression also passes its exact state comparison. These are uncommitted local candidate results based on HEAD 90c7e17, not release qualification; full joint-stack MDM/pg-trickle restore qualification remains open.

  23. grove commented on Oct 30, 2026

    @grove
    grove
    on Oct 30, 2026
    ContributorAuthor

    2026-10-30 — logical-restore recovery follow-up after pg-trickle PR #1288 merged.

    A focused local audit on React 90c7e17bb060a85166b6830394d1f0f0a7c7089a used merged pg-trickle source eb44093bbeca4f276a92733fe2bbc740d8ef254a and the reassembled 0.46.1 install/upgrade SQL. It passed 23 checks through pending managed-work delivery. The audit restored 18 producer streams with their owners, orchestration modes, and runtime MAINTAIN grants; rebuilt CDC triggers; rebound React metadata after producer OID changes (2 rules rebuilt, 0 blocked, complete state preserved); validated capture recovery as SAFE; replayed the original MDM receipt; resumed five streams; and delivered pending managed work.

    This is uncommitted local amd64 diagnostic evidence. It used an image-only compatibility admission for unreleased pg-trickle 0.108.4. The manifest records release_qualified=false. Exact-current joint-stack workload, mixed-backlog fairness, recovery/resource limits, and 90-day retention remain open.

    At last check, CI run 40973089476 was still queued, with two failed Sensitive E2E gates and three queued jobs. The run had not completed, so their failure causes are not established.

    Local evidence is retained at tests/integrations/pg-mdm/evidence/v0.49-logical-restore/assembled-1288-logical-restore-final-local-20261030/ (manifest.json and observations.json); it is not a published CI artifact.

  24. grove commented on Oct 30, 2026

    @grove
    grove
    on Oct 30, 2026
    ContributorAuthor

    CI failure-log follow-up — 2026-10-30

    The completed job logs for run 40973089476 identify these failures:

    • Sensitive E2E gate 1/2: test_scheduler_refreshes_multiple_healthy_sts timed out after 60 seconds; the second stream remained at one row instead of two.
    • Sensitive E2E gate 2/2: test_mixed_sched_parallel_sts_same_view_serial produced four grouped rows instead of five; the expected info=1 and error=3 groups were missing, while error=2 was extra.

    PR #1288 changes src/api/output_delta.rs, tests/e2e_pg_dump_tests.rs, and CHANGELOG.md; neither failing scheduler test file is changed. The base commit CI run 40862112017 skipped the Sensitive E2E gates, so it provides no direct baseline comparison. The failure symptoms are confirmed, but their root causes and relation to PR #1288 are not established; I am not classifying them as flakes or PR regressions.

    At the latest run check (2026-10-30 02:05 UTC), CI was still in progress with three jobs running.

  25. grove commented on Oct 30, 2026

    @grove
    grove
    on Oct 30, 2026
    ContributorAuthor

    2026-10-30 active-temporal logical-restore follow-up: manifest and full observations record 28 passing focused checks. An API-authored HYSTERESIS key 9002 remained ACTIVE (generation 1; input true/recovery false) before dump, after restore, and after React recovery; its recovery-condition OID was stale after restore and rebound by prepare_recovery(). Reinitialization validated SAFE across 19 producer streams; MDM reactivation and pending-work delivery passed. Later no-work cycles added only append-only OBSERVE history, preserving temporal state without lifecycle replay. This remains local, uncommitted amd64 evidence using image-only admission for unreleased pg-trickle 0.108.4 (release_qualified=false); exact-stack 300-case, fairness, resource/recovery, and 90-day retention gates remain open. PR #1288 is merged, but post-merge run 40973089476 completed with two Sensitive E2E failures and one queued PR-correctness job; causes remain unresolved, with no base-gate comparison.

  26. grove commented on Oct 30, 2026

    @grove
    grove
    on Oct 30, 2026
    ContributorAuthor

    CI status correction — 2026-10-30 02:27 UTC: run 40973089476 completed with failure. The PR-correctness aggregator job failed only because e2e-sensitive-gate failed; it did not surface a separate defect. The two Sensitive E2E failures are described in my prior comment. The run had 22 other executed checks pass and 18 skipped. The base run skipped both Sensitive gates, so there is no comparable baseline and no basis yet to attribute these scheduler failures to PR #1288 or call them flakes. PR #1288 remains merged as eb44093bbeca4f276a92733fe2bbc740d8ef254a; I recommend tracking these gate failures separately and leaving the merge intact pending baseline/root-cause evidence.

  27. grove commented on Oct 30, 2026

    @grove
    grove
    on Oct 30, 2026
    ContributorAuthor

    Evidence path correction: the manifest and observations URLs in my 02:26 UTC comment point to files not present on GitHub main. They are uncommitted local artifacts at tests/integrations/pg-mdm/evidence/v0.49-logical-restore/assembled-active-temporal-local-20261030-rerun3/manifest.json and the sibling observations.json in the shared worktree. I have not committed or pushed this workspace.

  28. grove commented on Oct 30, 2026

    @grove
    grove
    on Oct 30, 2026
    ContributorAuthor

    Timing clarification for run 40973089476: it was triggered by pull_request on PR head 9b90091ae9347c9ee6bda451fe609bb0688c6c28 at 2026-10-29 23:25 UTC, before the 23:31 UTC merge, and completed at 2026-10-30 02:27 UTC. These are PR-head workflow results, not a separate push workflow on the merge commit; the failure details and missing base comparison are unchanged.

  29. grove commented on Oct 30, 2026

    @grove
    grove
    on Oct 30, 2026
    ContributorAuthor

    Follow-up local diagnostic (2026-10-30; run pgr1288-committed-refresh-arm64-20261030-4; ARM64 pg-trickle 0.108.4 image sha256:ac4811ecc945acc7af593cff7f216505071838b440ad475f9dc104be3a9dd2fd) committed the source-write transaction before calling mdm.refresh. It completed 12 offers with no 40P01 retries, but took 172.837 seconds from first offer to last completion (4.17 admissions/minute), below the 40/minute floor. Only 2 of 12 refreshes reported changes; the other 10 found their staged rows already consumed by another refresh.

    At 834.866 seconds after the due deadline, 968/1,000 ESCALATE receipts were present; work showed 1,001 completed items, zero pending or claimable work, and 1,001 attempts. Case keys 9–40 had no receipt, intent request, or attempt record although their work episodes had completed. A later query found 32 level-2 candidates. submit_escalation_intent submits only when its candidate query finds a row, so an empty candidate result at callback time is plausible; candidate state at that exact time was not captured.

    This run is incomplete, far below the throughput floor, and does not establish a root cause or attribute the behavior to pg-trickle PR #1288. Next: add a deterministic regression for the missing-intent transition, then rerun the full paired 300-case profile. The compact record and artifact hashes are in the local worktree at tests/integrations/pg-mdm/evidence/v0.49-mixed-load-committed-refresh-local-20261030/; those files have not been pushed to GitHub.

  30. grove commented on Oct 30, 2026

    @grove
    grove
    on Oct 30, 2026
    ContributorAuthor

    Callback-time candidate follow-up — 2026-10-30

    A second isolated local ARM64 diagnostic (pgr1288-callback-probe-arm64-20261030-2, image sha256:ac4811ecc945acc7af593cff7f216505071838b440ad475f9dc104be3a9dd2fd) instrumented the candidate lookup in submit_escalation_intent. Of 1,000 due-case callbacks, 968 found an escalation candidate; 32 (case keys 9–40) found none. Those callbacks ran 12.394–17.482 seconds after the shared deadline while their source rows were open at escalation level 1, permitted ESCALATE, and action revision 2. At termination, the same 32 keys had no receipts; all 1,001 work items and attempts had completed, with zero pending or claimable work. A later query found all 32 in the candidate view.

    This ties the callback-time empty lookup to the exact missing-receipt set. The reason the authorized candidate view omitted them at dispatch remains unknown. The run offered only 12 arrivals at a 60/minute target, below the 40/minute qualification floor; it is incomplete and does not qualify the operating envelope. A follow-up local probe is capturing the authorized case row, route result, and binding runtime in the same SQL snapshot.

    Compact local evidence: tests/integrations/pg-mdm/evidence/v0.49-mixed-load-callback-probe-local-20261030/ (observations.json, manifest.json). Full logs remain in the recorded /private/tmp directory; these artifacts are uncommitted and unpublished.

  31. grove commented on Oct 30, 2026

    @grove
    grove
    on Oct 30, 2026
    ContributorAuthor

    Route-state follow-up — 2026-10-30

    The callback probe identified the cause of the 32 missing receipts. All 32 cases were present in the authorized view and had an applicable level-2 route, but the route explanation said deadline has not been reached. Every explanation captured statement_timestamp() at 07:49:12.039 UTC, 5.755 seconds before the deadline; the callbacks ran 12.737–22.955 seconds after it. The long managed_cycle() statement had started before the deadline, so its fixed statement timestamp made overdue cases appear not yet due. submit_escalation_intent then returned without submitting a request. The 32 affected keys exactly match the terminal missing-receipt set.

    The local working-tree fix passes clock_timestamp() explicitly from intent_escalation_candidates to the four-argument route_cases function. I added a same-statement regression that starts before the deadline, waits past it, and requires the level-2 candidate. Verification is underway. These diagnostics offered only 12 admissions at a 60/minute target, below the 40/minute floor; they are incomplete and not qualification evidence.

    Local compact evidence and hashes: tests/integrations/pg-mdm/evidence/v0.49-mixed-load-callback-route-state-local-20261030/. The artifacts are uncommitted and unpublished.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions