Skip to content

perf(remote): avoid full journal snapshots during event accounting - #285

Open
sambitcreate wants to merge 1 commit into
mainfrom
feature/perf-stream-accounting
Open

sambitcreate wants to merge 1 commit into
mainfrom
feature/perf-stream-accounting

Conversation

@sambitcreate

Copy link
Copy Markdown
Owner

Every Remote event previously deep-cloned and serialized the entire retained journal to enforce its byte budget, even with persistence disabled. Account for immutable owned event bytes plus the bounded metadata envelope instead, retaining exact UTF-8 JSON limits and existing replay/terminal-delivery behavior. Apply final state/timestamp before checking the budget.

The identical synthetic workload now performs zero full snapshots for 64 appends, down from 64, with byte-identical final snapshots at both retention sizes. Timings are exploratory; persistence still snapshots normally and no GPU/energy savings are claimed.

Validation: 62 stream tests; broader Remote suites 533 passed with one environmental occupied-port skip; TypeScript and scoped lint passed. Fresh-context GPT-6 Astra medium review found no actionable issues, independently passed the 62 tests, and passed 7,452 exact-byte oracle comparisons across Unicode, escaping, timestamps, resets, terminal transitions, pruning and revocation. Native consumers were inspected; wire/recovery contracts are unchanged.

Evidence and reproducible fixture: docs/performance/remote-stream-accounting.md and remote-stream-accounting-benchmark.mts.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes I reviewed the stream-accounting implementation, its behavioral tests, and the reproducible benchmark documentation.

  • Exact budget accounting Replaces full retained-journal serialization on append with per-event UTF-8 sizes and bounded metadata-envelope accounting.
  • Journal ownership and ordering Clones payloads, accounts normalized restore data, and applies final stream state and timestamp before checking the budget.
  • Behavioral validation Adds byte-oracle coverage for Unicode, restore and cleanup, trimming, aggregate pressure, and delivery, plus a synthetic benchmark.

Pullfrog  | View workflow run | Using GPT Luna | 𝕏

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant