Skip to content

Add real retrieval baseline benchmark - #4

Merged
princerm06 merged 12 commits into
mainfrom
benchmark-real-retrieval-baseline
Oct 7, 2026
Merged

princerm06 merged 12 commits into
mainfrom
benchmark-real-retrieval-baseline

Conversation

@princerm06

@princerm06 princerm06 commented Oct 7, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • freeze 25 natural-language retrieval queries against the current 21-page Synapse dataset
  • add a live runner using the existing embedding + pgvector + 4B deduplicated retrieval pipeline
  • preserve the first measured baseline: Hit Rate@5 0.800, Recall@5 0.800, MRR 0.800
  • diagnose that pages 1–4 have zero chunk rows, explaining the benchmark misses
  • add a transactional, explicit page reprocessing utility that rebuilds chunks from already-stored canonical page content
  • default the repair command to dry-run; require --apply before any DB mutation
  • add unit tests and CI coverage for the reprocessing service

Methodology

Ground-truth labels were frozen before running Synapse. The benchmark itself remains unchanged after discovering the legacy-data inconsistency, so the post-repair run can be compared directly with the initial baseline.

Safety

No credentials, migrations, deployment changes, or automatic production/database writes. The live benchmark and reprocessing command remain local. Reprocessing is limited to page IDs explicitly supplied by the user and commits transactionally.

First measured baseline

  • Queries: 25
  • Hit Rate@5: 0.800
  • Recall@5: 0.800
  • MRR: 0.800

Pages 1–4 were then verified to have zero chunks. The next local step is to dry-run and then rebuild only pages 1–4, followed by rerunning the exact frozen benchmark.

@princerm06
princerm06 marked this pull request as ready for review October 7, 2026 18:03
@princerm06
princerm06 merged commit 5f43f5c into main Oct 7, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant