Skip to content

Close phase 6 in the roadmap and open phase 7 - #20

Merged
rohan-hotdata merged 2 commits into
mainfrom
close-phase-6
Oct 14, 2026
Merged

rohan-hotdata merged 2 commits into
mainfrom
close-phase-6

Conversation

@rohan-hotdata

@rohan-hotdata rohan-hotdata commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

This PR closes phase 6 in the roadmap and opens phase 7 as issue #19. It changes docs/internal/roadmap.md and docs/internal/plan.md only.

How

  • roadmap.md: phase 6 is closed with PR Phase 6: the accuracy loop, with sample B and the diagnosis split #18, and phase 7 is open with issue Phase 7: the extractor #19. Phase 6 gets its closing paragraph, with what it changed for later phases.
  • plan.md: the phase 6 section is removed, because the plan keeps only the open phases. Its full text and decisions stay at commit 8a7318d.
  • plan.md: the start table gains the sample B baseline and the split of the recalled group.
  • plan.md: phase 7 names the target on both samples (19 on A, 16 on B), and lists the commands that phase 6 built.
  • plan.md: task 2 of phase 7 gains a note, read from the code and not observed. A reply at the token limit is not stored in the cache. A reply that fails to parse is stored, but it stops the run, so no completed result file holds either fault.

Verification

Notes

  • The untracked docs/internal/mem0-survey.md is not part of this PR.

Phase 6 closed with PR #18. The roadmap records what it changed for later
phases. The plan drops phase 6, adds the sample B baseline to its start
point, and gives phase 7 the commands that phase 6 built. Phase 7 is open
as issue #19.
@rohan-hotdata
rohan-hotdata requested a review from a team as a code owner October 14, 2026 09:59
@rohan-hotdata
rohan-hotdata requested review from shefeek-jinnah and removed request for a team October 14, 2026 09:59
Comment thread docs/internal/plan.md Outdated
memory".

Read from `bench/models.py`, not observed: a reply that reaches the limit raises
before the cache stores it, and a reply that fails to parse stops the run. So the

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: The note says the cache does not hold a reply that fails to parse. The cache does hold that reply (not blocking).

Fix: "a reply that fails to parse is stored in the cache, but it stops the run. So no completed result file holds either fault, and this task confirms it."

CachedModels.__call__ stores each reply at bench/models.py:76-80 before it returns the reply. json.loads runs later, at bench/e2e.py:160, so a parse failure raises only after the insert. The token-limit case at bench/models.py:137-138 raises inside complete, so the first half of the note is correct. Without the fix, task 2 can look for the wrong thing in the cache.

claude[bot]
claude Bot previously approved these changes Oct 14, 2026
@rohan-hotdata
rohan-hotdata merged commit 9a51cae into main Oct 14, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant