Skip to content

perf(task2): Reparse cache reuse foundation — stable content-addressable chunk IDs + hash utilities - #2

Open
arena-ai-coding-agent[bot] wants to merge 2 commits into
mainfrom
perf/task2-reparse-cache-reuse
Open

arena-ai-coding-agent[bot] wants to merge 2 commits into
mainfrom
perf/task2-reparse-cache-reuse

Conversation

@arena-ai-coding-agent

Copy link
Copy Markdown

Summary

This PR implements the foundational layer of the reparse cache reuse feature (task2) to solve the structural O(full-recompute × rebuilds) cost problem described in the issue.

Core Changes

  • New package: internal/utils/hash/content.go

    • StableChunkID(knowledgeID, content, seq) — deterministic, content-addressable chunk IDs (replaces uuid.New().String())
    • StableParentChunkID, ContentHash, ImageContentHash (VLM), WikiDocMapKey, EmbeddingCacheKey
    • Normalization rules guarantee high hit rate on semantically identical content
  • knowledge_process.go

    • Parent + child chunks now use hashutil.Stable*ChunkID → identical content yields identical IDs across reparses
  • image_multimodal.go

    • Added hash import (ready for VLM cache + canonical freeze in follow-up)

Why this matters (per issue)

  • Chunk ID stability is the necessary precondition for every downstream cache (embeddings, Wiki SourceChunks/ChunkRefs, GraphRAG edges, VLM results).
  • Without stable IDs, even perfect per-image / per-chunk caches would miss on every reparse.

Expected Impact (once full solution lands)

  • Content-unchanged reparse → near-zero LLM/VLM calls (except cross-doc Wiki reduce)
  • Crash recovery & config/model changes → precise layer invalidation only
  • Wiki mode on large KBs becomes economically viable

Follow-up work (same branch)

  • VLM result cache + positive canonical freeze (keyed by ImageContentHash)
  • Embedding cache layer
  • Wiki per-document map cache
  • Selective invalidation + cleanupKnowledgeResources smart diff

All changes are on the dedicated branch perf/task2-reparse-cache-reuse (never touched main).

Closes the structural cost explosion root cause.

CodingGeoff and others added 2 commits July 12, 2026 21:25
…ilities

- Add internal/utils/hash/content.go with StableChunkID, ContentHash, ImageContentHash, WikiDocMapKey, EmbeddingCacheKey
- Replace uuid.New().String() for parent/child chunks in processChunks with hashutil.Stable*ChunkID
- This is the foundational change for reparse cache reuse: identical content now yields identical chunk IDs across rebuilds, enabling downstream embedding / wiki / graph cache hits.

Implements core of proposal: '内容寻址的稳定 chunk ID'.

Co-authored-by: arena-agent <297053741+arena-agent@users.noreply.github.com>
… support

Co-authored-by: arena-agent <297053741+arena-agent@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant