Conversation
added 4 commits
October 2, 2026 15:05
…ch/rollback in KBService (list_versions, search_at_version, rollback)
…rsion (list_versions, *_at_version, rollback)
…ular (Dolt) files
Every file operation (add / update / delete) produces a new version on
both sides — a new LanceDB commit on the KB side and a new Dolt commit
on the DB side. Which pieces participate depends on the ingest mode:
Text files — two ingest modes:
* `direct` + lancedb: per-file, low-latency. `kb_add_file()` chunks
and writes straight into the vector store; re-adding the same path
replaces its chunks (upsert). **No** workspace-level scan and
**no** version history — the direct path bypasses the KB's
version-tracking layer.
* `cocoindex` + lancedb: workspace-level, versioned.
- `kb_add_file()` copies the file into `uploads/` (unless already
there), then triggers `app.update()`.
- `kb_update_workspace()` runs a full scan; only changed / added /
removed files are (re)indexed.
- `kb_delete_file()` requires the file to be gone from disk, then
syncs so its chunks disappear.
- **Every** above operation commits a new LanceDB version, so
`list_versions` / `search_at_version` / `rollback` all work.
Tabular files — Dolt (always versioned):
* `db_import_path()` / `db_update_path()` / `db_delete_path()` use
the `_source_file` column to scope mutations to the originating
file. Re-importing a file replaces only its own rows; deleting the
file drops only those rows.
* Every mutation is committed as a new Dolt version, so per-file
history (`list_versions`, `*_at_version`, `rollback`) is available
for any file operation.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR introduces version control and incremental update capabilities for the DataMind Knowledge Base (KB) and Database (DB), alongside related toolchain fixes and improvements.
Key Changes
1. KB Version Control (LanceDB)
add_versiontoLanceDBVectorStore. It compares snapshots via_snapshot_filesto automatically trackAdd,Update,Delete, andchunks_added/chunks_deleted.rollbackandlist_versionsto restore the KB to a historical version and syncversion_list._record_file_versioninIngestServiceto automatically trigger version commits after operations likekb_add_text,kb_add_file, andkb_delete_file.2. DB Version Control (Dolt)
DoltDialect:commit_version: Automatically records a commit for write operations.list_versions/list_tables_at_version/describe_at_version: Queries table schemas and metadata at historical versions.query_sql_at_version/execute_readonly_at_version: Executes read-only queries at a specific historical version (automatically injectsAS OF '<version>').rollback: Implements data rollback viaDOLT_REVERTwhile preserving full commit history.db_list_versions,db_list_tables_at_version,db_query_sql_at_version,db_query_nl_at_version,db_rollback, etc.).3. Ingest Incremental Update (CocoIndex + LanceDB)
CocoIndex-backed KB indexer(providers/cocoindex.py).memo=Trueandlocalfs.walk_dirto reprocess only files whose content or embedder identity has changed. State is persisted understorage_dir/<name>_state, significantly reducing redundant computation.kb_update_workspacefully integrates CocoIndex and syncs parsed chunk caches (.jsonl) for non-text files via_sync_prechunked_files.4. Toolchain Fixes & Bug Fixes
db_import_csv/db_update_*/db_delete_path. Clarified thatreplaceclears rows and preserves the table schema (instead of drop & recreate), and thatdb_delete_pathremoves both the on-disk file and its records across all tables.db_delete_pathto execute file deletion and DB record cleanup in a single transaction (supporting Dolt commit).Testing
Added and updated the following test files to cover KB/DB versioning and ingest logic. All local tests pass:
datamind/tests/test_kb_versioning.pydatamind/tests/test_db_versioning.pydatamind/tests/test_kb_ingest.pydatamind/tests/test_db_ingest.pyTest command: