feat(search-pipeline): compose search indexing as a pipeline instance#565
Merged
Conversation
ddeboer
force-pushed
the
feat/search-pipeline
branch
from
9062fc7 to
1cb1af9
Compare
ddeboer
force-pushed
the
feat/search-pipeline
branch
2 times, most recently
from
930010a to
e739e06
Compare
New @lde/search-pipeline package: searchIndexWriter is the one type-changing step (quad → document) between the pipeline’s quad world and a search engine’s document world, shared across engines. It buffers each dataset’s extracted CONSTRUCT quads until the dataset flushes, frames and projects them with @lde/search’s projectGraph, and writes the documents to the engine writer’s run before forwarding the flush – so an In-place stale sweep never races its own documents. Run context, per-dataset outcome, reset (dump-fallback discard) and commit/abort pass through unchanged; the engine writer’s update mode governs. Memory is bounded by one dataset’s extraction and released per flush; bounded entity batches within one huge dataset (the two-level iteration) are future work.
Same ERESOLVE as search-typesense: the ^0.31.4 peer predates the 0.33.0 release on main; the range now tracks the released version and the lockfile entry follows.
ddeboer
force-pushed
the
feat/search-pipeline
branch
from
e739e06 to
b9af21c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Slice 3 of #534, stacked on #560: the new
@lde/search-pipelinepackage that makes search indexing an@lde/pipelineConfigurable Pipeline instance.What it does
searchIndexWriter({ schema, writer })is the one type-changing step (quad → document), shared across engines – which is why it lives here and not inside an engine adapter:@lde/search’sprojectGraph(engine-agnostic);flushoutcome,resetfor the dump-fallback discard, commit/abort) – the engine writer’s update mode governs, and the pipeline never branches on it.With this, a search index rebuild is a plain
Pipelinewith an extractionSparqlConstructReaderandwriters: searchIndexWriter({ schema, writer: new BlueGreenRebuild(...) })– reusing Discovery → Selection → change-gate (skip-unchanged) instead of the bespoke full-rebuild indexer.Scope notes
Releasing
New package: after merge, the first version must be published manually per the
npm-bootstrap-packageflow (npm view @lde/search-pipeline→ 404 today).Part of #534 (slice 3 of 4; next: per-reference label sources + facet-by-name search).