Repository navigation
perf(paths): evaluate transitive paths on term ids - #31
Merged
Merged
Conversation
Part of #11 (phase 4). `?s <p>+ ?o`, `?s ^<p>+ ?o`, and `<p>*` with a constant end are now in the id tables: the predicate's edges are read once, sorted, and searched from each start, each end once per start as SPARQL's paths give them. Counts, grouped counts and row parts with such a path use them; a part with a path is always taken, since the evaluator finds a closure one start at a time. On a tree of 200,000 nodes (1.48 million pairs), release build: - GROUP BY ?o COUNT over ?s <p>+ ?o: 1.27 -> 0.49 s - the path joined with a class, counted: 1.89 -> 0.47 s - the path's rows alone: 1.63 -> 1.20 s Answers identical (sorted outputs compared). Left to the evaluator: the same variable at both ends, `*` between two variables, a constant the store doesn't have, and other path forms.
A path joined with other patterns starts from the values they bind (backwards when they bind the object), looking edges up level by level for few starts, instead of building the whole closure. Parts with a path joined with patterns outside them are left to the evaluator, and keyed row counts leave paths to the seeded join.
…-n3oul3 # Conflicts: # CHANGELOG.md
…mory The level search now counts the pairs a level can add (with the ids held in seen, next and the edges read) before making any, and leaves the path to the evaluator past the limit; cancellation is checked inside the level. The search over all edges checks the limit as each target is added, so a node with many edges cannot pass it either.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #11 (phase 4: transitive paths).
What
The id tables now evaluate these transitive paths:
?s <p>+ ?o;?s ^<p>+ ?o(also written(^<p>)+);<p>*when one end is a constant, or bound by the patterns it is joined with.Where the search starts:
A few starts are searched by looking up their edges level by level; many, by reading all the predicate's edges once. A part with a path that is joined with patterns outside it (a
VALUESblock, say) is left to the evaluator, which also starts from those values. Keyed row counts leave paths to the seeded join.These are left to the evaluator:
*between two free variables;Measured
Release build, server with the result cache off, on a tree of 200,000 nodes (1.48 million pairs). Rewrite on vs off:
GROUP BY ?oCOUNTover?s <p>+ ?oCOUNTover?s <p>+ ?oORDER BY ?s ?o LIMIT 5over the path?s ex:id "rare" . ?s <p>+ ?oand the other selective joinsAnswers are identical on all 25 probe queries.
Tests
tests/id_paths.rs: a differential test against#pragma tt.id-rewrites offover 25 path shapes, comparing rows and grouped counts. Shapes include cycles, a self-loop, a literal end, a named graph, unknown terms, MINUS and FILTER, and joins withVALUES, OPTIONAL, UNION, two paths, and bound objects.tripletrailcrate passes; fmt and clippy are clean.Docs