Skip to content

perf(paths): evaluate transitive paths on term ids - #31

Merged
MathiasVDA merged 5 commits into
mainfrom
claude/project-thread-n3oul3
Nov 18, 2026
Merged

MathiasVDA merged 5 commits into
mainfrom
claude/project-thread-n3oul3

Conversation

@MathiasVDA

@MathiasVDA MathiasVDA commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Part of #11 (phase 4: transitive paths).

What

The id tables now evaluate these transitive paths:

  • ?s <p>+ ?o;
  • ?s ^<p>+ ?o (also written (^<p>)+);
  • <p>* when one end is a constant, or bound by the patterns it is joined with.

Where the search starts:

  • from a constant end;
  • else from the values the joined patterns give (the path comes last in the join), searching backwards when they bind the object;
  • else from every node with an edge.

A few starts are searched by looking up their edges level by level; many, by reading all the predicate's edges once. A part with a path that is joined with patterns outside it (a VALUES block, say) is left to the evaluator, which also starts from those values. Keyed row counts leave paths to the seeded join.

These are left to the evaluator:

  • the same variable at both ends;
  • * between two free variables;
  • a constant the store doesn't have;
  • any other path form;
  • a closure larger than the query's id budget (the search's visited pairs count against it too).

Measured

Release build, server with the result cache off, on a tree of 200,000 nodes (1.48 million pairs). Rewrite on vs off:

Query On Off
GROUP BY ?o COUNT over ?s <p>+ ?o 0.19 s 1.05 s
path joined with a class, counted 0.05 s 1.61 s
chain of 3,000, COUNT over ?s <p>+ ?o 0.02 s 2.95 s
ORDER BY ?s ?o LIMIT 5 over the path 0.19 s 1.70 s
?s ex:id "rare" . ?s <p>+ ?o and the other selective joins 0.001–0.005 s 0.001–0.002 s

Answers are identical on all 25 probe queries.

Tests

  • tests/id_paths.rs: a differential test against #pragma tt.id-rewrites off over 25 path shapes, comparing rows and grouped counts. Shapes include cycles, a self-loop, a literal end, a named graph, unknown terms, MINUS and FILTER, and joins with VALUES, OPTIONAL, UNION, two paths, and bound objects.
  • A timing guard on a 60,000-node tree: each selective join must take under a tenth of the whole closure's time, with the same count.
  • Every test file of the tripletrail crate passes; fmt and clippy are clean.

Docs

  • docs/store.md: the id-table section, including the selective case.
  • CHANGELOG.

Part of #11 (phase 4). `?s <p>+ ?o`, `?s ^<p>+ ?o`, and `<p>*` with a
constant end are now in the id tables: the predicate's edges are read
once, sorted, and searched from each start, each end once per start as
SPARQL's paths give them. Counts, grouped counts and row parts with
such a path use them; a part with a path is always taken, since the
evaluator finds a closure one start at a time.

On a tree of 200,000 nodes (1.48 million pairs), release build:
- GROUP BY ?o COUNT over ?s <p>+ ?o: 1.27 -> 0.49 s
- the path joined with a class, counted: 1.89 -> 0.47 s
- the path's rows alone: 1.63 -> 1.20 s
Answers identical (sorted outputs compared).

Left to the evaluator: the same variable at both ends, `*` between two
variables, a constant the store doesn't have, and other path forms.
A path joined with other patterns starts from the values they bind (backwards when
they bind the object), looking edges up level by level for few starts, instead of
building the whole closure. Parts with a path joined with patterns outside them are
left to the evaluator, and keyed row counts leave paths to the seeded join.
…mory

The level search now counts the pairs a level can add (with the ids held
in seen, next and the edges read) before making any, and leaves the path
to the evaluator past the limit; cancellation is checked inside the level.
The search over all edges checks the limit as each target is added, so a
node with many edges cannot pass it either.
@MathiasVDA
MathiasVDA merged commit b589655 into main Nov 18, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant