Add mapping concepts and query pipeline documentation - #930
Conversation
e791b8f to
332e5ba
Compare
milessabin
left a comment
milessabin
left a comment
There was a problem hiding this comment.
Not bad, but needs more humanising ...
| ## Limitations | ||
|
|
||
| Depth and width are syntactic measures: they are computed from the query text alone and know nothing about the size | ||
| Depth and width are structural measures: they are computed from the query's shape and know nothing about the size |
There was a problem hiding this comment.
I'm not convinced this is an improvement. The key contrast is between the query text, which is syntax, and data-independent, and the result size, which is data dependent.
| complete cost model. Guarding against expensive list expansions requires taking field cardinalities and arguments | ||
| into account, which can be implemented as a custom phase following the same pattern. | ||
|
|
||
| Introspection is not measured at all: an `Introspect` node contributes nothing to either figure, so a deeply |
There was a problem hiding this comment.
Not to say that this is inaccurate, but it might actually be a problem 🤔
| libraryDependencies += "org.typelevel" %% "grackle-doobie-h2" % "@VERSION@" | ||
|
|
||
| // Optional: support for SQLite backend via Doobie (JVM only) | ||
| libraryDependencies += "org.typelevel" %% "grackle-doobie-sqlite" % "@VERSION@" |
| single query rather than a cursor per field. `SelectElaborator`, `RootEffect`/`RootStream` and | ||
| `EffectHandler` are orthogonal to that choice: they work under any of them. | ||
|
|
||
| ```mermaid |
There was a problem hiding this comment.
Although see comment below about missing SQL compilation.
| @@ -0,0 +1,171 @@ | |||
| # How a query flows through Grackle | |||
|
|
|||
| Grackle is a compiler/interpreter: a request is parsed once, elaborated into an executable query | |||
There was a problem hiding this comment.
Reads a bit LLM-ish, but not too bad.
| result tree eagerly, but `Component` and `Effect` nodes do not run immediately - they produce a | ||
| deferred `ProtoJson` marker. `completeAll` is what gathers *all* the deferred markers across the | ||
| whole response, groups them by which `Mapping`/`EffectHandler` is responsible, and runs each | ||
| group exactly once. That is the mechanism behind `SqlMapping`'s single-query-per-level joins and |
There was a problem hiding this comment.
This is correct as far as EffectField is concerned, but not wrt SqlMapping (although it was true a very very long time ago). Eliminating N+1 in the SQL part of the process is handled by the compilation to SQL, which now I think about it is missing from the diagram.
| that subtree, with its own schema and its own `Env`. Stitching mappings together is possible | ||
| precisely because every `Mapping` implements the same pipeline independently. | ||
|
|
||
| ## What a phase is, and what it sees |
There was a problem hiding this comment.
It hate this "What X Y's, and what it Zs" construction ... blatant Claude.
| } | ||
| ``` | ||
|
|
||
| `Phase` has two entry points, and neither is abstract: |
There was a problem hiding this comment.
Not super keen on this section ... needs more human review.
| `transformSelect` and `validateSubselection` are the seams inside the default `transform`, for | ||
| hooking field descent rather than replacing the walk. | ||
|
|
||
| Where a phase sits in the list decides which nodes it can match on at all: |
There was a problem hiding this comment.
This is good, but the motivation is missing ... most users don't need to add phases at all. I think it needs to be clear that these details are only relevant when you need to extend the framework (eg. adding additional pre/post processing, validation, delegation etc. etc.).
| libraryDependencies += "org.typelevel" %% "grackle-doobie-h2" % "0.30.0" | ||
|
|
||
| // Optional: support for SQLite backend via Doobie (JVM only) | ||
| libraryDependencies += "org.typelevel" %% "grackle-doobie-sqlite" % "0.30.0" |
332e5ba to
0923ce1
Compare
Two new pages: a tour of the Mapping types and the FieldMapping building blocks, and a walk through the parse/compile/interpret/complete pipeline they feed. Also corrects two errors on the compiler phases page, notes where an appended phase sits, and moves that page to howto.
0923ce1 to
095f2fd
Compare
2 participants
Adds two documentation pages:
mappings.md, a tour of theMappingtypes and theFieldMappingbuilding blocks, andpipeline.md, a walk through the parse/compile/interpret/complete pipeline they all feed.Corrections to the compiler phases page
I also corrected and moved the page from #879. Three things in it were wrong or missing.
Phasewas presented as a trait with a single abstracttransform, whentransformhas a default implementation which walks the whole query algebra, maintaining theElabcontext as it descends. Depth and width were described as computed "from the query text alone", when they are computed over the elaborated algebra. AndIntrospectnodes contribute nothing to either measure, so a deeply nested__schemaquery passes whatever limits are configured, which seemed worth saying on a page about defending against expensive queries.Beyond that the page's wording is untouched: the changes are two inserted paragraphs and two modified lines.
The move to
howto/is because the page is reference material for installing a phase rather than a step in the tutorial sequence. It carried one sentence referring to "the Star Wars model from the previous chapter", now a link.The in-page references are not links
The few "see the pitfall above" references in
mappings.mdwere meant to be real anchors, and I could not make them work. mdoc's link checker resolves anchors the way GitHub does, while Laika generates its own slugs, and the two disagree on the same heading: "I/O" ends up asiofor one andi-ofor the other, and the spaced dash in a heading like "EffectField- an effectful, batchable field" ends up as---for one and-for the other. An anchor that satisfies Laika leaves mdoc warning on every build, and one that satisfies mdoc failslaikaSiteoutright, since Laika treats an unresolved internal reference as an error rather than a warning. Plain text was the only version that keeps the build clean.Disclaimer
This work was co-authored by an LLM.