Add docs about multiple start rules.#649
Conversation
| A "A" | ||
| B "B" | ||
| [^\p{any}] "START_A" | ||
| [^\p{any}] "START_B" |
There was a problem hiding this comment.
These are mostly just to silence the missing from lexer warning, it seemed like the easiest way at the time.
There is also allow_missing_terms_in_lexer, though that doesn't appear to allow you to say set a list of terms.
Although these regex should be very fast, I suppose their existence in the list of lex rules, still means you're
going to check the empty regex for each character.
I don't know in practice how much of a difference that makes, for however many synthetic start tokens people have, over each unmatched character. I suppose that if they're at the very end, they'll only match unmatched characters in practice.
As such, I should probably move the next line above these rules.
There was a problem hiding this comment.
allow_missing_terms_in_lexer
Good point. I wonder if we should add an equivalent of lrpar's %expect-unused to lrlex? Arguably we could then remove the Rust API allow_missing_terms_in_lexer entirely (or, perhaps, make it a no-op and deprecated for 0.15.0?).
There was a problem hiding this comment.
That sounds like a good idea, I'll try work on this some more mid week.
|
Thanks for this! I think you've got the right technical solution in here, which is the most important thing to get right. What I would probably suggest is that perhaps slightly cut down some of the lead-up and getting to the solution quicker. We can briefly allude to lesser alternatives if we want, but I think for this sort of thing most readers most want to know what we recommend (and less what we don't recommend). [Part of me can't quite believe that I had never thought of this way of doing things as a possibility! It's a really neat trick.] |
|
Yeah, I'm also a bit surprised I've never thought of it either (nor run across it before). I'll work on the text some more as soon as I can. |
2 participants
Here is a first attempt at documenting the current ability to perform the hack for faking multiple start rules by jump starting the lexer with a synthesized token.
When doing this I noticed some issues in the Hand-written lexer section, but will handle those in a separate patch.
I don't know if the progression I used here is good, e.g. it goes from the hard implementation explaining the underlying details, to the easier one.
But it was just a first attempt I suppose.