Repository navigation
GH-1324: bad performance when parsing huge literals - #4423
Open
OyvindLGjesdal wants to merge 4 commits into
Open
OyvindLGjesdal wants to merge 4 commits into
OyvindLGjesdal wants to merge 4 commits into
Conversation
The SimpleCharStream that javacc generates grows its buffer by 2048 chars at a time, so reading a long token, such as a large literal, takes quadratic time. javacc's master branch grows the buffer exponentially since this commit, from javacc PR 303: javacc/javacc@3bb3dbf javacc/javacc#303 It uses bufsize >> 1, which halves the buffer: javacc/javacc#316 Open javacc PR 317 corrects it to bufsize << 1: javacc/javacc#317 Neither is in a javacc release; the latest is 7.0.13. Jena uses its own SimpleCharStream, which doubles the buffer. Set the javacc option USER_CHAR_STREAM in main.jj, so javacc no longer generates SimpleCharStream.java for the ARQ and SPARQL 1.2 parsers. The parser class adds the InputStream and Reader constructors and ReInit operations that javacc leaves out with this option. SimpleCharStream.java is kept in the source tree. It is based on the javacc 7.0.12 output, from this template: https://github.com/javacc/javacc/blob/javacc-7.0.12/src/main/resources/templates/SimpleCharStream.template with these changes: - ExpandBuff doubles the buffer instead of adding 2048 chars, so reading a long token takes linear time. - The buffer grows to at most Integer.MAX_VALUE - 8 chars, the largest array length that is safe on common JVMs. A longer token is a TokenMgrError, which the query and update parsers report as a parse error. - Line and column are no longer stored for every buffered char. They are kept for the start of the token, and the end of the token is calculated when asked for. The buffer uses 2 bytes per char instead of 10. There is one copy of SimpleCharStream.java in each parser's package. TestSimpleCharStreamCopies checks that the copies are the same apart from the package. The parsers are regenerated in the next commit. This commit does not compile on its own: javacc generates the CharStream interface. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run jena-arq/Grammar/grammar with javacc 7.0.12 and GNU cpp after the USER_CHAR_STREAM change to main.jj. javacc now generates the CharStream interface, which SimpleCharStream implements. The parsers and token managers use CharStream in place of SimpleCharStream, and the InputStream and Reader constructors come from the parser class in main.jj. javacc names its lookahead methods after their line in the .jj file, and the lines added to main.jj move the grammar down, so most of the ARQParser.java diff is renamed methods. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
OyvindLGjesdal
marked this pull request as draft
November 16, 2026 21:33
Contributor
Author
|
I see that this is a something that can wait for a future release, if usable, since we are preparing for a new Jena version, but have been thinking about it. |
Contributor
|
Will this be removed if/when the next JavaCC version is released? Or if Jena moves to JavaCC 8? |
OyvindLGjesdal
force-pushed
the
GH-1324-bad-performance-when-parsing-huge-literals
branch
from
November 16, 2026 22:10
c89a4b2 to
d6cd74f
Compare
TestParseLargeLiteral parses a query with one literal of 1, 2, 10, 100 and 250MB, with the ARQ and the SPARQL 1.2 parsers. The literal is WKT polygon coordinates: in the issue, a SERVICE request with a geometry literal of about 100MB stalled in the parser. The same queries are parsed with Jena 5.6.0 up to 100MB. Its SimpleCharStream grows the buffer by 2048 chars at a time, so the parse time grows with the square of the literal's length: one parse of 100MB takes about a quarter of an hour. Jena 5.6.0 runs separately, with fewer iterations and a 30 minute timeout per iteration. With the buffer doubling, the time grows linearly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- lineColumnTo counted the chars of a token as (pos - tokenBegin + bufsize) % bufsize. That overflows int once the buffer is at its cap of Integer.MAX_VALUE - 8 chars, so line and column stopped advancing after a token of more than 2^30 chars. - An exception from the Reader, checked or unchecked, is now a TokenMgrError. The token manager takes an exception from the char stream as the end of the input, so the rest of the input was silently dropped: an update with a byte that is not valid UTF-8 (MalformedInputException) parsed as an empty update. Only the end of the input is an IOException, and the Reader, closed at the end of the input, is not read again. TestSimpleCharStream checks that the buffer grows by doubling: 8 expansions for a token of 1,000,000 chars after the buffer has wrapped round, where growing by 2048 chars takes 487. For both parsers, it checks that a failing Reader is an error, that an update with invalid UTF-8 is a parse error, that queries ending on a token the lexer reads past still parse, and the line and column of a parse error after a long multi-line literal. The grammar script removes CharStream.java before running javacc, so that it is regenerated. The script and Grammar/README say that SimpleCharStream.java is maintained in the source tree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
OyvindLGjesdal
force-pushed
the
GH-1324-bad-performance-when-parsing-huge-literals
branch
from
November 17, 2026 08:02
d6cd74f to
dba571e
Compare
Contributor
Author
Yes, the idea was to fix the parsing now, by using the USER_CHAR_STREAM mechanism. However I think CC8 has the same growth problem and that is was only fixed for 7 (without a release). Note that I think this PR solves the common case, but still has some parsers with the same issue:
|
OyvindLGjesdal
marked this pull request as ready for review
November 17, 2026 10:13
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GitHub issue resolved #1324
Pull request Description:
I also add a test that fails if the supplied SimpleCharStreams are no longer copies of each other, to stop them from drifting apart.
it could be worth redoing the second commit/javacc build with 7.0.13, 7,0.12 was chosen to use the same version as last time, to keep the difference minimal.
changes to code in first commit, generated from javacc in second commit, benchmark in third commit, and review changes in forth. Can squash if this looks correct and understandable.
By submitting this pull request, I acknowledge that I am making a contribution to the Apache Software Foundation under the terms and conditions of the Contributor's Agreement.