Skip to content

GH-1324: bad performance when parsing huge literals - #4423

Open
OyvindLGjesdal wants to merge 4 commits into
apache:mainfrom
OyvindLGjesdal:GH-1324-bad-performance-when-parsing-huge-literals
Open

OyvindLGjesdal wants to merge 4 commits into
apache:mainfrom
OyvindLGjesdal:GH-1324-bad-performance-when-parsing-huge-literals

Conversation

@OyvindLGjesdal

@OyvindLGjesdal OyvindLGjesdal commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

GitHub issue resolved #1324

Pull request Description:

The SimpleCharStream that javacc generates grows its buffer by 2048 chars at a time, so parsing a long token, such as a 100MB literal, takes quadratic time. The javacc fix (javacc/javacc#303, corrected by the still open javacc/javacc#317) is not in a release.

  • The ARQ and SPARQL 1.2 grammars use USER_CHAR_STREAM with a Jena-maintained SimpleCharStream, based on the javacc 7.0.12 output. It doubles the buffer, up to Integer.MAX_VALUE - 8 chars (a longer token is a parse error), and no longer stores line and column for every buffered char. TestSimpleCharStreamCopies checks that the two package copies stay the same.
  • The parsers are regenerated with javacc 7.0.12 in a separate commit.
  • An error from the Reader, such as invalid UTF-8 in an update request, is a parse error. The token manager took it as the end of the input, so on main the rest of the update is silently dropped.
  • TestSimpleCharStream covers buffer growth, reader errors, the end of the input, and line and column after a long literal.
  • TestParseLargeLiteral in jena-benchmarks-jmh parses a query with a literal of 1MB to 250MB, and with Jena 5.6.0 up to 100MB.

JMH, ms per parse (this PR: average of 4; Jena 5.6.0: average of 2):

Literal Jena 5.6.0 This PR, ARQ This PR, SPARQL 1.2
1MB 290 142 144
2MB 821 247 253
10MB 9,657 1,149 1,122
100MB 855,886 (14 min) 11,351 11,062
250MB 27,791 27,988

I also add a test that fails if the supplied SimpleCharStreams are no longer copies of each other, to stop them from drifting apart.

it could be worth redoing the second commit/javacc build with 7.0.13, 7,0.12 was chosen to use the same version as last time, to keep the difference minimal.

changes to code in first commit, generated from javacc in second commit, benchmark in third commit, and review changes in forth. Can squash if this looks correct and understandable.

  • Tests are included.
  • Documentation change and updates are provided for the Apache Jena website
  • Commits have been squashed to remove intermediate development commit messages.
  • Key commit messages start with the issue number (GH-xxxx)

By submitting this pull request, I acknowledge that I am making a contribution to the Apache Software Foundation under the terms and conditions of the Contributor's Agreement.

OyvindLGjesdal and others added 2 commits November 16, 2026 21:53
The SimpleCharStream that javacc generates grows its buffer by 2048
chars at a time, so reading a long token, such as a large literal,
takes quadratic time. javacc's master branch grows the buffer
exponentially since this commit, from javacc PR 303:
javacc/javacc@3bb3dbf
javacc/javacc#303
It uses bufsize >> 1, which halves the buffer:
javacc/javacc#316
Open javacc PR 317 corrects it to bufsize << 1:
javacc/javacc#317
Neither is in a javacc release; the latest is 7.0.13. Jena uses its
own SimpleCharStream, which doubles the buffer.

Set the javacc option USER_CHAR_STREAM in main.jj, so javacc no longer
generates SimpleCharStream.java for the ARQ and SPARQL 1.2 parsers. The
parser class adds the InputStream and Reader constructors and ReInit
operations that javacc leaves out with this option.

SimpleCharStream.java is kept in the source tree. It is based on the
javacc 7.0.12 output, from this template:
https://github.com/javacc/javacc/blob/javacc-7.0.12/src/main/resources/templates/SimpleCharStream.template
with these changes:
- ExpandBuff doubles the buffer instead of adding 2048 chars, so
  reading a long token takes linear time.
- The buffer grows to at most Integer.MAX_VALUE - 8 chars, the largest
  array length that is safe on common JVMs. A longer token is a
  TokenMgrError, which the query and update parsers report as a parse
  error.
- Line and column are no longer stored for every buffered char. They
  are kept for the start of the token, and the end of the token is
  calculated when asked for. The buffer uses 2 bytes per char
  instead of 10.

There is one copy of SimpleCharStream.java in each parser's package.
TestSimpleCharStreamCopies checks that the copies are the same apart
from the package.

The parsers are regenerated in the next commit. This commit does not
compile on its own: javacc generates the CharStream interface.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run jena-arq/Grammar/grammar with javacc 7.0.12 and GNU cpp after the
USER_CHAR_STREAM change to main.jj.

javacc now generates the CharStream interface, which SimpleCharStream
implements. The parsers and token managers use CharStream in place of
SimpleCharStream, and the InputStream and Reader constructors come from
the parser class in main.jj.

javacc names its lookahead methods after their line in the .jj file,
and the lines added to main.jj move the grammar down, so most of the
ARQParser.java diff is renamed methods.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@OyvindLGjesdal OyvindLGjesdal changed the title Gh 1324 bad performance when parsing huge literals GH-1324: bad performance when parsing huge literals Nov 16, 2026
@OyvindLGjesdal OyvindLGjesdal added the AI Assisted Assisted or generated by AI label Nov 16, 2026
@OyvindLGjesdal
OyvindLGjesdal marked this pull request as draft November 16, 2026 21:33
@OyvindLGjesdal

Copy link
Copy Markdown
Contributor Author

I see that this is a something that can wait for a future release, if usable, since we are preparing for a new Jena version, but have been thinking about it.

@ThomasThelen

ThomasThelen commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Will this be removed if/when the next JavaCC version is released? Or if Jena moves to JavaCC 8?

@OyvindLGjesdal
OyvindLGjesdal force-pushed the GH-1324-bad-performance-when-parsing-huge-literals branch from c89a4b2 to d6cd74f Compare November 16, 2026 22:10
OyvindLGjesdal and others added 2 commits November 17, 2026 09:02
TestParseLargeLiteral parses a query with one literal of 1, 2, 10, 100
and 250MB, with the ARQ and the SPARQL 1.2 parsers. The literal is WKT
polygon coordinates: in the issue, a SERVICE request with a geometry
literal of about 100MB stalled in the parser.

The same queries are parsed with Jena 5.6.0 up to 100MB. Its
SimpleCharStream grows the buffer by 2048 chars at a time, so the
parse time grows with the square of the literal's length: one parse of
100MB takes about a quarter of an hour. Jena 5.6.0 runs separately,
with fewer iterations and a 30 minute timeout per iteration. With the
buffer doubling, the time grows linearly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- lineColumnTo counted the chars of a token as
  (pos - tokenBegin + bufsize) % bufsize. That overflows int once the
  buffer is at its cap of Integer.MAX_VALUE - 8 chars, so line and
  column stopped advancing after a token of more than 2^30 chars.
- An exception from the Reader, checked or unchecked, is now a
  TokenMgrError. The token manager takes an exception from the char
  stream as the end of the input, so the rest of the input was
  silently dropped: an update with a byte that is not valid UTF-8
  (MalformedInputException) parsed as an empty update. Only the end
  of the input is an IOException, and the Reader, closed at the end of
  the input, is not read again.

TestSimpleCharStream checks that the buffer grows by doubling: 8
expansions for a token of 1,000,000 chars after the buffer has wrapped
round, where growing by 2048 chars takes 487. For both parsers, it
checks that a failing Reader is an error, that an update with invalid
UTF-8 is a parse error, that queries ending on a token the lexer reads
past still parse, and the line and column of a parse error after a long
multi-line literal.

The grammar script removes CharStream.java before running javacc, so
that it is regenerated. The script and Grammar/README say that
SimpleCharStream.java is maintained in the source tree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@OyvindLGjesdal
OyvindLGjesdal force-pushed the GH-1324-bad-performance-when-parsing-huge-literals branch from d6cd74f to dba571e Compare November 17, 2026 08:02
@OyvindLGjesdal

Copy link
Copy Markdown
Contributor Author

Will this be removed if/when the next JavaCC version is released? Or if Jena moves to JavaCC 8?

Yes, the idea was to fix the parsing now, by using the USER_CHAR_STREAM mechanism. However I think CC8 has the same growth problem and that is was only fixed for 7 (without a release).

Note that I think this PR solves the common case, but still has some parsers with the same issue:

  • SPARQL 1.1 parser: jena-arq/.../sparql/lang/sparql_11/javacc/JavaCharStream.java (only when chosen)
  • SPARQL 1.0 parser: jena-arq/.../sparql/lang/sparql_10/javacc/JavaCharStream.java (only when chosen)
  • SSE parser: jena-arq/.../sparql/sse/lang/javacc/SimpleCharStream.java (algebra)
  • CDT literal parser: jena-arq/.../cdt/javacc/SimpleCharStream.java (could be large lists and sets in CDTS?)
  • javacc Turtle parser: jena-arq/.../riot/lang/turtlejcc/javacc/SimpleCharStream.java
  • SHACL compact syntax: jena-shacl/.../shacl/compact/reader/javacc/SimpleCharStream.java (on by default)
  • ShEx compact syntax: jena-shex/.../shex/parser/javacc/SimpleCharStream.java (on by default)

@OyvindLGjesdal
OyvindLGjesdal marked this pull request as ready for review November 17, 2026 10:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI Assisted Assisted or generated by AI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Poor performance when parsing huge literal in query (e.g. 100MB)

2 participants