[IO] Fast Streaming Scanner for Turtle and TriG data ingestion #638

Summary

Design and implement a high-throughput, low-allocation streaming scanner for Turtle and TriG RDF formats to eliminate the ANTLR parsing bottleneck on bulk data loading.

Context & Motivation

Following the replacement of ANTLR with high-throughput streaming scanners for N-Triples and N-Quads (ingestion runtime dropped from 47.7 s to 11.1 s, 4x speedup), rich syntax formats (Turtle and TriG) remain bound to heavyweight AST instantiation. Real-world benchmarks (DBpedia, Wikidata, BSBM) require streaming parsers matching industry standards (Jena RIOT, RDF4J Rio). Note: ANTLR is strictly maintained for SPARQL query grammar.

Technical Scope & Objectives

1. Low-Allocation State Machine (FSM):
   • Implement an event-driven byte-streaming scanner for Turtle/TriG directives (@prefix, PREFIX, BASE) and triples/quads.
   • Maintain a lightweight context stack to handle nested blank nodes [ ... ] and RDF collections ( ... ).
2. Direct Term Resolution:
   • Stream resolved terms directly into DenseTermTable and InMemoryStatementStore without allocating intermediate AST nodes.
3. Conformance & Robustness:
   • Validate against standard W3C Turtle and TriG test suites, handling multiline strings and escape sequences.

Acceptance Criteria

• Streaming parser operates on input streams with minimal memory allocations.
• 100% pass on official W3C Turtle and TriG syntax test suites.
• Ingestion throughput on Turtle datasets increases significantly towards 150k-200k triples/s.