Skip to content

refactor(sql): read every SQL span through SQLNonCodeSpan and run one shared script corpus against the core - #4289

Merged
datlechin merged 10 commits into
mainfrom
refactor/one-sql-lexer-4271
Nov 15, 2026
Merged

datlechin merged 10 commits into
mainfrom
refactor/one-sql-lexer-4271

Conversation

@datlechin

@datlechin datlechin commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Import (SQLFileParser) reads every comment, literal, quoted name and SQLite parameter with SQLNonCodeSpan on its chunk buffer. Its own quote, comment, dollar and q'' state machine is gone (SQLFileParser+Literals.swift deleted).
  • Import misreads fixed: a PostgreSQL E'...' string continued after a line break, a SQLite $name(...) parameter holding a quote, a dropped comment gluing its neighbours (a/* x */FROM sent as aFROM), a client delimiter of 3+ units cut by a chunk.
  • A span the end of a chunk can still change waits for more text (SQLNonCodeSpan.needsMoreText), and the next read at least doubles what is held, so a literal of many megabytes is lexed a few times, not once per chunk.
  • A single string or comment longer than 128 Mi units, the core reader's statement limit, is refused with its line (SQLFileParseError.spanTooLong), by the count and the import, instead of being held in memory.
  • SQL diagnostics (QueryBracketScanner) read with the session's grammar: no false "No matching opening bracket" inside $$...$$, [a)], a MySQL # comment, PostgreSQL 'C:\' or a nested comment; a ) inside a run /*! */ is code and its */ is not.
  • The query fingerprint tokenizes with the engine's grammar; the grammar-blind tokenizer, its dollar stripper and bracket merger are deleted. A psql COPY ... FROM STDIN keeps its command in the shape, a SurrealQL ⟨name⟩ stays a name, and @@version stays a variable. History and plan snapshots group and match by a new shape_hash column the backfill fills. fingerprint_hash keeps the previous stable's reading, from a frozen copy (SQLLegacyFingerprint, pinned to v0.86.0's shapes) that nothing else reads and that goes once no supported release reads the column.
  • iOS: the Roll Back check skips comments by the connection's grammar (# undo on MySQL, nested comments on PostgreSQL).
  • One shared corpus, core/crates/core/tests/fixtures/sql_script_corpus.json (52 cases: strings, comments, dollar quotes, DELIMITER, GO batches, per dialect), runs in the core (ScriptPlan, and ScriptReader cut 1, 2, 7 and whole), the statement scanner, the import cut 1, 7 and 64 KiB, and the kit's lexer. The core reads every case right, so it is unchanged.
  • The kit keeps SQLFeatureLexer and SQLiteSQLScanner: a plugin cannot link the grammar package. The corpus now holds SQLFeatureLexer to the app's lexer.
  • detect-changed-paths.sh: macos and ios run when the corpus changes.
  • Tests changed on purpose: SQLFingerprintGoldenTests and systemVariableFingerprint (shapes now follow the engine's grammar), QueryHistoryMigrationTests (plan snapshots gain shape_hash), SQLTokenizerTests (grammarless cases name their grammar), QueryDiagnosticsTests and TransactionRollBackClassifierTests (take a grammar).

Test plan

  • New: SQLScriptSharedCorpusTests, SQLFileParserSharedCorpusTests, script_shared_corpus.rs, SQLLexicalFeatureMappingTests.kitLexerAgreesOnTheSharedCorpus, SQLNonCodeSpanTests buffer-end cases, QueryDiagnosticsTests grammar cases, SQLFileParserTests continuation, Tcl parameter, comment, long tag, split comment, long delimiter and long literal cases, SQLFileParserTests.spanLongerThanTheLimitIsRefused, QueryHistoryFingerprintMigrationTests.shapeColumnLeavesThePreviousStableItsFingerprint, QueryPlanSnapshotStorageTests.planKeepsThePreviousStablesFingerprint, SQLLegacyFingerprintTests, TransactionRollBackClassifierTests.commentsFollowTheGrammar.
  • Local: Unit plan, swift test for TableProSQLGrammar, TableProAutocomplete, TableProPluginKit and TableProQuery, cargo test -p tablepro-core --test script_shared_corpus and clippy. The iOS suite runs in CI only: this Mac has no iOS simulator runtime.

Closes #4271

@datlechin
datlechin merged commit b451ff2 into main Nov 15, 2026
46 of 47 checks passed
@datlechin
datlechin deleted the refactor/one-sql-lexer-4271 branch November 15, 2026 07:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SQL lexers: remaining audit items

1 participant