Skip to content

perf(macos): free each chunk the SQL import parser reads instead of holding the whole file - #4279

Merged
datlechin merged 2 commits into
mainfrom
fix/perf-parser-autorelease
Nov 13, 2026
Merged

datlechin merged 2 commits into
mainfrom
fix/perf-parser-autorelease

Conversation

@datlechin

@datlechin datlechin commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Summary

Performance audit 2026-11-11, MEM-02 rest (the parser part #4212 left for after #4210).

  • The SQL import parser read each 64 KB chunk with readData(ofLength:), an autoreleased NSData. A statement count, the SQL Server GO scan and a driver reading a COPY ... FROM STDIN section run many reads with nothing draining the thread's pool, so every chunk, and with the MySQL grammar every statement's text, stayed alive until the scan ended and for a while after.
  • Each step now drains its own pool: one per chunk read (readChunk), one per statement cut (readNextStatement), one per COPY piece (nextRowsPiece).
  • Reads use read(upToCount:), which throws on a read error. readData(ofLength:) raises an Objective-C exception there, which quits the app. Same change for the Import dialog's preview read.
  • Peak memory growth, 128 MB file, Debug build, phys_footprint sampled every 1 ms:
Case Before After
Count, PostgreSQL +129 MB +2 MB
Count, MySQL (keeps statement text) +300 MB +0 MB
Count, SQL Server (GO scan + count) +232 MB +5 MB
Import's first statement, SQL Server (GO scan) +130 MB +0 MB
Count, Latin-1 +131 MB +0 MB
Count, .sql.gz (decompress + count) +126 MB +1 MB
COPY rows read on a driver queue +132 MB +2 MB
Count of a file with a COPY +129 MB +0 MB
  • Before, the memory was still held 30 s after the count returned.
  • At 256 MB the PostgreSQL count was +260 MB before and +1 MB after: the peak no longer follows the file size.
  • Same statement counts and COPY bytes in every case.

Test plan

  • SQLFileParserTests: an empty file, a last statement with no ; when the file ends on a full read chunk, a UTF-8 file cut inside its last character. They pin end-of-file behavior across the read change, so they pass on main too.
  • Local: the parser suites (SQLFileParser*Tests, SQLScriptTextTests, ImportStatementCountTests) and the import suites (SqlFileImportSource*Tests, ImportService*Tests, ImportExecutionGateTests), 221 tests, all pass.
  • The memory numbers come from a local probe test that is not committed: a memory bound in the shared test host would flake.

@datlechin
datlechin enabled auto-merge (squash) November 13, 2026 15:51
@datlechin
datlechin disabled auto-merge November 13, 2026 16:23
@datlechin
datlechin enabled auto-merge (squash) November 13, 2026 16:23
@datlechin
datlechin disabled auto-merge November 13, 2026 16:23
@datlechin
datlechin merged commit 19722ac into main Nov 13, 2026
11 checks passed
@datlechin
datlechin deleted the fix/perf-parser-autorelease branch November 13, 2026 16:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant