Skip to content

perf(macos): move result, grid and editor work off the main actor, stream exports and imports, and cap pool and driver reads - #4181

Merged
datlechin merged 2 commits into
mainfrom
perf/audit-wave-3-macos
Nov 11, 2026
Merged

datlechin merged 2 commits into
mainfrom
perf/audit-wave-3-macos

Conversation

@datlechin

Copy link
Copy Markdown
Member

Summary

Performance fixes from the 2026-11-10 code audit (wave 3, Phase 4) for the macOS app. Results stay the same; the work moves off the main actor, becomes incremental, streams, or is capped.

Grid

  • PF-02: with Find open, an edit, insert, removal or undo re-reads only the changed cells and the rows a value filter shows again, then merges; anything else the search depends on falls back to a full search. Replace and Replace All no longer scan twice.
  • GE-05: a row's pending change is looked up by index; undo repaints only the rows each step names. Copy as INSERT/UPDATE and loading unloaded values use the same index.
  • PF-05: a cell edit mutates the tab's row buffer in place instead of copying every row.
  • PF-08: Filter Values sorts off the main actor and opens when done, only while its header is in a visible window; a reload, edit or reopen cancels a pending sort.

Results

  • PF-03: Fetch All, the first page, parameterized runs and the SQL Server batch path build rows off the main actor (TableRows.preparing); a mask or Reveal change in between rebuilds on the main actor as before. Stop ends the build within 4,096 rows.
  • PF-04 (partly): result chooser names are counted once, not per entry; Run All builds each statement's rows off the main actor. Collapsing column-less results and capping how many a run keeps change what the user sees, so they are left.

Drivers and pool

  • DL-04 (partly): the metadata pool holds its 6 connections per connection. Callers that need a new one wait in line, interactive before bulk, with one place kept for interactive work; a nested lease of the same work skips the line only while its lease lasts. Pool keys still include the schema.
  • DL-06 (partly): Stop on an export from a driver without its own row stream cancels the query through that driver's cancel; PluginKit's per-table defaults stop once cancelled. HANA, Teradata, InfluxDB, R2 SQL and Memcached still load the whole result first.
  • HANA export over 5,000,000 rows reported success: PluginKit's default stream now carries the cut in the header (PluginStreamHeader.rowLimit, additive), and export warns as for other drivers.
  • DL-07: fetchForeignKeys(forTables:) reads only the tables asked for unless the driver has a bulk query.
  • SD-08: a capped read inside a PostgreSQL transaction runs as SAVEPOINT, DECLARE, FETCH cap+1, MOVE FORWARD ALL, CLOSE, RELEASE in one round trip, so only cap+1 rows cross the wire and side effects match the old drain. Used only after a savepoint probe succeeds on the session; Redshift and CockroachDB keep the old path. An export cancelled inside a transaction rolls back to its savepoint.
  • SD-12: Oracle streams rows 5,000 per element, the first row alone; the cap stays exact.
  • ND-08: etcd pages, filters, export and the table list read in batches pinned to the first batch's revision; a compacted revision restarts a page once, and fails an export with a plain error.
  • SD-10 dropped: fix(macos): stop the data loss, wrong writes and gate bypasses the code audit found #4174 already stops the DuckDB stream when the consumer leaves.

Import and export

  • IE-09: XLSX sheets stream to temp files with at most 1 MB plus a row in memory; output is byte-identical (pinned by SHA-256). The 4 GB check runs per flush. More than 65,535 ZIP entries throws instead of trapping. Folders left by a crashed export are swept at the next export.
  • IE-08: JSON import reads UTF-8 files in chunks and parses one element at a time; an element that does not parse fails the import as before; a non-object element is now reported and counted as skipped. UTF-16 and UTF-32 files still read whole.

Editor and sync

  • ES-12: the statement band resumes from the last statement boundary before the caret instead of scanning from offset 0; running, Cmd+Enter and statement navigation still scan from the top.
  • ES-13: column suggestions before FROM are built once per column cache, table list and schema.
  • CS-11: pulled connections apply to one in-memory list and write once; a record that reads or writes the library flushes first. A failed write now applies none of the batch, and the token is withheld as before.

Behavior changes to know

  • Inside a transaction a capped PostgreSQL query without ORDER BY may return rows in another order (cursor planning), and costs one more round trip; the probe costs one per connection.
  • An error in a row past the cap now shows as that error; before, the rows showed and the transaction was left aborted without a word. A Stop on the driver's own probe or SAVEPOINT sends nothing of the user's statement; a Stop during the read sends the server cancel.
  • An etcd export that outlives the server's compaction interval fails and has to run again.
  • A large metadata refresh on a far server takes longer, since it now waits for pool places.

Tests changed on purpose

  • trimClosesOnlyIdleConnections replaced by waitingLeaseTakesOnlyAnIdlePlace (the cap is now hard).
  • unreadableElementFollowsTheErrorMode replaced by unreadableElementFailsTheImport(errorHandling:).
  • XLSX writer beginSheet, addRows, finishSheet, continueSheet now throw; their callers and tests gained try.

PluginKit

  • Additive: PluginStreamHeader.rowLimit and new default bodies. Kit stays 43 (unreleased), minimum 19; the ABI check passes against v0.86.0-beta.1.

Not done here

  • AR-04 (unit tests on the app host): test infrastructure.
  • Typing a Find term and formatting Filter Values rows still run on the main actor: CellDisplayFormatter is main-actor only.
  • MongoDB and Redis statement highlighting still rescan; the iOS completion catalog rebuilds as ES-13 did.
  • Teradata's cancel closes the socket (its abort message is never sent); MongoDB's stream does not set rowLimit at 5,000,000; Object Copy and Data Compare can still copy a HANA table short without saying so.

Test plan

  • verify.sh: build, AllPlugins, the PluginKit ABI check, the Unit plan, TableProCore and TableProOracle package tests, an iOS build with DataBrowserViewModelTests, lint and the catalog check.
  • New suites include TableRowsPreparedRowsTests, ResultRowsBuildOffMainTests, ResultSetMenuModelTests, UndoStepRepaintTests, ForeignKeysForTablesReadTests, PluginDriverDefaultCancellationTests, PluginDefaultStreamRowLimitTests, ExportDriverRowCutTests, LibPQCappedCursorReadTests, OracleRowBatcherTests, EtcdRangeScanTests, SyncConnectionPullBatchTests, plus cases in GridFindEngineTests, RowVisualIndexTests, TabSessionRegistryTests, DataGridValueFilterTests, MetadataConnectionPoolCapacityTests, the XLSX and JSON tests. Each pins work done (cells read, rows built off main, reads issued, bytes held), never time.
  • SD-08's statement sequence was run by the reviewer against a scratch PostgreSQL 17.11 server: side effects of a capped SELECT bump(g), nextval('s') all kept, a refused DECLARE runs nothing, a cancel during MOVE ends the tail. LibPQStopBeforeStatementTests checks what reaches a loopback server on Stop. etcd was not run against a live server.
  • Reviewed in two or three rounds per area by Claude agents that did not write the change.

@datlechin
datlechin enabled auto-merge (squash) November 11, 2026 01:51
@datlechin
datlechin merged commit 754a3f7 into main Nov 11, 2026
11 checks passed
@datlechin
datlechin deleted the perf/audit-wave-3-macos branch November 11, 2026 07:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant