Skip to content

perf(plugins): read row streams only as fast as copies, compares and exports take them - #4227

Merged
datlechin merged 9 commits into
mainfrom
fix/perf-streams
Nov 12, 2026
Merged

datlechin merged 9 commits into
mainfrom
fix/perf-streams

Conversation

@datlechin

@datlechin datlechin commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

Summary

  • MEM-01: the rest of the streaming drivers now read through the bounded stream fix(drivers): keep Oracle values typed, fix Teradata ANSI transactions, complete SQL Server, Oracle and PostgreSQL DDL, and bound driver streams #4223 added (PluginRowStream.makeBounded, deliverBlocking, deliver), so a copy, transfer, compare or slow export no longer holds the whole source table: SQLite (local), LibSQL (local), Cassandra, Oracle, Teradata, DynamoDB, InfluxDB, Snowflake, BigQuery, Bigtable, Firebase, etcd, Qdrant, Weaviate, Typesense, Redis (SCAN and KEYS), Memcached, Neo4j.
    • Every element after the first goes through deliver, tails included, since a full bounded buffer drops a plain yield.
    • Inner buffers that read ahead are gone: Oracle core hands rows to an awaited callback, Snowflake keeps at most 4 chunks downloading or held, Qdrant scrolls one page per page taken, etcd, Firebase, Typesense and Neo4j wait in their page callbacks.
    • InfluxDB and Typesense send the paced read on a second session without the one-hour resource cap: a suspended URLSession task still counts toward timeoutIntervalForResource (measured).
  • MEM-04: DynamoDB reads its spool back only as the consumer takes rows, and each 1 MB read drains in its own autorelease pool.
  • A stream's abort now fires only when the consumer ends it (.cancelled), not when the producer finishes. Every finished read ran its producer's cancel actions: a ClickHouse grid read sent KILL QUERY after each read (20 KILL QUERY requests per 20 finished reads before, 0 after, counted on the test stub), and HANA, InfluxDB, SQL Server, Memcached, Oracle, Spanner, Teradata and Trino cancelled their finished tasks.
    • Spanner now drops its Stop entry when a stream task ends, since a finished stream no longer aborts.
  • Export Results reads unloaded values one chunk per pull instead of a task filling every chunk ahead of the writer.
  • Data Compare lifts the query timeout before it builds its streams (CompareRowService.readingBothSides): a stream starts its statement when built, and on PostgreSQL, MySQL, SQL Server and HANA the SET waited behind a bounded stream parked at a full buffer, so a compare of more than 16 batches hung.
  • Fixed on the way: a ClickHouse stream could stop for good when its reader finished a chunk before the transfer was suspended; Beancount and Cassandra never cancelled their task when the consumer left; main did not build (TransactionEngineFamily.failedStatementAbortsTransaction and AgentTransactionGuard.Savepoint.of had no .teradata case after feat(macos): add manual commit, table auto refresh, query tab history, server objects in the sidebar and composite ER foreign keys #4214 and fix(drivers): keep Oracle values typed, fix Teradata ANSI transactions, complete SQL Server, Oracle and PostgreSQL DDL, and bound driver streams #4223); the hunks are fix(macos): give Teradata its answer for a failed statement in an open transaction #4237's, byte for byte, so Teradata keeps the answers it had as .other.
  • PluginKit stays kit 44; nothing public was added or removed.

Needs the maintainer

  • MEM-01 partly: ClickHouse, Elasticsearch, Trino, MongoDB and Spanner stay unbounded. Bounded pacing would drop the connection past the server's idle limit (http_send_timeout 30 s, point in time 1m, query.client.timeout 5 min, cursorTimeoutMillis 10 min, an idle read-write transaction); they need a pacing mode that reads on past it. SQLite over the SSH agent stays unbounded too: a waiting reader stops the relay forwarding the agent's heartbeats.
  • Teradata keeps failedStatementAbortsTransaction == false. Teradata mode (BTET) rolls back the whole transaction on a failed request, and DEFAULT follows the server's mode, so true would protect a user's open transaction from an agent write.

Test plan

  • SQLite (local file), 1,000,000 rows of about 200 bytes, consumer waiting 20 ms per batch (apple/scripts/probes/check-stream-backpressure.sh sqlite): peak footprint over baseline 244.9 MB on main, 33.2 MB here, 6.2 s and 5.8 s.
  • DynamoDB read-back loop over a 200 MB spool: footprint growth 208 MB before, 6 MB after.
  • New: PluginRowStreamTerminationTests (a finished or failed stream runs no abort action; a dropped stream or a cancelled consumer does; the first two fail on main), MaskingExportTests.exportResultsLoadsValuesAsTheWriterPulls, CompareTimeoutLiftOrderTests (a stream of 3 x boundedBufferBatches on a connection queue the lift also needs: passes here, hangs past its time limit with the lift after the streams, as before).
  • verify.sh: build, AllPlugins, abi, lint, Unit suites for streams, export, copy, transfer, compare and the bundled drivers, TableProCore (PluginKit, DuckDB, Qdrant), TableProOracle.

# Conflicts:
#	apple/Plugins/MemcachedDriverPlugin/MemcachedPluginDriver.swift
# Conflicts:
#	apple/Plugins/DuckDBDriverPlugin/DuckDBConnection.swift
#	apple/Plugins/HanaDriverPlugin/HanaPluginDriver.swift
#	apple/Plugins/MSSQLDriverPlugin/FreeTDSConnection.swift
#	apple/Plugins/MSSQLDriverPlugin/MSSQLPlugin.swift
#	apple/Plugins/MySQLDriverPlugin/MariaDBPluginConnection.swift
#	apple/Plugins/MySQLDriverPlugin/MySQLPluginDriver.swift
#	apple/Plugins/PostgreSQLDriverPlugin/LibPQPluginConnection.swift
#	apple/TablePro/Core/Compare/CompareRowService.swift
#	apple/TablePro/Core/Compare/StreamingRowProvider.swift
# Conflicts:
#	apple/Plugins/SnowflakeDriverPlugin/SnowflakeConnection.swift
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant