Repository navigation
perf(plugins): read row streams only as fast as copies, compares and exports take them - #4227
Merged
Merged
Conversation
datlechin
enabled auto-merge (squash)
November 12, 2026 06:21
datlechin
disabled auto-merge
November 12, 2026 06:22
datlechin
enabled auto-merge (squash)
November 12, 2026 06:22
# Conflicts: # apple/Plugins/MemcachedDriverPlugin/MemcachedPluginDriver.swift
# Conflicts: # apple/Plugins/DuckDBDriverPlugin/DuckDBConnection.swift # apple/Plugins/HanaDriverPlugin/HanaPluginDriver.swift # apple/Plugins/MSSQLDriverPlugin/FreeTDSConnection.swift # apple/Plugins/MSSQLDriverPlugin/MSSQLPlugin.swift # apple/Plugins/MySQLDriverPlugin/MariaDBPluginConnection.swift # apple/Plugins/MySQLDriverPlugin/MySQLPluginDriver.swift # apple/Plugins/PostgreSQLDriverPlugin/LibPQPluginConnection.swift # apple/TablePro/Core/Compare/CompareRowService.swift # apple/TablePro/Core/Compare/StreamingRowProvider.swift
…hips, abort only when the consumer leaves
# Conflicts: # apple/Plugins/SnowflakeDriverPlugin/SnowflakeConnection.swift
…the agent guard, add a SQLite stream probe
…lifts the timeout before its streams
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PluginRowStream.makeBounded,deliverBlocking,deliver), so a copy, transfer, compare or slow export no longer holds the whole source table: SQLite (local), LibSQL (local), Cassandra, Oracle, Teradata, DynamoDB, InfluxDB, Snowflake, BigQuery, Bigtable, Firebase, etcd, Qdrant, Weaviate, Typesense, Redis (SCAN and KEYS), Memcached, Neo4j.deliver, tails included, since a full bounded buffer drops a plain yield.timeoutIntervalForResource(measured)..cancelled), not when the producer finishes. Every finished read ran its producer's cancel actions: a ClickHouse grid read sent KILL QUERY after each read (20 KILL QUERY requests per 20 finished reads before, 0 after, counted on the test stub), and HANA, InfluxDB, SQL Server, Memcached, Oracle, Spanner, Teradata and Trino cancelled their finished tasks.CompareRowService.readingBothSides): a stream starts its statement when built, and on PostgreSQL, MySQL, SQL Server and HANA theSETwaited behind a bounded stream parked at a full buffer, so a compare of more than 16 batches hung.TransactionEngineFamily.failedStatementAbortsTransactionandAgentTransactionGuard.Savepoint.ofhad no.teradatacase after feat(macos): add manual commit, table auto refresh, query tab history, server objects in the sidebar and composite ER foreign keys #4214 and fix(drivers): keep Oracle values typed, fix Teradata ANSI transactions, complete SQL Server, Oracle and PostgreSQL DDL, and bound driver streams #4223); the hunks are fix(macos): give Teradata its answer for a failed statement in an open transaction #4237's, byte for byte, so Teradata keeps the answers it had as.other.Needs the maintainer
http_send_timeout30 s, point in time1m,query.client.timeout5 min,cursorTimeoutMillis10 min, an idle read-write transaction); they need a pacing mode that reads on past it. SQLite over the SSH agent stays unbounded too: a waiting reader stops the relay forwarding the agent's heartbeats.failedStatementAbortsTransaction == false. Teradata mode (BTET) rolls back the whole transaction on a failed request, and DEFAULT follows the server's mode, sotruewould protect a user's open transaction from an agent write.Test plan
apple/scripts/probes/check-stream-backpressure.sh sqlite): peak footprint over baseline 244.9 MB on main, 33.2 MB here, 6.2 s and 5.8 s.PluginRowStreamTerminationTests(a finished or failed stream runs no abort action; a dropped stream or a cancelled consumer does; the first two fail on main),MaskingExportTests.exportResultsLoadsValuesAsTheWriterPulls,CompareTimeoutLiftOrderTests(a stream of 3 xboundedBufferBatcheson a connection queue the lift also needs: passes here, hangs past its time limit with the lift after the streams, as before).verify.sh: build, AllPlugins, abi, lint, Unit suites for streams, export, copy, transfer, compare and the bundled drivers, TableProCore (PluginKit, DuckDB, Qdrant), TableProOracle.