Repository navigation
perf(plugins): pace Elasticsearch, Trino and MongoDB streams and keep the server's side alive while the reader waits - #4245
Merged
Merged
Conversation
… the server's side alive while the reader waits
datlechin
enabled auto-merge (squash)
November 12, 2026 19:46
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PluginRowStream.makeBounded, 16 batches). Part of MEM-01 that perf(plugins): read row streams only as fast as copies, compares and exports take them #4227 left open (FU-02).deliver/deliverBlocking(..., keepingAliveEvery:_:). Kit stays 44.size: 0search names the point in time every 30 s (keep-alive 1m). Each search uses the newestpit_id, as the PIT docs require.refreshSessionsevery 5 minutes (a sixth of a shorter session timeout), the manual's way to hold a cursor idle. From 4.4.8 a session's cursors no longer expire aftercursorTimeoutMillis; before it a find addsnoCursorTimeout. With no sessions, a refusedrefreshSessions(DocumentDB, Cosmos DB) or an aggregate before 4.4.8, the stream reads as fast as the server sends, as before.allow_partial_search_results=false: a shard that lost its context answered 200 without its hits, so an export could miss documents without a word.apple/scripts/probes/check-stream-idle-pacing.sh(Elasticsearch, MongoDB, Trino on real servers).Stays unbounded:
http_send_timeoutat server start (30 s since 23.7).HTTPServerRequestsets it on the socket before any URL setting is read, so no request can raise it, and nothing keeps a stalled response alive.version_retention_period(1 h by default) from its start. A paced read lasts as long as its reader, so a slow copy would fail withFAILED_PRECONDITIONwhere it finishes now.Needs the maintainer
version_retention_period), or keep it unbounded (this PR).enableTimeoutOfInactiveSessionCursors=true, or a compatible server that acceptsrefreshSessionsyet ends idle cursors (Azure DocumentDB vCore), fails a paused read with CursorNotFound.Test plan
tpch.sf1.orders(1.5M rows), 40 ms per 1,000: 292.5 MB to 73.0 MB.query.client.timeout=1m: a page held 120 s without contact isABANDONED_QUERY, and asking for it again every 30 s keeps the query. The paced read held 150 s on HEAD heartbeats reads all 1.5M rows.TrinoHeartbeatTests(a stub coordinator that abandons after 5 minutes without contact: a page held 20 minutes survives on HEAD, on the fallback and on the repeated page, and is abandoned without heartbeats; a reader that stops sends DELETE), keep-alive cases inPluginRowStreamDeliveryTests.verify.sh: build, plugins, abi (changed, kit 44), lint, PluginKit stream and Trino core suites (42 passed).