Skip to content

perf(plugins): pace Elasticsearch, Trino and MongoDB streams and keep the server's side alive while the reader waits - #4245

Merged
datlechin merged 4 commits into
mainfrom
fix/perf-idle-pacing
Nov 12, 2026
Merged

datlechin merged 4 commits into
mainfrom
fix/perf-idle-pacing

Conversation

@datlechin

@datlechin datlechin commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Elasticsearch export, Trino/Presto and MongoDB row streams now read only as fast as a copy, transfer, compare or export takes rows (PluginRowStream.makeBounded, 16 batches). Part of MEM-01 that perf(plugins): read row streams only as fast as copies, compares and exports take them #4227 left open (FU-02).
  • While the buffer is full, each stream keeps its server's side alive the way that engine documents it, through new additive PluginKit overloads deliver / deliverBlocking(..., keepingAliveEvery:_:). Kit stays 44.
    • Elasticsearch: a size: 0 search names the point in time every 30 s (keep-alive 1m). Each search uses the newest pit_id, as the PIT docs require.
    • Trino 475+: HEAD on the next page's URI every 30 s, as the official client does. Older Trino and Presto: the page already received is asked for again, which the coordinator answers from its cache and counts as contact. A HEAD answered 404, 405 or 501 switches to that.
    • MongoDB: each stream runs in its own session and sends refreshSessions every 5 minutes (a sixth of a shorter session timeout), the manual's way to hold a cursor idle. From 4.4.8 a session's cursors no longer expire after cursorTimeoutMillis; before it a find adds noCursorTimeout. With no sessions, a refused refreshSessions (DocumentDB, Cosmos DB) or an aggregate before 4.4.8, the stream reads as fast as the server sends, as before.
  • MongoDB streams send up to 1,000 documents (4 MB) per element instead of one, and free each document's autoreleased objects as they go.
  • Elasticsearch export pages send allow_partial_search_results=false: a shard that lost its context answered 200 without its hits, so an export could miss documents without a word.
  • A Trino reader that stops taking pages now ends the query with DELETE.
  • Probe: apple/scripts/probes/check-stream-idle-pacing.sh (Elasticsearch, MongoDB, Trino on real servers).
  • Docs: docs(databases): say an Elasticsearch export stops when a shard cannot answer docs#225.

Stays unbounded:

  • ClickHouse: the HTTP response's send timeout is the default profile's http_send_timeout at server start (30 s since 23.7). HTTPServerRequest sets it on the socket before any URL setting is read, so no request can raise it, and nothing keeps a stalled response alive.
  • Spanner: a read-only stream has no idle limit, but it must finish within version_retention_period (1 h by default) from its start. A paced read lasts as long as its reader, so a slow copy would fail with FAILED_PRECONDITION where it finishes now.

Needs the maintainer

  • Spanner: pace it and let reads longer than the retention period fail (the user can raise version_retention_period), or keep it unbounded (this PR).
  • A MongoDB with enableTimeoutOfInactiveSessionCursors=true, or a compatible server that accepts refreshSessions yet ends idle cursors (Azure DocumentDB vCore), fails a paused read with CursorNotFound.

Test plan

  • Peak footprint over baseline, reader slower than the server, docker servers, main then this branch:
    • MongoDB 8.0, 1,000,000 documents, 20 ms per 1,000 rows: 879.1 MB to 107.1 MB.
    • Elasticsearch 8.19.7, 300,000 documents, 200 ms per 1,000: 69.8 MB to 6.4 MB.
    • Trino 484, tpch.sf1.orders (1.5M rows), 40 ms per 1,000: 292.5 MB to 73.0 MB.
  • Reader holds the stream past the idle limit after the first batch:
    • MongoDB, session timeout 1 min: a session cursor idle 150 s gets CursorNotFound. The paced read held 180 s reads all 200,000 rows; a 200 s read at 1 ms per row also completes.
    • Elasticsearch: a 1m point in time idle 90 s gets 404. The paced export held 150 s reads all 100,000.
    • Trino 484, query.client.timeout=1m: a page held 120 s without contact is ABANDONED_QUERY, and asking for it again every 30 s keeps the query. The paced read held 150 s on HEAD heartbeats reads all 1.5M rows.
  • New: TrinoHeartbeatTests (a stub coordinator that abandons after 5 minutes without contact: a page held 20 minutes survives on HEAD, on the fallback and on the repeated page, and is abandoned without heartbeats; a reader that stops sends DELETE), keep-alive cases in PluginRowStreamDeliveryTests.
  • verify.sh: build, plugins, abi (changed, kit 44), lint, PluginKit stream and Trino core suites (42 passed).

@datlechin
datlechin enabled auto-merge (squash) November 12, 2026 19:46
@datlechin
datlechin merged commit 40db66c into main Nov 12, 2026
11 checks passed
@datlechin
datlechin deleted the fix/perf-idle-pacing branch November 13, 2026 02:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant