Repository navigation
feat(scheduler): drop a session's DataFrame::cache() data when the session is removed - #2889
akshaychitneni wants to merge 1 commit into
Conversation
…ssion is removed Removing a session (a gRPC RemoveSession call, or a Flight SQL session expiring) now drops its caches: built ones leave the registry at once, builds in flight are cancelled and what waits on them fails with "its session was closed", and using the session again rebuilds them. A dropped cache's data is now deleted once no running query reads it, after the usual delay, rather than a fixed delay after the drop. This applies to every way a cache is dropped, so a healthy cache dropped by another query's failure is no longer deleted under a query still reading it.
|
@andygrove @milenkovicm @comphead could you review this small follow-up? When a session is removed, its caches are now dropped, and a cache's files are only deleted once no running query is reading them. Would be nice to get it into the release so caches are freed, not just built and recovered. thanks |
andygrove
left a comment
There was a problem hiding this comment.
Thanks @akshaychitneni, this is a really useful change. Making the drop wait for running readers before deleting build data closes a real gap, and the cancellation loop and orphan bookkeeping read clearly. The new tests are nice too. I traced one path that I think can still leave a cache pinned for a removed session, inline below. The rest are small.
| /// data is cleaned up once no running query reads it. Its builds still in | ||
| /// flight are cancelled, and what waits on them fails. Using the session | ||
| /// again builds its caches anew. | ||
| pub(crate) async fn remove_session_caches( |
There was a problem hiding this comment.
One case I traced: if a session is removed while a query is still reading its cache, and that query then loses the cache data, rebuild_lost_inputs builds it again under the removed session. new_context succeeds for a removed session, and record_materialized_cache writes a fresh entry for it. Nothing removes that entry afterwards, so its stages stay pinned. Would it make sense to fail that reader with CACHE_DATA_LOST instead, so a retry on the session rebuilds as usual? Or drop the rebuilt entry once provide_input has handed it over? orphaned alone can't tell the two cases apart, since invalidate_build in the rebuild path orphans too. A test along the lines of removing_a_session_spares_running_reads_and_rebuilds_on_reuse plus a lose_executor would pin this down.
| // again each time rather than walk a list taken up front. | ||
| while let Some((build, key)) = self.cache_linker.inflight_of(session_id) { | ||
| match self.cache_linker.cancel(&build) { | ||
| Some(held) => self.fail_pending(&held, SESSION_CLOSED, sender).await, |
There was a problem hiding this comment.
The tests cover a running build nicely. Could we add one with a nested cache, like cache_over("c2", cache_over("c1", aggregation_logical_plan(2))), removed before c1 finishes? That exercises this Some(held) branch, where c2's build is still held. It also checks that the loop ends with both keys gone.
| match self.cache_linker.cancel(&build) { | ||
| Some(held) => self.fail_pending(&held, SESSION_CLOSED, sender).await, | ||
| None => { | ||
| self.fail_cache_job(&build, Some(&key), SESSION_CLOSED, sender) |
There was a problem hiding this comment.
Small question here. If another submission on the same session is between track and register when this loop runs, the build isn't held yet. So we abort its claim and post JobCancel for a job that hasn't been queued. The submitter then registers and submits the build anyway, and its read fails with a cache-miss error rather than "its session was closed". It's a narrow race, so I'm fine leaving it. Maybe worth a comment, or a check in register that the build is still in flight?
| let session_params = request.into_inner(); | ||
| let sender = self | ||
| .query_stage_event_loop | ||
| .get_sender() |
There was a problem hiding this comment.
Tiny thing. get_sender() errors out before the session is removed, which slightly undercuts the "removed even if dropping its caches fails" doc on SchedulerState::remove_session. Same in flight_sql.rs. It's unlikely in practice, so feel free to ignore.
| - Cached data is not evicted yet: it is kept until it is found lost, so cache | ||
| only what you reuse. | ||
| - Cached data is not evicted yet: it is kept until it is found lost, or its | ||
| session is removed with a `RemoveSession` call, so cache only what you reuse. |
There was a problem hiding this comment.
Could we say RemoveSession is a scheduler gRPC call that the client libraries don't expose yet? Otherwise users may look for a method on SessionContext that isn't there.
Part of #1829.
Rationale for this change
DataFrame::cache()data is never freed: removing a session leaves its cachespinned, and a dropped cache's files are deleted a fixed 300 s later even if a
query is still reading them.
What changes are included in this PR?
RemoveSessionand Flight SQLsession expiry now go through
SchedulerState::remove_session:"its session was closed";
path (session removal, validate-on-read, recovery, the cleanup after a
cache-lost failure) now orphans the build while queries read it, then deletes
it after the usual delay. A healthy cache dropped by another query's failure
is no longer deleted under a query still reading it.
Deferred:
RemoveSessiontoday(Rust, Python, CLI), and Flight SQL sessions hold no caches. Next: idle-session
expiry on the scheduler, and an explicit close API for clients.
job read is still dropped; this PR only stops their data being deleted under
other readers. Narrowing the drop needs structured lost-input reporting.
Are there any user-facing changes?
Yes: removing a session frees its cached data, and queries held on that
session's in-flight builds fail with "its session was closed". There's no
public API, proto or configuration change.