Skip to content

feat(scheduler): drop a session's DataFrame::cache() data when the session is removed - #2889

Open
akshaychitneni wants to merge 1 commit into
apache:mainfrom
akshaychitneni:feat/cache-remove-session
Open

akshaychitneni wants to merge 1 commit into
apache:mainfrom
akshaychitneni:feat/cache-remove-session

Conversation

@akshaychitneni

Copy link
Copy Markdown
Contributor

Part of #1829.

Rationale for this change

DataFrame::cache() data is never freed: removing a session leaves its caches
pinned, and a dropped cache's files are deleted a fixed 300 s later even if a
query is still reading them.

What changes are included in this PR?

  • Removing a session drops its caches. gRPC RemoveSession and Flight SQL
    session expiry now go through SchedulerState::remove_session:
    • built caches leave the registry at once;
    • builds still in flight are cancelled, and what waits on them fails with
      "its session was closed";
    • using the session again rebuilds its caches.
  • Dropped cache data is deleted once no running query reads it. Every drop
    path (session removal, validate-on-read, recovery, the cleanup after a
    cache-lost failure) now orphans the build while queries read it, then deletes
    it after the usual delay. A healthy cache dropped by another query's failure
    is no longer deleted under a query still reading it.

Deferred:

  • A trigger clients use. No client library sends RemoveSession today
    (Rust, Python, CLI), and Flight SQL sessions hold no caches. Next: idle-session
    expiry on the scheduler, and an explicit close API for clients.
  • Dropping only the lost caches. After a cache-lost failure, every cache the
    job read is still dropped; this PR only stops their data being deleted under
    other readers. Narrowing the drop needs structured lost-input reporting.

Are there any user-facing changes?

Yes: removing a session frees its cached data, and queries held on that
session's in-flight builds fail with "its session was closed". There's no
public API, proto or configuration change.

…ssion is removed

Removing a session (a gRPC RemoveSession call, or a Flight SQL session
expiring) now drops its caches: built ones leave the registry at once, builds
in flight are cancelled and what waits on them fails with "its session was
closed", and using the session again rebuilds them.

A dropped cache's data is now deleted once no running query reads it, after
the usual delay, rather than a fixed delay after the drop. This applies to
every way a cache is dropped, so a healthy cache dropped by another query's
failure is no longer deleted under a query still reading it.
@akshaychitneni

Copy link
Copy Markdown
Contributor Author

@andygrove @milenkovicm @comphead could you review this small follow-up? When a session is removed, its caches are now dropped, and a cache's files are only deleted once no running query is reading them. Would be nice to get it into the release so caches are freed, not just built and recovered. thanks

@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Nov 17, 2026

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @akshaychitneni, this is a really useful change. Making the drop wait for running readers before deleting build data closes a real gap, and the cancellation loop and orphan bookkeeping read clearly. The new tests are nice too. I traced one path that I think can still leave a cache pinned for a removed session, inline below. The rest are small.

/// data is cleaned up once no running query reads it. Its builds still in
/// flight are cancelled, and what waits on them fails. Using the session
/// again builds its caches anew.
pub(crate) async fn remove_session_caches(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One case I traced: if a session is removed while a query is still reading its cache, and that query then loses the cache data, rebuild_lost_inputs builds it again under the removed session. new_context succeeds for a removed session, and record_materialized_cache writes a fresh entry for it. Nothing removes that entry afterwards, so its stages stay pinned. Would it make sense to fail that reader with CACHE_DATA_LOST instead, so a retry on the session rebuilds as usual? Or drop the rebuilt entry once provide_input has handed it over? orphaned alone can't tell the two cases apart, since invalidate_build in the rebuild path orphans too. A test along the lines of removing_a_session_spares_running_reads_and_rebuilds_on_reuse plus a lose_executor would pin this down.

// again each time rather than walk a list taken up front.
while let Some((build, key)) = self.cache_linker.inflight_of(session_id) {
match self.cache_linker.cancel(&build) {
Some(held) => self.fail_pending(&held, SESSION_CLOSED, sender).await,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tests cover a running build nicely. Could we add one with a nested cache, like cache_over("c2", cache_over("c1", aggregation_logical_plan(2))), removed before c1 finishes? That exercises this Some(held) branch, where c2's build is still held. It also checks that the loop ends with both keys gone.

match self.cache_linker.cancel(&build) {
Some(held) => self.fail_pending(&held, SESSION_CLOSED, sender).await,
None => {
self.fail_cache_job(&build, Some(&key), SESSION_CLOSED, sender)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small question here. If another submission on the same session is between track and register when this loop runs, the build isn't held yet. So we abort its claim and post JobCancel for a job that hasn't been queued. The submitter then registers and submits the build anyway, and its read fails with a cache-miss error rather than "its session was closed". It's a narrow race, so I'm fine leaving it. Maybe worth a comment, or a check in register that the build is still in flight?

let session_params = request.into_inner();
let sender = self
.query_stage_event_loop
.get_sender()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tiny thing. get_sender() errors out before the session is removed, which slightly undercuts the "removed even if dropping its caches fails" doc on SchedulerState::remove_session. Same in flight_sql.rs. It's unlikely in practice, so feel free to ignore.

- Cached data is not evicted yet: it is kept until it is found lost, so cache
only what you reuse.
- Cached data is not evicted yet: it is kept until it is found lost, or its
session is removed with a `RemoveSession` call, so cache only what you reuse.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we say RemoveSession is a scheduler gRPC call that the client libraries don't expose yet? Otherwise users may look for a method on SessionContext that isn't there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants