Skip to content

fix: allow standalone scheduler to use a session-configured partition slicing resolver - #2886

Open
vikrantmehta123 wants to merge 2 commits into
apache:mainfrom
vikrantmehta123:fix/2839-standalone-partition-resolver
Open

vikrantmehta123 wants to merge 2 commits into
apache:mainfrom
vikrantmehta123:fix/2839-standalone-partition-resolver

Conversation

@vikrantmehta123

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #2839 .

Rationale for this change

Standalone setup creates its own SchedulerConfig, leaving no way to supply a PartitionSliceableResolver. Custom plan nodes that require partition slicing therefore cannot run when their stages are split between tasks. The issue suggested two options. I have chosen the option of registering resolver next to the codec.

What changes are included in this PR?

  • Add resolver setter and getter methods to SessionConfigExt, following the existing codec extension pattern.
  • Forward the registered resolver to SchedulerConfig during standalone setup.
  • Add an integration test verifying that the standalone scheduler consults a resolver registered through session configuration.
  • Update the extending-components guide and upgrade notes.

Note: AI helped me identify that there are docs to be changed. AI also helped me come up with the integration test. I have read and understood both these changes, and they looked reasonable to me.

Are there any user-facing changes?

Yes, users can optionally register a resolver in standalone mode. The SessionConfigExt gets two more methods. This is documented in the upgrade guide.

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @vikrantmehta123, this looks good. Reading the resolver from the session config next to the codec is a clean fit, and the upgrade note is clear. I added the api-change label since the note calls out the two new SessionConfigExt methods.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api-change documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Let the standalone scheduler take a PartitionSliceableResolver

2 participants