Fix post_send firing after task execution with await_inplace=True#639
Open
apoorvdarshan wants to merge 1 commit into
Open
Fix post_send firing after task execution with await_inplace=True#639apoorvdarshan wants to merge 1 commit into
apoorvdarshan wants to merge 1 commit into
Conversation
With InMemoryBroker(await_inplace=True) the middleware hooks fired in the order pre_send -> pre_execute -> task -> post_execute -> post_send, because kick() awaited the full receiver callback (pre_execute/task/post_execute) before control returned to the sender that invokes post_send. Now kick() schedules the in-place receiver as a task and returns immediately, and the task is awaited in a new _finish_kick() hook that runs right after post_send. This restores the expected ordering pre_send -> post_send -> pre_execute -> task -> post_execute while the task is still fully executed before kiq() returns. Fixes taskiq-python#586.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #586.
With
InMemoryBroker(await_inplace=True), the client-sidepost_sendmiddleware hook fired after the task finished executing, giving the ordering:instead of the expected:
This breaks the semantics of
post_send(e.g. a "task queued" log ends up with a later timestamp than "task started").Root cause
InMemoryBroker.kick()awaited the full receiver callback (runningpre_execute->task->post_execute) inline before control returned toAsyncKicker.kiq(), which only then invoked thepost_sendmiddleware hook.Fix
InMemoryBroker.kick()now schedules the in-place receiver as anasyncio.Taskand returns immediately (into a dedicated_inplace_tasksset) instead of awaiting it inline. This letspost_sendrun before the task starts executing.AsyncBroker._finish_kick()hook (a no-op by default) is awaited inAsyncKicker.kiq()right after thepost_sendloop.InMemoryBrokeroverrides it to await the scheduled in-place task.The task is still fully executed before
kiq()returns, so the existingawait_inplacecontract is preserved (result ready on return, no leftover running tasks). Other brokers are unaffected:_finish_kick()defaults to a no-op.Tests
tests/brokers/test_inmemory.py::test_inplace_middleware_order, which asserts the hook order ispre_send -> post_send -> pre_execute -> task -> post_executeand that the task is fully executed by the timekiq()returns. This test fails onmasterand passes with the fix.pytest -n auto, 293 passed), plusblack,ruff, andmypy(strict) all clean.Disclosure: prepared with AI assistance; reviewed and verified locally.