Skip to content

fix(server): save and restore call_context in cancel() to prevent race with producer#1127

Open
isheng-eqi wants to merge 4 commits into
a2aproject:mainfrom
isheng-eqi:fix/cancel-call-context-race
Open

fix(server): save and restore call_context in cancel() to prevent race with producer#1127
isheng-eqi wants to merge 4 commits into
a2aproject:mainfrom
isheng-eqi:fix/cancel-call-context-race

Conversation

@isheng-eqi

Copy link
Copy Markdown

Summary

Fix the _call_context race condition in ActiveTask.cancel() — use a save/restore pattern so the cancel path doesn't overwrite the shared _call_context while the producer loop is using it.

Problem

cancel() directly set self._task_manager._call_context = call_context, which races with the producer loop at line ~512 that also sets _call_context for each incoming request:

# cancel() line 705:
self._task_manager._call_context = call_context  

# _run_producer loop line 512:
self._task_manager._call_context = request_context.call_context

_call_context is used by task_store.get() and task_store.save() for authorization checks. When cancel and a normal request happen concurrently, _call_context gets overwritten — the wrong caller identity could be used for task store operations. A TODO at the site documented this known issue.

Fix

Wrap cancel()'s body in a save/restore pattern:

saved_call_context = self._task_manager._call_context
try:
    self._task_manager._call_context = call_context
    # ... all cancel operations ...
finally:
    self._task_manager._call_context = saved_call_context

This ensures the cancel uses its own call_context for its operations, then restores whatever was there before — whether that's the producer's or the start() original.

Test Plan

  • test_cancel_restores_call_context: creates ActiveTask with original call_context, calls cancel() with different context, verifies original is restored
  • ruff check + ruff format + ty check pass

… GC warning (Fixes a2aproject#1121)

On the normal-completion teardown path, _run_consumer's finally block
was releasing the ActiveTask reference (via _maybe_cleanup) before the
producer task had finished, leaving it parked on _request_queue.get().
When GC reclaimed the ActiveTask, the still-pending producer was
destroyed, triggering asyncio 'Task was destroyed but it is pending!'
warnings.

Fix: cancel _producer_task and await it (suppressing CancelledError)
before calling _maybe_cleanup(), mirroring what cancel() already does
on the explicit cancel path. This ensures deterministic teardown and
prevents the GC warning.
…th is configured (Fixes a2aproject#585)

The A2A protocol spec requires servers to authenticate push notifications
when the client provides an authentication scheme in
PushNotificationConfig. Previously BasePushNotificationSender only sent
the X-A2A-Notification-Token header and completely ignored the
authentication field.

Now the _dispatch_notification method checks push_info.HasField('authentication')
and adds an Authorization header with '{scheme} {credentials}' when
both scheme and credentials are non-empty. This supports Bearer and
any other scheme the client provides.
…e with producer

The cancel() method was directly setting self._task_manager._call_context,
which races with the producer loop that also sets _call_context for
incoming requests. When both run concurrently, the shared _call_context
can be overwritten, causing task_store operations to use the wrong
authorization context.

Fix: save the original _call_context before cancel() operations and
restore it in a finally block. This resolves the TODO at line ~512
documenting the conflict.
@isheng-eqi isheng-eqi requested a review from a team as a code owner July 7, 2026 19:11

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request addresses asyncio warnings by ensuring the producer task is cancelled and awaited before cleanup, and introduces a save/restore pattern for the call context during task cancellation to prevent races. It also adds support for push notification authentication by appending an Authorization header when configured, along with corresponding unit tests. The review feedback recommends narrowing the window of the call context save/restore pattern to avoid concurrency issues during long-running awaits, and simplifying the header initialization logic by using an empty dictionary instead of checking for None.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment on lines +707 to +746
saved_call_context = self._task_manager._call_context
try:
self._task_manager._call_context = call_context

task = await self._task_manager.get_task()
request_context = RequestContext(
call_context=call_context,
task_id=self._task_id,
context_id=task.context_id if task else None,
task=task,
)

async with self._lock:
if not self._is_finished.is_set() and self._producer_task:
logger.debug(
'Cancel[%s]: Cancelling producer task', self._task_id
)
self._producer_task.cancel()
try:
await self._agent_executor.cancel(
request_context, self._event_queue_agent
async with self._lock:
if not self._is_finished.is_set() and self._producer_task:
logger.debug(
'Cancel[%s]: Cancelling producer task', self._task_id
)
except Exception as e:
logger.exception(
'Cancel[%s]: Agent cancel failed', self._task_id
self._producer_task.cancel()
try:
await self._agent_executor.cancel(
request_context, self._event_queue_agent
)
except Exception as e:
logger.exception(
'Cancel[%s]: Agent cancel failed', self._task_id
)
await self._mark_task_as_failed(e)
raise
else:
logger.debug(
'Cancel[%s]: Task already finished [%s] or producer not started [%s], not cancelling',
self._task_id,
self._is_finished.is_set(),
self._producer_task,
)
await self._mark_task_as_failed(e)
raise
else:
logger.debug(
'Cancel[%s]: Task already finished [%s] or producer not started [%s], not cancelling',
self._task_id,
self._is_finished.is_set(),
self._producer_task,
)

await self._is_finished.wait()
task = await self._task_manager.get_task()
await self._is_finished.wait()
task = await self._task_manager.get_task()
finally:
self._task_manager._call_context = saved_call_context

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Modifying the shared self._task_manager._call_context with a save/restore pattern across a long-running await self._is_finished.wait() is highly vulnerable to race conditions. If multiple cancel() calls or other concurrent operations occur, the shared context can be corrupted or restored to the wrong value.

To minimize the race window and ensure robustness, we should only set _call_context temporarily during the active cancellation phase, restore it before awaiting _is_finished.wait(), and then temporarily set it again to retrieve the final task state.

        saved_call_context = self._task_manager._call_context
        try:
            self._task_manager._call_context = call_context

            task = await self._task_manager.get_task()
            request_context = RequestContext(
                call_context=call_context,
                task_id=self._task_id,
                context_id=task.context_id if task else None,
                task=task,
            )

            async with self._lock:
                if not self._is_finished.is_set() and self._producer_task:
                    logger.debug(
                        'Cancel[%s]: Cancelling producer task', self._task_id
                    )
                    self._producer_task.cancel()
                    try:
                        await self._agent_executor.cancel(
                            request_context, self._event_queue_agent
                        )
                    except Exception as e:
                        logger.exception(
                            'Cancel[%s]: Agent cancel failed', self._task_id
                        )
                        await self._mark_task_as_failed(e)
                        raise
                else:
                    logger.debug(
                        'Cancel[%s]: Task already finished [%s] or producer not started [%s], not cancelling',
                        self._task_id,
                        self._is_finished.is_set(),
                        self._producer_task,
                    )
        finally:
            self._task_manager._call_context = saved_call_context

        await self._is_finished.wait()

        saved_call_context = self._task_manager._call_context
        try:
            self._task_manager._call_context = call_context
            task = await self._task_manager.get_task()
        finally:
            self._task_manager._call_context = saved_call_context

Comment on lines +85 to +100
headers: dict[str, str] | None = None
if push_info.token:
headers = {'X-A2A-Notification-Token': push_info.token}

# Add Authorization header when authentication is configured.
# The A2A spec requires servers to authenticate push notifications
# when the client provides an authentication scheme in
# PushNotificationConfig.
if push_info.HasField('authentication'):
auth = push_info.authentication
scheme = auth.scheme
credentials = auth.credentials
if scheme and credentials:
if headers is None:
headers = {}
headers['Authorization'] = f'{scheme} {credentials}'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

low

Instead of initializing headers as None and performing repetitive None checks, we can initialize headers as an empty dictionary and conditionally populate it. This simplifies the logic and improves readability.

            headers: dict[str, str] = {}
            if push_info.token:
                headers['X-A2A-Notification-Token'] = push_info.token

            # Add Authorization header when authentication is configured.
            # The A2A spec requires servers to authenticate push notifications
            # when the client provides an authentication scheme in
            # PushNotificationConfig.
            if push_info.HasField('authentication'):
                auth = push_info.authentication
                scheme = auth.scheme
                credentials = auth.credentials
                if scheme and credentials:
                    headers['Authorization'] = f'{scheme} {credentials}'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant