Skip to content

Add optional queue-based non-blocking logging - #54

Merged
markbac merged 1 commit into
mainfrom
feat/23-queue-logging
Oct 3, 2026
Merged

markbac merged 1 commit into
mainfrom
feat/23-queue-logging

Conversation

@markbac

@markbac markbac commented Oct 3, 2026

Copy link
Copy Markdown
Owner

Summary

File, syslog and network handlers can block the thread that logs.

Changes

  • setup_logging(use_queue=True) puts records on a queue.SimpleQueue. A QueueListener thread owns the real handlers, and per-handler levels are still respected.
  • shutdown_logging() removes the queue handlers, writes whatever is still queued, stops the listener threads and closes their handlers. It is idempotent and registered with atexit. Calling setup_logging() again for a logger stops its previous listener, so threads do not leak.
  • The context filter runs on the queue handler, that is on the logging thread. The listener thread cannot see that thread's context variables.
  • logging.handlers.QueueHandler formats the record and drops exc_info, which would fold tracebacks into the message and out of the JSON exc_info field. A small subclass only resolves the message arguments, so tracebacks are unchanged.
  • Documented in the README and changelog, and shutdown_logging is exported.

Trade-offs

The queue is unbounded, so a producer that outruns the handlers grows memory. A bounded queue needs a drop or block policy, which is a design decision beyond this issue, so it is not included.

Verification

  • Tests cover ordering and completeness after shutdown, captured context, handler levels, tracebacks in JSON, a handler that blocks while the caller does not, reconfiguration, switching the queue off, and idempotent shutdown.
  • pytest: 133 passed (latest dependencies) and 112 passed with 21 skipped (no extras). ruff and mypy --strict are clean.

Closes #23

🤖 Generated with Claude Code

https://claude.ai/code/session_01U2nYHVGDdthQpcwNBPbp8r


Generated by Claude Code

## Summary

File, syslog and network handlers can block the thread that logs.

## Changes

- `setup_logging(use_queue=True)` puts records on a `queue.SimpleQueue`. A `QueueListener` thread owns the real handlers, and per-handler levels are still respected.
- `shutdown_logging()` removes the queue handlers, writes whatever is still queued, stops the listener threads and closes their handlers. It is idempotent and registered with `atexit`. Calling `setup_logging()` again for a logger stops its previous listener, so threads do not leak.
- The context filter runs on the queue handler, that is on the logging thread. The listener thread cannot see that thread's context variables.
- `logging.handlers.QueueHandler` formats the record and drops `exc_info`, which would fold tracebacks into the message and out of the JSON `exc_info` field. A small subclass only resolves the message arguments, so tracebacks are unchanged.
- Documented in the README and changelog, and `shutdown_logging` is exported.

## Trade-offs

The queue is unbounded, so a producer that outruns the handlers grows memory. A bounded queue needs a drop or block policy, which is a design decision beyond this issue, so it is not included.

## Verification

- Tests cover ordering and completeness after shutdown, captured context, handler levels, tracebacks in JSON, a handler that blocks while the caller does not, reconfiguration, switching the queue off, and idempotent shutdown.
- `pytest`: 133 passed (latest dependencies) and 112 passed with 21 skipped (no extras). `ruff` and `mypy --strict` are clean.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U2nYHVGDdthQpcwNBPbp8r
@markbac
markbac merged commit 88db94e into main Oct 3, 2026
9 checks passed
@markbac
markbac deleted the feat/23-queue-logging branch October 19, 2026 08:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature: optional non-blocking logging via QueueHandler/QueueListener

2 participants