Repository navigation
Add optional queue-based non-blocking logging - #54
Merged
Merged
Conversation
## Summary File, syslog and network handlers can block the thread that logs. ## Changes - `setup_logging(use_queue=True)` puts records on a `queue.SimpleQueue`. A `QueueListener` thread owns the real handlers, and per-handler levels are still respected. - `shutdown_logging()` removes the queue handlers, writes whatever is still queued, stops the listener threads and closes their handlers. It is idempotent and registered with `atexit`. Calling `setup_logging()` again for a logger stops its previous listener, so threads do not leak. - The context filter runs on the queue handler, that is on the logging thread. The listener thread cannot see that thread's context variables. - `logging.handlers.QueueHandler` formats the record and drops `exc_info`, which would fold tracebacks into the message and out of the JSON `exc_info` field. A small subclass only resolves the message arguments, so tracebacks are unchanged. - Documented in the README and changelog, and `shutdown_logging` is exported. ## Trade-offs The queue is unbounded, so a producer that outruns the handlers grows memory. A bounded queue needs a drop or block policy, which is a design decision beyond this issue, so it is not included. ## Verification - Tests cover ordering and completeness after shutdown, captured context, handler levels, tracebacks in JSON, a handler that blocks while the caller does not, reconfiguration, switching the queue off, and idempotent shutdown. - `pytest`: 133 passed (latest dependencies) and 112 passed with 21 skipped (no extras). `ruff` and `mypy --strict` are clean. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U2nYHVGDdthQpcwNBPbp8r
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
File, syslog and network handlers can block the thread that logs.
Changes
setup_logging(use_queue=True)puts records on aqueue.SimpleQueue. AQueueListenerthread owns the real handlers, and per-handler levels are still respected.shutdown_logging()removes the queue handlers, writes whatever is still queued, stops the listener threads and closes their handlers. It is idempotent and registered withatexit. Callingsetup_logging()again for a logger stops its previous listener, so threads do not leak.logging.handlers.QueueHandlerformats the record and dropsexc_info, which would fold tracebacks into the message and out of the JSONexc_infofield. A small subclass only resolves the message arguments, so tracebacks are unchanged.shutdown_loggingis exported.Trade-offs
The queue is unbounded, so a producer that outruns the handlers grows memory. A bounded queue needs a drop or block policy, which is a design decision beyond this issue, so it is not included.
Verification
pytest: 133 passed (latest dependencies) and 112 passed with 21 skipped (no extras).ruffandmypy --strictare clean.Closes #23
🤖 Generated with Claude Code
https://claude.ai/code/session_01U2nYHVGDdthQpcwNBPbp8r
Generated by Claude Code