Skip to content

mpm_event, mpm_worker: fixes for child exits and ungraceful stops, listener wake-ups out of mutexes - #1045

Open
ylavic wants to merge 10 commits into
apache:trunkfrom
ylavic:trunk-event-fixes-aca57319
Open

ylavic wants to merge 10 commits into
apache:trunkfrom
ylavic:trunk-event-fixes-aca57319

Conversation

@ylavic

@ylavic ylavic commented Nov 16, 2026

Copy link
Copy Markdown
Member

This fixes, in mpm_event and mpm_worker:

  • a possible crash or wrong descriptor use by an exiting child on ungraceful stops/restarts: the main thread closed the
    workers' sockets while they were using them;
  • a double free at exit in one-process mode, and (mpm_event) a crash at exit in one-process mode with HTTP/2
    connections open on an ungraceful stop;
  • a child that could not exit when its listener was stuck.

On ungraceful stops, workers busy on a connection, lingering ones included, are now woken up at once where the system
allows it.

Changes entry (the last commit):
mpm_event, mpm_worker: Fix possible crashes of child processes on an ungraceful stop or restart, or at exit in
one-process mode (-X), and a child never exiting on a graceful stop when its listener thread is stuck (mpm_event). A
child now exits at once on an ungraceful stop even when its connections are busy, and no longer runs the parent
process' exit handlers (e.g. OpenSSL's).

Commits:

  • mpm_event, mpm_worker: hand a worker's socket over atomically and shut it down rather than close it on an
    ungraceful stop. — Exactly one thread owns a worker's published socket: on an ungraceful stop the main thread takes
    each one and shuts it down (never closing it or touching its pool), and a worker takes its socket back before
    closing it or handing the connection over, waiting for the main thread's scan if it lost the race. shutdown() also
    wakes up workers blocked on their socket (Linux, and per the FreeBSD, macOS and illumos kernel sources), so the
    child exits at once instead of waiting for the parent's SIGTERM/SIGKILL resends. The protocol is shared by both
    MPMs through a small helper in mpm_fdqueue (ap_worker_sockets_*()).
  • mpm_event, mpm_worker: free what the child still holds before its pools go in one-process mode. — In -X,
    clean_child_exit() closes the connections the listener left (timeout queues, pollset, deferred lingering closes,
    the workers' queue; mpm_event) and frees the idle pools before pchild is destroyed, instead of letting them go with
    pconf after it (mod_http2's session cleanup crashed there; the idle pools were freed twice).
  • mpm_event, mpm_worker: _exit() forked children. — Forked children no longer run atexit() handlers: those inherited
    from the parent (e.g. OpenSSL's global cleanup) can crash a child whose other threads still use them, and none is a
    safe place for a child's own cleanup. Complements "mpm_event: Fix just_die() calling non-async-signal-safe code."
    (already in trunk).
  • mpm_event: wake up the listener out of the timers' and the timeout queues' mutexes. — No syscall under
    g_timer_skiplist_mtx / timeout_mutex, one way the listener could be held up.
  • mpm_worker: rename wakeup_listener() to shutdown_listener(), as in mpm_event. — The same name in both MPMs for the
    function making the listener exit.
  • mpm_event: terminate the workers' queue in join_workers() too, failing pushes to it. — A child whose listener is
    stuck (now only through a bug) no longer waits forever for its workers on a graceful stop; a push to the terminated
    queue fails instead of asserting: the connection is closed at once, or the timer event dropped.
  • mpm_event: allocate the threads' start info and the thread starter from pruntime, not malloc(). — No
    malloc()/free() hand-off, no allocation from pchild after child_init, and the starter outlives pchild.
  • mpm_event: initialize the child's main thread, scoreboard and privileges on pconf, not pchild. — Consistency:
    nothing the child keeps until its exit lives in pchild.
  • test: check ungraceful stops and restarts with busy or lingering connections, and the -X exit.
  • changes-entries: add the entry for mpm_event's and mpm_worker's exit fixes.

Testing:

  • New tests in test/modules/core/test_008_restart.py (busy and lingering connections at an ungraceful restart or
    stop, the -X exit). On trunk, the busy ones fail with mpm_event: the parent resends SIGTERM, then SIGKILLs the child
    after about 9 s. With mpm_motorz, which doesn't wake a child reading a request, the busy test is skipped.
  • Not in CI (tested locally): -X with mod_http2 (h2c) and HTTP/2 connections busy at SIGTERM: on trunk mpm_event
    crashes at exit (h2_workers_join() after pchild's destruction), with this PR it exits cleanly.
  • Not in CI (tested locally): a stuck listener, simulated by freezing its thread with ptrace (that thread only): on a
    graceful stop the child exits after AH00475 (about 5.5 s); on trunk it hangs until the thread is released.

Risk:

  • join_workers() gives up on the listener if it hasn't exited 5 s after a graceful stop, and its pending connections
    are then closed (the queue is terminated, its pushes fail and close them). With "wake up the listener out of the
    timers' and the timeout queues' mutexes", the listener should no longer be held up that long; only a bug blocking
    it would, and then exiting the child is better than never exiting, as on trunk.
  • Forked children no longer run atexit() handlers or flush stdio at exit: a module relying on either in children
    would no longer get it, though an atexit() handler was never safe there (it runs after pchild is destroyed,
    possibly while other threads still run); pchild's cleanups and the child_stopped hook are the places for a child's
    cleanup, and still run. httpd's logs don't use stdio.

…t it down rather than close it on an ungraceful stop.

On an ungraceful stop or restart, the child's main thread closed the
sockets the workers publish in worker_sockets[] (close_worker_sockets())
while the workers could still use them, close them or recycle their pools:
a crash on freed memory, or a worker using a closed, maybe reused,
descriptor. mpm_event's worker also cleared its slot only after the
connection could have been handed over to the listener and closed.

Let exactly one thread take a socket out of its slot, atomically. The main
thread only shuts down the sockets it takes, never closes them nor touches
their pools. The worker unpublishes its socket (takes it back) before
closing it, clearing its pool or handing the connection over; if the main
thread took it, the worker waits for the scan to end (the main thread waits
for no worker), then closes the connection rather than hand it over. So
shutdown() always hits the live descriptor, and the worker frees its
connection itself while the child's pools are alive. workers_may_exit
becomes atomic with three states (AP_WORKERS_*): the worker waits while
STOPPING, i.e. during the main thread's scan.

Publish the socket only once ap_run_create_connection() has succeeded (the
core closes it on failure), then re-read the flag and shut the socket down
if set: the main thread stores it before swapping the slot, the worker swaps
the slot before its re-read, and their swaps on the same slot order them, so
at least one sees the other.

shutdown() also wakes up workers blocked on their socket (Linux, and per the
FreeBSD, macOS and illumos kernel sources), so the child exits sooner. For
that, mpm_worker runs its lingering close itself, the socket published until
the close, so that an ungraceful stop ends the lingering reads at once.

Both MPMs share this in mpm_fdqueue (ap_worker_sockets_*()).

* server/mpm_fdqueue.h:
  (ap_worker_sockets_t, AP_WORKERS_RUNNING, AP_WORKERS_STOPPING,
  AP_WORKERS_STOPPED): New, the workers' sockets and the exit flag's values.
  (ap_worker_sockets_create, ap_worker_sockets_publish,
  ap_worker_sockets_unpublish, ap_worker_sockets_shutdown): Declare.
* server/mpm_fdqueue.c:
  (struct ap_worker_sockets_t, ap_worker_sockets_create,
  ap_worker_sockets_publish, ap_worker_sockets_unpublish,
  ap_worker_sockets_shutdown): New.
* server/mpm/event/event.c:
  (workers_may_exit): Make it atomic, with three states (AP_WORKERS_*).
  (worker_sockets): Make it an ap_worker_sockets_t.
  (struct event_conn_state_t): Add tslot, the slot publishing the socket.
  (close_socket_nonblocking_): Update the comment.
  (close_worker_sockets): Remove, ap_worker_sockets_shutdown() instead.
  (publish_worker_socket, unpublish_worker_socket): New, record tslot and
  call ap_worker_sockets_publish() or ap_worker_sockets_unpublish().
  (signal_threads): Set AP_WORKERS_STOPPING, then shut down the sockets.
  (start_threads): Set AP_WORKERS_STOPPED.
  (setup_threads_runtime): Create worker_sockets.
  (close_connection): Unpublish the socket before closing it.
  (process_socket): Initialize tslot, publish the socket once the
  connection is created or resumed, unpublish it before every hand-off,
  close the connection instead if ap_worker_sockets_shutdown() took it.
  (process_lingering_close): Likewise before queuing.
  (worker_thread): Let process_socket() publish the socket.
  (setup_slave_conn): Initialize tslot.
* server/mpm/worker/worker.c:
  (workers_may_exit): Make it atomic, with three states (AP_WORKERS_*).
  (worker_sockets): Make it an ap_worker_sockets_t.
  (close_worker_sockets): Remove, ap_worker_sockets_shutdown() instead.
  (signal_threads): Set AP_WORKERS_STOPPING, then shut down the sockets.
  (start_threads): Set AP_WORKERS_STOPPED.
  (setup_threads_runtime): Create worker_sockets.
  (worker_lingering_close): New, the core's ap_lingering_close() with the
  socket published during the lingering reads.
  (process_socket): Publish the socket once the connection is created, use
  worker_lingering_close().
  (worker_thread): Let process_socket() publish and unpublish the socket.
* server/connection.c:
  (ap_lingering_close): Note mpm_worker's copy.
…ls go in one-process mode.

On an ungraceful stop the listener exits at once, closing none of the
connections it holds (timeout queues and pollset, deferred lingering
closes, the workers' queue): their pools, children of pconf, go only with
pconf. In one-process mode (-X) that is at exit, after pchild was
destroyed, so the modules' cleanups of these connections may use what
pchild freed: mod_http2's session cleanup crashes in h2_workers_join() for
an HTTP/2 connection waiting for I/O at SIGTERM.

Once the workers may exit, make the workers close the connection rather
than hand it over to the listener, and once the threads are joined make the
main thread close what is left in clean_child_exit(), before pchild goes. Do
it in one-process mode only, where the crash is (a forked child never
destroys pconf), and only if the listener was joined, so that no MPM thread
uses the queues or the pollset anymore. Nothing is left on a graceful stop.
A suspended connection resumed after this (or whose resume timer is dropped
here) still goes with pconf.

The idle (recycled) connection pools are children of pconf, created after
pruntime, so pconf destroys them before pruntime's cleanup of the queue info
frees its idle list, which destroys them again: "double free or corruption"
at exit with -X, the only mode where pconf is destroyed (by ap_terminate()
from atexit()). The listener frees the idle list when it stops, but a worker
can push its pool concurrently: an intermittent abort. Free the idle pools
in clean_child_exit() in one-process mode too, in both MPMs.

* server/mpm/event/event.c:
  (process_socket, process_lingering_close): Close the connection rather
  than hand it over to the listener once the workers may exit.
  (join_workers): Return whether the listener was joined.
  (just_close, close_held_connections): New.
  (listener_joined): New, set by child_main() from join_workers().
  (clean_child_exit): In one-process mode, call close_held_connections()
  when the listener was joined, and free the idle pools.
  (child_main): Set listener_joined from join_workers().
* server/mpm/worker/worker.c:
  (clean_child_exit): Free the idle pools in one-process mode.
Call _exit() rather than exit() in forked children, so that the atexit()
handlers inherited from the parent (e.g. OpenSSL's, registered by default)
don't run there, possibly while threads the child could not join or
modules' threads still use what they clean up. Nor is any atexit() handler
a safe place for a child's own cleanup (after pchild, other threads maybe
running): pchild's cleanups and the child_stopped hooks still run. This
also skips gcov's and LeakSanitizer's handlers in the children (-X still
calls exit()) and stdio flushing, which httpd's logs don't use. Keep
exit() with -X, which runs ap_terminate() from atexit().

* server/mpm/event/event.c:
  (clean_child_exit): Call _exit() unless in ONE_PROCESS mode.
* server/mpm/worker/worker.c:
  (clean_child_exit): Call _exit() unless in ONE_PROCESS mode.
…eues' mutexes.

Waking up the listener is a syscall, which should not be made under a
mutex taken for every timer registered (g_timer_skiplist_mtx) or every
connection added to a timeout queue (timeout_mutex). Decide it under the
mutex and wake up the listener after releasing it (once woken up, the
listener re-reads the next expiry all the same).

Rename the existing wakeup_listener(), which makes the listener exit, to
shutdown_listener(), so that wakeup_listener() names the plain poll()
wake-up of the listener.

* server/mpm/event/event.c:
  (event_get_timer_event): Call apr_pollset_wakeup() after unlocking
  g_timer_skiplist_mtx.
  (TO_QUEUE_APPEND): Return whether the listener has to be woken up.
  (wakeup_listener): New, wake up the listener if it's poll()ing.
  (shutdown_listener): Renamed from wakeup_listener(), callers updated.
  (process_socket, event_resume_suspended, process_lingering_close): Wake
  up the listener after releasing timeout_mutex when TO_QUEUE_APPEND()
  says so.
…m_event.

mpm_worker's wakeup_listener() makes the listener exit, like mpm_event's
former one now named shutdown_listener(): use the same name in both MPMs.

* server/mpm/worker/worker.c:
  (shutdown_listener): Renamed from wakeup_listener(), callers updated.
…g pushes to it.

The listener terminates the workers' queue when it exits, which lets the
idle workers leave ap_queue_pop_something() on a graceful stop.
join_workers() gives up on a listener that has not stopped accepting
after 5 seconds (AH00475), which now happens only when a bug blocks it (in
a module or in its loop), then joins the workers anyway
(shutdown_listener() terminates only the queue info): the idle workers
wait forever and the child never exits, keeping its scoreboard slot (the
parent waits for it on a graceful stop without GracefulShutdownTimeout).

Terminate the queue before joining the workers, a noop if the listener did
it already: the forever-hang becomes an exit. Queued connections are still
handled before APR_EOF.

The stuck listener could still push to the terminated queue, which
ap_queue_push_socket() and ap_queue_push_timer() AP_DEBUG_ASSERT()ed
against (the element queued anyway without maintainer mode). Return
APR_EOF instead, checked under the queue's mutex, the element not queued.
The callers drop the connection or the timer event quietly (a debug log),
the child stopping already: mpm_event's push2worker() closes the
connection at once (close_connection(), not shutdown_connection() whose
deferred lingering close no worker would take), without signaling a
graceful stop; push_timer2worker() puts the timer event back on
timer_free_ring, its callback not called, on any push failure (its callers
ignored the status, losing the event); mpm_worker's listener closes the
socket, as on any push failure.

* server/mpm_fdqueue.c:
  (ap_queue_push_socket, ap_queue_push_timer): Return APR_EOF if the queue
  is terminated rather than assert it is not.
* server/mpm/event/event.c:
  (push2worker): On APR_EOF, log at debug level, close an existing
  connection at once and don't signal a graceful stop.
  (push_timer2worker): Move after the timers' declarations, add a locked
  flag (g_timer_skiplist_mtx held), drop the timer event if it can't be
  queued.
  (listener_thread): Pass the locked flag to push_timer2worker().
  (join_workers): Call ap_queue_term() before joining the workers.
* server/mpm/worker/worker.c:
  (listener_thread): Log at debug level on APR_EOF.
…om pruntime, not malloc().

Each thread's start info is malloc()ed by the start thread and free()d by
the thread it starts (leaked if that can't be created), the threads array
is freed only at child_main()'s end (not on the clean_child_exit() paths),
and the thread starter, its attributes and the start thread come from
pchild after the child_init() hooks, which may have started threads using
pchild (why pruntime exists), and are destroyed in clean_child_exit()
while a listener join_workers() gave up on may still run.

Allocate them all from pruntime, the MPM's pool for its threads, so they
live as long as the threads using them. child_main() allocates from it
only until it creates the start thread, the only one allocating from it
then. join_workers() takes the thread starter.

* server/mpm/event/event.c:
  (listener_thread, worker_thread): Don't free() the start info.
  (create_listener_thread, start_threads): Allocate it from pruntime.
  (join_workers): Take the thread starter.
  (child_main): Allocate the thread starter, its threads array and the
  thread attributes from pruntime, create the start thread on pruntime.
…ges on pconf, not pchild.

Nothing the child keeps until its exit should live in pchild, which
clean_child_exit() destroys before the process ends: create the child's
main thread, reopen the scoreboard and drop privileges on pconf, as the
threads' runtime uses pruntime. No change of behavior is intended.

* server/mpm/event/event.c:
  (child_main): Pass pconf rather than pchild to ap_thread_main_create(),
  ap_reopen_scoreboard() and ap_run_drop_privileges().
…ections, and the -X exit.

Add tests stopping or restarting the server ungracefully while its
workers are busy, checking that it does not wait for the connections (the
time taken, and no "still did not exit" from the parent, which resends
SIGTERM and SIGKILL) and serves again afterwards; and that with -X it
exits with status 0 on SIGTERM, nothing crashing.

* test/modules/core/test_008_restart.py:
  (_Sockets): New, client connections closed whatever happens.
  (TestUngracefulExit): New.
  (test_core_008_04_busy): New, partial requests and slow readers
  (skipped with mpm_motorz, which doesn't wake a child reading a request).
  (test_core_008_05_lingering): New, connections in their lingering close.
  (test_core_008_06_one_process): New, -X started and stopped by SIGTERM.
…fixes.

One entry for the previous commits' fixes.

* changes-entries/event-worker-child-exit.txt: New.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant