Repository navigation
mpm_event, mpm_worker: fixes for child exits and ungraceful stops, listener wake-ups out of mutexes - #1045
Open
ylavic wants to merge 10 commits into
Open
mpm_event, mpm_worker: fixes for child exits and ungraceful stops, listener wake-ups out of mutexes#1045ylavic wants to merge 10 commits into
ylavic wants to merge 10 commits into
Conversation
…t it down rather than close it on an ungraceful stop. On an ungraceful stop or restart, the child's main thread closed the sockets the workers publish in worker_sockets[] (close_worker_sockets()) while the workers could still use them, close them or recycle their pools: a crash on freed memory, or a worker using a closed, maybe reused, descriptor. mpm_event's worker also cleared its slot only after the connection could have been handed over to the listener and closed. Let exactly one thread take a socket out of its slot, atomically. The main thread only shuts down the sockets it takes, never closes them nor touches their pools. The worker unpublishes its socket (takes it back) before closing it, clearing its pool or handing the connection over; if the main thread took it, the worker waits for the scan to end (the main thread waits for no worker), then closes the connection rather than hand it over. So shutdown() always hits the live descriptor, and the worker frees its connection itself while the child's pools are alive. workers_may_exit becomes atomic with three states (AP_WORKERS_*): the worker waits while STOPPING, i.e. during the main thread's scan. Publish the socket only once ap_run_create_connection() has succeeded (the core closes it on failure), then re-read the flag and shut the socket down if set: the main thread stores it before swapping the slot, the worker swaps the slot before its re-read, and their swaps on the same slot order them, so at least one sees the other. shutdown() also wakes up workers blocked on their socket (Linux, and per the FreeBSD, macOS and illumos kernel sources), so the child exits sooner. For that, mpm_worker runs its lingering close itself, the socket published until the close, so that an ungraceful stop ends the lingering reads at once. Both MPMs share this in mpm_fdqueue (ap_worker_sockets_*()). * server/mpm_fdqueue.h: (ap_worker_sockets_t, AP_WORKERS_RUNNING, AP_WORKERS_STOPPING, AP_WORKERS_STOPPED): New, the workers' sockets and the exit flag's values. (ap_worker_sockets_create, ap_worker_sockets_publish, ap_worker_sockets_unpublish, ap_worker_sockets_shutdown): Declare. * server/mpm_fdqueue.c: (struct ap_worker_sockets_t, ap_worker_sockets_create, ap_worker_sockets_publish, ap_worker_sockets_unpublish, ap_worker_sockets_shutdown): New. * server/mpm/event/event.c: (workers_may_exit): Make it atomic, with three states (AP_WORKERS_*). (worker_sockets): Make it an ap_worker_sockets_t. (struct event_conn_state_t): Add tslot, the slot publishing the socket. (close_socket_nonblocking_): Update the comment. (close_worker_sockets): Remove, ap_worker_sockets_shutdown() instead. (publish_worker_socket, unpublish_worker_socket): New, record tslot and call ap_worker_sockets_publish() or ap_worker_sockets_unpublish(). (signal_threads): Set AP_WORKERS_STOPPING, then shut down the sockets. (start_threads): Set AP_WORKERS_STOPPED. (setup_threads_runtime): Create worker_sockets. (close_connection): Unpublish the socket before closing it. (process_socket): Initialize tslot, publish the socket once the connection is created or resumed, unpublish it before every hand-off, close the connection instead if ap_worker_sockets_shutdown() took it. (process_lingering_close): Likewise before queuing. (worker_thread): Let process_socket() publish the socket. (setup_slave_conn): Initialize tslot. * server/mpm/worker/worker.c: (workers_may_exit): Make it atomic, with three states (AP_WORKERS_*). (worker_sockets): Make it an ap_worker_sockets_t. (close_worker_sockets): Remove, ap_worker_sockets_shutdown() instead. (signal_threads): Set AP_WORKERS_STOPPING, then shut down the sockets. (start_threads): Set AP_WORKERS_STOPPED. (setup_threads_runtime): Create worker_sockets. (worker_lingering_close): New, the core's ap_lingering_close() with the socket published during the lingering reads. (process_socket): Publish the socket once the connection is created, use worker_lingering_close(). (worker_thread): Let process_socket() publish and unpublish the socket. * server/connection.c: (ap_lingering_close): Note mpm_worker's copy.
…ls go in one-process mode. On an ungraceful stop the listener exits at once, closing none of the connections it holds (timeout queues and pollset, deferred lingering closes, the workers' queue): their pools, children of pconf, go only with pconf. In one-process mode (-X) that is at exit, after pchild was destroyed, so the modules' cleanups of these connections may use what pchild freed: mod_http2's session cleanup crashes in h2_workers_join() for an HTTP/2 connection waiting for I/O at SIGTERM. Once the workers may exit, make the workers close the connection rather than hand it over to the listener, and once the threads are joined make the main thread close what is left in clean_child_exit(), before pchild goes. Do it in one-process mode only, where the crash is (a forked child never destroys pconf), and only if the listener was joined, so that no MPM thread uses the queues or the pollset anymore. Nothing is left on a graceful stop. A suspended connection resumed after this (or whose resume timer is dropped here) still goes with pconf. The idle (recycled) connection pools are children of pconf, created after pruntime, so pconf destroys them before pruntime's cleanup of the queue info frees its idle list, which destroys them again: "double free or corruption" at exit with -X, the only mode where pconf is destroyed (by ap_terminate() from atexit()). The listener frees the idle list when it stops, but a worker can push its pool concurrently: an intermittent abort. Free the idle pools in clean_child_exit() in one-process mode too, in both MPMs. * server/mpm/event/event.c: (process_socket, process_lingering_close): Close the connection rather than hand it over to the listener once the workers may exit. (join_workers): Return whether the listener was joined. (just_close, close_held_connections): New. (listener_joined): New, set by child_main() from join_workers(). (clean_child_exit): In one-process mode, call close_held_connections() when the listener was joined, and free the idle pools. (child_main): Set listener_joined from join_workers(). * server/mpm/worker/worker.c: (clean_child_exit): Free the idle pools in one-process mode.
Call _exit() rather than exit() in forked children, so that the atexit() handlers inherited from the parent (e.g. OpenSSL's, registered by default) don't run there, possibly while threads the child could not join or modules' threads still use what they clean up. Nor is any atexit() handler a safe place for a child's own cleanup (after pchild, other threads maybe running): pchild's cleanups and the child_stopped hooks still run. This also skips gcov's and LeakSanitizer's handlers in the children (-X still calls exit()) and stdio flushing, which httpd's logs don't use. Keep exit() with -X, which runs ap_terminate() from atexit(). * server/mpm/event/event.c: (clean_child_exit): Call _exit() unless in ONE_PROCESS mode. * server/mpm/worker/worker.c: (clean_child_exit): Call _exit() unless in ONE_PROCESS mode.
…eues' mutexes. Waking up the listener is a syscall, which should not be made under a mutex taken for every timer registered (g_timer_skiplist_mtx) or every connection added to a timeout queue (timeout_mutex). Decide it under the mutex and wake up the listener after releasing it (once woken up, the listener re-reads the next expiry all the same). Rename the existing wakeup_listener(), which makes the listener exit, to shutdown_listener(), so that wakeup_listener() names the plain poll() wake-up of the listener. * server/mpm/event/event.c: (event_get_timer_event): Call apr_pollset_wakeup() after unlocking g_timer_skiplist_mtx. (TO_QUEUE_APPEND): Return whether the listener has to be woken up. (wakeup_listener): New, wake up the listener if it's poll()ing. (shutdown_listener): Renamed from wakeup_listener(), callers updated. (process_socket, event_resume_suspended, process_lingering_close): Wake up the listener after releasing timeout_mutex when TO_QUEUE_APPEND() says so.
…m_event. mpm_worker's wakeup_listener() makes the listener exit, like mpm_event's former one now named shutdown_listener(): use the same name in both MPMs. * server/mpm/worker/worker.c: (shutdown_listener): Renamed from wakeup_listener(), callers updated.
…g pushes to it. The listener terminates the workers' queue when it exits, which lets the idle workers leave ap_queue_pop_something() on a graceful stop. join_workers() gives up on a listener that has not stopped accepting after 5 seconds (AH00475), which now happens only when a bug blocks it (in a module or in its loop), then joins the workers anyway (shutdown_listener() terminates only the queue info): the idle workers wait forever and the child never exits, keeping its scoreboard slot (the parent waits for it on a graceful stop without GracefulShutdownTimeout). Terminate the queue before joining the workers, a noop if the listener did it already: the forever-hang becomes an exit. Queued connections are still handled before APR_EOF. The stuck listener could still push to the terminated queue, which ap_queue_push_socket() and ap_queue_push_timer() AP_DEBUG_ASSERT()ed against (the element queued anyway without maintainer mode). Return APR_EOF instead, checked under the queue's mutex, the element not queued. The callers drop the connection or the timer event quietly (a debug log), the child stopping already: mpm_event's push2worker() closes the connection at once (close_connection(), not shutdown_connection() whose deferred lingering close no worker would take), without signaling a graceful stop; push_timer2worker() puts the timer event back on timer_free_ring, its callback not called, on any push failure (its callers ignored the status, losing the event); mpm_worker's listener closes the socket, as on any push failure. * server/mpm_fdqueue.c: (ap_queue_push_socket, ap_queue_push_timer): Return APR_EOF if the queue is terminated rather than assert it is not. * server/mpm/event/event.c: (push2worker): On APR_EOF, log at debug level, close an existing connection at once and don't signal a graceful stop. (push_timer2worker): Move after the timers' declarations, add a locked flag (g_timer_skiplist_mtx held), drop the timer event if it can't be queued. (listener_thread): Pass the locked flag to push_timer2worker(). (join_workers): Call ap_queue_term() before joining the workers. * server/mpm/worker/worker.c: (listener_thread): Log at debug level on APR_EOF.
…om pruntime, not malloc(). Each thread's start info is malloc()ed by the start thread and free()d by the thread it starts (leaked if that can't be created), the threads array is freed only at child_main()'s end (not on the clean_child_exit() paths), and the thread starter, its attributes and the start thread come from pchild after the child_init() hooks, which may have started threads using pchild (why pruntime exists), and are destroyed in clean_child_exit() while a listener join_workers() gave up on may still run. Allocate them all from pruntime, the MPM's pool for its threads, so they live as long as the threads using them. child_main() allocates from it only until it creates the start thread, the only one allocating from it then. join_workers() takes the thread starter. * server/mpm/event/event.c: (listener_thread, worker_thread): Don't free() the start info. (create_listener_thread, start_threads): Allocate it from pruntime. (join_workers): Take the thread starter. (child_main): Allocate the thread starter, its threads array and the thread attributes from pruntime, create the start thread on pruntime.
…ges on pconf, not pchild. Nothing the child keeps until its exit should live in pchild, which clean_child_exit() destroys before the process ends: create the child's main thread, reopen the scoreboard and drop privileges on pconf, as the threads' runtime uses pruntime. No change of behavior is intended. * server/mpm/event/event.c: (child_main): Pass pconf rather than pchild to ap_thread_main_create(), ap_reopen_scoreboard() and ap_run_drop_privileges().
…ections, and the -X exit. Add tests stopping or restarting the server ungracefully while its workers are busy, checking that it does not wait for the connections (the time taken, and no "still did not exit" from the parent, which resends SIGTERM and SIGKILL) and serves again afterwards; and that with -X it exits with status 0 on SIGTERM, nothing crashing. * test/modules/core/test_008_restart.py: (_Sockets): New, client connections closed whatever happens. (TestUngracefulExit): New. (test_core_008_04_busy): New, partial requests and slow readers (skipped with mpm_motorz, which doesn't wake a child reading a request). (test_core_008_05_lingering): New, connections in their lingering close. (test_core_008_06_one_process): New, -X started and stopped by SIGTERM.
…fixes. One entry for the previous commits' fixes. * changes-entries/event-worker-child-exit.txt: New.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This fixes, in mpm_event and mpm_worker:
workers' sockets while they were using them;
connections open on an ungraceful stop;
On ungraceful stops, workers busy on a connection, lingering ones included, are now woken up at once where the system
allows it.
Changes entry (the last commit):
mpm_event, mpm_worker: Fix possible crashes of child processes on an ungraceful stop or restart, or at exit in
one-process mode (-X), and a child never exiting on a graceful stop when its listener thread is stuck (mpm_event). A
child now exits at once on an ungraceful stop even when its connections are busy, and no longer runs the parent
process' exit handlers (e.g. OpenSSL's).
Commits:
ungraceful stop. — Exactly one thread owns a worker's published socket: on an ungraceful stop the main thread takes
each one and shuts it down (never closing it or touching its pool), and a worker takes its socket back before
closing it or handing the connection over, waiting for the main thread's scan if it lost the race. shutdown() also
wakes up workers blocked on their socket (Linux, and per the FreeBSD, macOS and illumos kernel sources), so the
child exits at once instead of waiting for the parent's SIGTERM/SIGKILL resends. The protocol is shared by both
MPMs through a small helper in mpm_fdqueue (ap_worker_sockets_*()).
clean_child_exit() closes the connections the listener left (timeout queues, pollset, deferred lingering closes,
the workers' queue; mpm_event) and frees the idle pools before pchild is destroyed, instead of letting them go with
pconf after it (mod_http2's session cleanup crashed there; the idle pools were freed twice).
from the parent (e.g. OpenSSL's global cleanup) can crash a child whose other threads still use them, and none is a
safe place for a child's own cleanup. Complements "mpm_event: Fix just_die() calling non-async-signal-safe code."
(already in trunk).
g_timer_skiplist_mtx / timeout_mutex, one way the listener could be held up.
function making the listener exit.
stuck (now only through a bug) no longer waits forever for its workers on a graceful stop; a push to the terminated
queue fails instead of asserting: the connection is closed at once, or the timer event dropped.
malloc()/free() hand-off, no allocation from pchild after child_init, and the starter outlives pchild.
nothing the child keeps until its exit lives in pchild.
Testing:
stop, the -X exit). On trunk, the busy ones fail with mpm_event: the parent resends SIGTERM, then SIGKILLs the child
after about 9 s. With mpm_motorz, which doesn't wake a child reading a request, the busy test is skipped.
crashes at exit (h2_workers_join() after pchild's destruction), with this PR it exits cleanly.
graceful stop the child exits after AH00475 (about 5.5 s); on trunk it hangs until the thread is released.
Risk:
are then closed (the queue is terminated, its pushes fail and close them). With "wake up the listener out of the
timers' and the timeout queues' mutexes", the listener should no longer be held up that long; only a bug blocking
it would, and then exiting the child is better than never exiting, as on trunk.
would no longer get it, though an atexit() handler was never safe there (it runs after pchild is destroyed,
possibly while other threads still run); pchild's cleanups and the child_stopped hook are the places for a child's
cleanup, and still run. httpd's logs don't use stdio.