From 9c02b845fa31fe2c7b0f48d910f0699799afbe07 Mon Sep 17 00:00:00 2001 From: ylavic Date: Sun, 15 Nov 2026 18:31:24 +0100 Subject: [PATCH 01/10] mpm_event, mpm_worker: hand a worker's socket over atomically and shut it down rather than close it on an ungraceful stop. On an ungraceful stop or restart, the child's main thread closed the sockets the workers publish in worker_sockets[] (close_worker_sockets()) while the workers could still use them, close them or recycle their pools: a crash on freed memory, or a worker using a closed, maybe reused, descriptor. mpm_event's worker also cleared its slot only after the connection could have been handed over to the listener and closed. Let exactly one thread take a socket out of its slot, atomically. The main thread only shuts down the sockets it takes, never closes them nor touches their pools. The worker unpublishes its socket (takes it back) before closing it, clearing its pool or handing the connection over; if the main thread took it, the worker waits for the scan to end (the main thread waits for no worker), then closes the connection rather than hand it over. So shutdown() always hits the live descriptor, and the worker frees its connection itself while the child's pools are alive. workers_may_exit becomes atomic with three states (AP_WORKERS_*): the worker waits while STOPPING, i.e. during the main thread's scan. Publish the socket only once ap_run_create_connection() has succeeded (the core closes it on failure), then re-read the flag and shut the socket down if set: the main thread stores it before swapping the slot, the worker swaps the slot before its re-read, and their swaps on the same slot order them, so at least one sees the other. shutdown() also wakes up workers blocked on their socket (Linux, and per the FreeBSD, macOS and illumos kernel sources), so the child exits sooner. For that, mpm_worker runs its lingering close itself, the socket published until the close, so that an ungraceful stop ends the lingering reads at once. Both MPMs share this in mpm_fdqueue (ap_worker_sockets_*()). * server/mpm_fdqueue.h: (ap_worker_sockets_t, AP_WORKERS_RUNNING, AP_WORKERS_STOPPING, AP_WORKERS_STOPPED): New, the workers' sockets and the exit flag's values. (ap_worker_sockets_create, ap_worker_sockets_publish, ap_worker_sockets_unpublish, ap_worker_sockets_shutdown): Declare. * server/mpm_fdqueue.c: (struct ap_worker_sockets_t, ap_worker_sockets_create, ap_worker_sockets_publish, ap_worker_sockets_unpublish, ap_worker_sockets_shutdown): New. * server/mpm/event/event.c: (workers_may_exit): Make it atomic, with three states (AP_WORKERS_*). (worker_sockets): Make it an ap_worker_sockets_t. (struct event_conn_state_t): Add tslot, the slot publishing the socket. (close_socket_nonblocking_): Update the comment. (close_worker_sockets): Remove, ap_worker_sockets_shutdown() instead. (publish_worker_socket, unpublish_worker_socket): New, record tslot and call ap_worker_sockets_publish() or ap_worker_sockets_unpublish(). (signal_threads): Set AP_WORKERS_STOPPING, then shut down the sockets. (start_threads): Set AP_WORKERS_STOPPED. (setup_threads_runtime): Create worker_sockets. (close_connection): Unpublish the socket before closing it. (process_socket): Initialize tslot, publish the socket once the connection is created or resumed, unpublish it before every hand-off, close the connection instead if ap_worker_sockets_shutdown() took it. (process_lingering_close): Likewise before queuing. (worker_thread): Let process_socket() publish the socket. (setup_slave_conn): Initialize tslot. * server/mpm/worker/worker.c: (workers_may_exit): Make it atomic, with three states (AP_WORKERS_*). (worker_sockets): Make it an ap_worker_sockets_t. (close_worker_sockets): Remove, ap_worker_sockets_shutdown() instead. (signal_threads): Set AP_WORKERS_STOPPING, then shut down the sockets. (start_threads): Set AP_WORKERS_STOPPED. (setup_threads_runtime): Create worker_sockets. (worker_lingering_close): New, the core's ap_lingering_close() with the socket published during the lingering reads. (process_socket): Publish the socket once the connection is created, use worker_lingering_close(). (worker_thread): Let process_socket() publish and unpublish the socket. * server/connection.c: (ap_lingering_close): Note mpm_worker's copy. --- server/connection.c | 3 + server/mpm/event/event.c | 109 +++++++++++++++++++++++++++-------- server/mpm/worker/worker.c | 115 +++++++++++++++++++++++++++++-------- server/mpm_fdqueue.c | 66 +++++++++++++++++++++ server/mpm_fdqueue.h | 47 +++++++++++++++ 5 files changed, 292 insertions(+), 48 deletions(-) diff --git a/server/connection.c b/server/connection.c index f32a1f3712c..30e543f9db3 100644 --- a/server/connection.c +++ b/server/connection.c @@ -180,6 +180,9 @@ AP_DECLARE(int) ap_start_lingering_close(conn_rec *c) #endif } +/* mpm_worker's worker_lingering_close() repeats this function (with the + * socket published until the close): keep them in sync. + */ AP_DECLARE(void) ap_lingering_close(conn_rec *c) { char dummybuf[512]; diff --git a/server/mpm/event/event.c b/server/mpm/event/event.c index 53bafb41232..07d05755d67 100644 --- a/server/mpm/event/event.c +++ b/server/mpm/event/event.c @@ -180,9 +180,9 @@ static int server_limit = 0; /* ServerLimit */ static int thread_limit = 0; /* ThreadLimit */ static int had_healthy_child = 0; static volatile int dying = 0; -static volatile int workers_may_exit = 0; static volatile int start_thread_may_exit = 0; static volatile int listener_may_exit = 0; +static /*atomic*/ apr_uint32_t workers_may_exit = AP_WORKERS_RUNNING; static int listener_is_wakeable = 0; /* Pollset supports APR_POLLSET_WAKEABLE */ static int num_listensocks = 0; static apr_int32_t conns_this_child; /* MaxConnectionsPerChild, only access @@ -254,6 +254,10 @@ struct event_conn_state_t { conn_state_t pub; /** chaining in defer_linger_chain */ struct event_conn_state_t *chain; + /** the worker thread whose slot in worker_sockets publishes the + * connection's socket while it processes it, -1 once unpublished (see + * unpublish_worker_socket()) */ + int tslot; unsigned int /** Is lingering close from defer_lingering_close()? */ deferred_linger :1, @@ -498,10 +502,10 @@ static int ap_child_slot; /* Current child process slot in scoreboard */ */ #define LISTENER_SIGNAL SIGHUP -/* An array of socket descriptors in use by each thread used to - * perform a non-graceful (forced) shutdown of the server. +/* The socket descriptors in use by each thread used to perform a + * non-graceful (forced) shutdown of the server (see mpm_fdqueue.h). */ -static apr_socket_t **worker_sockets; +static ap_worker_sockets_t *worker_sockets; static volatile apr_uint32_t listensocks_disabled; @@ -580,7 +584,7 @@ static void close_socket_nonblocking_(apr_socket_t *csd, apr_status_t rv; apr_os_sock_t fd = -1; - /* close_worker_sockets() may have closed it already */ + /* Closed already? */ rv = apr_os_sock_get(&fd, csd); ap_log_error(APLOG_MARK, APLOG_TRACE8, 0, ap_server_conf, "closing socket %i/%pp from %s:%i", (int)fd, csd, from, line); @@ -599,16 +603,35 @@ static void close_socket_nonblocking_(apr_socket_t *csd, #define close_socket_nonblocking(csd) \ close_socket_nonblocking_(csd, __FUNCTION__, __LINE__) -static void close_worker_sockets(void) +/* A worker publishes its connection's socket in worker_sockets while it + * processes the connection (see mpm_fdqueue.h for the protocol), cs->tslot + * recording the slot, and unpublishes it before closing the connection + * (close_connection()) or handing it over (to the listener's queues and + * pollset, or to the module suspending it); if ap_worker_sockets_shutdown() + * took it, the worker closes the connection rather than hand it over to the + * listener. + */ +static void publish_worker_socket(event_conn_state_t *cs, int tslot) { - int i; - for (i = 0; i < threads_per_child; i++) { - apr_socket_t *csd = worker_sockets[i]; - if (csd) { - worker_sockets[i] = NULL; - close_socket_nonblocking(csd); - } + cs->tslot = tslot; + ap_worker_sockets_publish(worker_sockets, tslot, cs->pfd.desc.s); +} + +/* Unpublish the connection's socket before closing the connection or handing + * it over (a noop if this thread didn't publish it). Returns zero if + * ap_worker_sockets_shutdown() took it: the child is exiting, the caller + * closes the connection rather than hand it over. + */ +static int unpublish_worker_socket(event_conn_state_t *cs) +{ + int tslot = cs->tslot; + + if (tslot < 0) { + return 1; } + cs->tslot = -1; + return ap_worker_sockets_unpublish(worker_sockets, tslot, + cs->pfd.desc.s); } static void wakeup_listener(void) @@ -673,9 +696,11 @@ static void signal_threads(int mode) * workers to exit once it has stopped accepting new connections */ if (mode == ST_UNGRACEFUL) { - workers_may_exit = 1; + /* ap_worker_sockets_shutdown() sets AP_WORKERS_STOPPED once done */ + apr_atomic_set32(&workers_may_exit, AP_WORKERS_STOPPING); ap_queue_interrupt_all(worker_queue); - close_worker_sockets(); /* forcefully kill all current connections */ + /* forcefully kill all current connections */ + ap_worker_sockets_shutdown(worker_sockets); } ap_run_child_stopping(pchild, mode == ST_GRACEFUL); @@ -942,6 +967,8 @@ static void close_connection(event_conn_state_t *cs) ap_log_cerror(APLOG_MARK, APLOG_TRACE6, 0, cs->c, "closing connection from state %i", (int)cs->pub.state); + /* Ours again either way, see unpublish_worker_socket() */ + (void)unpublish_worker_socket(cs); close_socket_nonblocking(cs->pfd.desc.s); ap_queue_info_push_pool(worker_queue_info, cs->p); } @@ -1086,6 +1113,7 @@ static void process_socket(apr_thread_t *thd, apr_pool_t * p, apr_socket_t * soc if (cs == NULL) { /* This is a new connection */ listener_poll_type *pt = apr_pcalloc(p, sizeof(*pt)); cs = apr_pcalloc(p, sizeof(event_conn_state_t)); + cs->tslot = -1; cs->bucket_alloc = apr_bucket_alloc_create(p); ap_create_sb_handle(&cs->sbh, p, my_child_num, my_thread_num); c = ap_run_create_connection(p, ap_server_conf, sock, @@ -1111,6 +1139,7 @@ static void process_socket(apr_thread_t *thd, apr_pool_t * p, apr_socket_t * soc cs->pfd.client_data = pt; apr_pool_pre_cleanup_register(p, cs, ptrans_pre_cleanup); TO_QUEUE_ELEM_INIT(cs); + publish_worker_socket(cs, my_thread_num); ap_update_vhost_given_ip(c); @@ -1140,6 +1169,7 @@ static void process_socket(apr_thread_t *thd, apr_pool_t * p, apr_socket_t * soc } else { c = cs->c; + publish_worker_socket(cs, my_thread_num); ap_update_sb_handle(cs->sbh, my_child_num, my_thread_num); notify_resume(cs, 0); c->current_thread = thd; @@ -1241,6 +1271,11 @@ static void process_socket(apr_thread_t *thd, apr_pool_t * p, apr_socket_t * soc * the default is CONN_SENSE_WANT_READ still. */ update_reqevents_from_sense(cs, CONN_SENSE_WANT_READ); + if (!unpublish_worker_socket(cs)) { + /* The child is exiting, close rather than hand it over */ + close_connection(cs); + return; + } apr_thread_mutex_lock(timeout_mutex); TO_QUEUE_APPEND(cs->sc->io_q, cs); rv = apr_pollset_add(event_pollset, &cs->pfd); @@ -1284,6 +1319,11 @@ static void process_socket(apr_thread_t *thd, apr_pool_t * p, apr_socket_t * soc /* Add work to pollset. */ cs->pub.sense = CONN_SENSE_DEFAULT; update_reqevents_from_sense(cs, CONN_SENSE_WANT_WRITE); + if (!unpublish_worker_socket(cs)) { + /* The child is exiting, close rather than hand it over */ + close_connection(cs); + return; + } apr_thread_mutex_lock(timeout_mutex); TO_QUEUE_APPEND(cs->sc->wc_q, cs); rv = apr_pollset_add(event_pollset, &cs->pfd); @@ -1335,6 +1375,11 @@ static void process_socket(apr_thread_t *thd, apr_pool_t * p, apr_socket_t * soc /* Add work to pollset. */ cs->pub.sense = CONN_SENSE_DEFAULT; update_reqevents_from_sense(cs, CONN_SENSE_WANT_READ); + if (!unpublish_worker_socket(cs)) { + /* The child is exiting, close rather than hand it over */ + close_connection(cs); + return; + } apr_thread_mutex_lock(timeout_mutex); TO_QUEUE_APPEND(cs->sc->ka_q, cs); rv = apr_pollset_add(event_pollset, &cs->pfd); @@ -1355,6 +1400,11 @@ static void process_socket(apr_thread_t *thd, apr_pool_t * p, apr_socket_t * soc } if (cs->pub.state == CONN_STATE_SUSPENDED) { + /* The connection is the suspending module's from now on, even if + * the child is exiting (ap_worker_sockets_shutdown() took the socket + * and shut it down): it owns the connection's pool and may resume it + */ + (void)unpublish_worker_socket(cs); cs->c->suspended_baton = cs; apr_atomic_inc32(&suspended_count); notify_suspend(cs); @@ -1860,6 +1910,11 @@ static void process_lingering_close(event_conn_state_t *cs) /* (Re)queue the connection to come back when readable */ update_reqevents_from_sense(cs, CONN_SENSE_WANT_READ); q = (cs->pub.state == CONN_STATE_LINGER_SHORT) ? short_linger_q : linger_q; + if (!unpublish_worker_socket(cs)) { + /* The child is exiting, close rather than hand it over */ + close_connection(cs); + return; + } apr_thread_mutex_lock(timeout_mutex); TO_QUEUE_APPEND(q, cs); rv = apr_pollset_add(event_pollset, &cs->pfd); @@ -2530,7 +2585,7 @@ static void *APR_THREAD_FUNC worker_thread(apr_thread_t * thd, void *dummy) dying ? SERVER_GRACEFUL : SERVER_READY, NULL); worker_pop: - if (workers_may_exit) { + if (apr_atomic_read32(&workers_may_exit)) { break; } if (dying && worker_thread_should_exit_early()) { @@ -2562,7 +2617,7 @@ static void *APR_THREAD_FUNC worker_thread(apr_thread_t * thd, void *dummy) goto worker_pop; } /* We got some other error. */ - else if (!workers_may_exit) { + else if (!apr_atomic_read32(&workers_may_exit)) { ap_log_error(APLOG_MARK, APLOG_CRIT, rv, ap_server_conf, APLOGNO(03099) "ap_queue_pop_socket failed"); } @@ -2579,14 +2634,16 @@ static void *APR_THREAD_FUNC worker_thread(apr_thread_t * thd, void *dummy) else { is_idle = 0; if (csd != NULL) { - worker_sockets[thread_slot] = csd; + /* process_socket() publishes the socket (worker_sockets) + * and unpublishes it before closing the connection or + * handing it over (see unpublish_worker_socket()) + */ process_socket(thd, ptrans, csd, cs, process_slot, thread_slot); - worker_sockets[thread_slot] = NULL; } } /* If there are deferred lingering closes, handle them now. */ - while (!workers_may_exit) { + while (!apr_atomic_read32(&workers_may_exit)) { cs = defer_linger_chain; if (!cs) { break; @@ -2599,9 +2656,8 @@ static void *APR_THREAD_FUNC worker_thread(apr_thread_t * thd, void *dummy) cs->chain = NULL; AP_DEBUG_ASSERT(cs->pub.state == CONN_STATE_LINGER); - worker_sockets[thread_slot] = csd = cs->pfd.desc.s; + csd = cs->pfd.desc.s; process_socket(thd, cs->p, csd, cs, process_slot, thread_slot); - worker_sockets[thread_slot] = NULL; } } @@ -2790,8 +2846,8 @@ static void setup_threads_runtime(void) lr->accept_func = ap_unixd_accept; } - worker_sockets = apr_pcalloc(pruntime, threads_per_child * - sizeof(apr_socket_t *)); + ap_worker_sockets_create(&worker_sockets, threads_per_child, + &workers_may_exit, pruntime); } /* XXX under some circumstances not understood, children can get stuck @@ -2855,7 +2911,9 @@ static void *APR_THREAD_FUNC start_threads(apr_thread_t * thd, void *dummy) resource_shortage = 1; signal_threads(ST_GRACEFUL); if (!listener_started) { - workers_may_exit = 1; + /* No ap_worker_sockets_shutdown() to wait for */ + apr_atomic_set32(&workers_may_exit, + AP_WORKERS_STOPPED); ap_queue_term(worker_queue); /* wake up main POD thread too */ kill(ap_my_pid, SIGTERM); @@ -3915,6 +3973,7 @@ static void setup_slave_conn(conn_rec *c, void *csd) } cs = apr_pcalloc(c->pool, sizeof(*cs)); + cs->tslot = -1; cs->c = c; cs->r = NULL; cs->sc = mcs->sc; diff --git a/server/mpm/worker/worker.c b/server/mpm/worker/worker.c index 616546fa3b7..baf59b13846 100644 --- a/server/mpm/worker/worker.c +++ b/server/mpm/worker/worker.c @@ -30,6 +30,7 @@ #include "apr_thread_mutex.h" #include "apr_proc_mutex.h" #include "apr_poll.h" +#include "apr_atomic.h" #include @@ -127,9 +128,9 @@ static int server_limit = 0; static int thread_limit = 0; static int had_healthy_child = 0; static volatile int dying = 0; -static int workers_may_exit = 0; static int start_thread_may_exit = 0; static int listener_may_exit = 0; +static /*atomic*/ apr_uint32_t workers_may_exit = AP_WORKERS_RUNNING; static int listener_is_wakeable = 0; /* Pollset supports APR_POLLSET_WAKEABLE */ static int requests_this_child; static int num_listensocks = 0; @@ -251,7 +252,7 @@ static apr_os_thread_t *listener_os_thread; * This ensures that on systems (i.e., Linux) where closing the worker * socket doesn't awake the worker thread when it is polling on the socket * (especially in apr_wait_for_io_or_timeout() when handling - * Keep-Alive connections), close_worker_sockets() and join_workers() + * Keep-Alive connections), ap_worker_sockets_shutdown() and join_workers() * still function in timely manner and allow ungraceful shutdowns to * proceed to completion. Otherwise join_workers() doesn't return * before the main process decides the child process is non-responsive @@ -259,19 +260,80 @@ static apr_os_thread_t *listener_os_thread; */ #define WORKER_SIGNAL AP_SIG_GRACEFUL -/* An array of socket descriptors in use by each thread used to - * perform a non-graceful (forced) shutdown of the server. */ -static apr_socket_t **worker_sockets; +/* The socket descriptors in use by each thread used to perform a + * non-graceful (forced) shutdown of the server (see mpm_fdqueue.h). A worker + * publishes its connection's socket once the connection is created and + * unpublishes it after the lingering reads (worker_lingering_close()), + * before closing it or clearing its pool. + */ +static ap_worker_sockets_t *worker_sockets; + +/* The lingering close timeouts of the core's ap_lingering_close(), which + * connection.c defines locally (not exported, mpm_event mirrors them too). + */ +#ifndef MAX_SECS_TO_LINGER +#define MAX_SECS_TO_LINGER 30 +#endif +#define SECONDS_TO_LINGER 2 -static void close_worker_sockets(void) +/* The core's ap_lingering_close(), but with the socket still published in + * the worker's slot during the lingering reads, so that an ungraceful stop + * (ap_worker_sockets_shutdown()'s shutdown()) makes them return at once; the + * socket is unpublished before it is closed. + */ +static void worker_lingering_close(conn_rec *c, apr_socket_t *sock, int slot) { - int i; - for (i = 0; i < threads_per_child; i++) { - if (worker_sockets[i]) { - apr_socket_close(worker_sockets[i]); - worker_sockets[i] = NULL; - } + char dummybuf[512]; + apr_size_t nbytes; + apr_time_t now, timeup = 0; + apr_socket_t *csd = ap_get_conn_socket(c); + + if (!csd) { + /* Be safe with third-party modules that: + * ap_set_core_module_config(c->conn_config, NULL) + * to no-op ap_lingering_close(); sock is unpublished here since + * clearing the connection's pool closes it. + */ + c->aborted = 1; + (void)ap_worker_sockets_unpublish(worker_sockets, slot, sock); + return; } + + if (!ap_start_lingering_close(c)) { + /* Read available data from the client whilst it continues sending + * it, for a maximum time of MAX_SECS_TO_LINGER. If the client + * does not send any data within SECONDS_TO_LINGER, give up. + */ + apr_socket_timeout_set(csd, apr_time_from_sec(SECONDS_TO_LINGER)); + apr_socket_opt_set(csd, APR_INCOMPLETE_READ, 1); + + /* The common path here is that the initial apr_socket_recv() call + * will return 0 bytes read; so that case must avoid the expensive + * apr_time_now() call and time arithmetic. */ + do { + nbytes = sizeof(dummybuf); + if (apr_socket_recv(csd, dummybuf, &nbytes) || nbytes == 0) + break; + + now = apr_time_now(); + if (timeup == 0) { + /* First time through: wait for MAX_SECS_TO_LINGER, or + * SECONDS_TO_LINGER only if some module requested a + * shortened waiting period (e.g. mitigating DoS attacks). + */ + if (apr_table_get(c->notes, "short-lingering-close")) { + timeup = now + apr_time_from_sec(SECONDS_TO_LINGER); + } + else { + timeup = now + apr_time_from_sec(MAX_SECS_TO_LINGER); + } + continue; + } + } while (now < timeup); + } + + (void)ap_worker_sockets_unpublish(worker_sockets, slot, sock); + apr_socket_close(csd); } static void wakeup_listener(void) @@ -330,9 +392,11 @@ static void signal_threads(int mode) * workers to exit once it has stopped accepting new connections */ if (mode == ST_UNGRACEFUL) { - workers_may_exit = 1; + /* ap_worker_sockets_shutdown() sets AP_WORKERS_STOPPED once done */ + apr_atomic_set32(&workers_may_exit, AP_WORKERS_STOPPING); ap_queue_interrupt_all(worker_queue); - close_worker_sockets(); /* forcefully kill all current connections */ + /* forcefully kill all current connections */ + ap_worker_sockets_shutdown(worker_sockets); } ap_run_child_stopping(pchild, mode == ST_GRACEFUL); @@ -492,9 +556,10 @@ static void process_socket(apr_thread_t *thd, apr_pool_t *p, apr_socket_t *sock, current_conn = ap_run_create_connection(p, ap_server_conf, sock, conn_id, sbh, bucket_alloc); if (current_conn) { + ap_worker_sockets_publish(worker_sockets, my_thread_num, sock); current_conn->current_thread = thd; ap_process_connection(current_conn, sock); - ap_lingering_close(current_conn); + worker_lingering_close(current_conn, sock, my_thread_num); } } @@ -773,7 +838,7 @@ static void * APR_THREAD_FUNC worker_thread(apr_thread_t *thd, void * dummy) unblock_signal(WORKER_SIGNAL); #endif - while (!workers_may_exit) { + while (!apr_atomic_read32(&workers_may_exit)) { if (!is_idle) { rv = ap_queue_info_set_idle(worker_queue_info, last_ptrans); last_ptrans = NULL; @@ -790,7 +855,7 @@ static void * APR_THREAD_FUNC worker_thread(apr_thread_t *thd, void * dummy) ap_update_child_status_from_indexes(process_slot, thread_slot, SERVER_READY, NULL); worker_pop: - if (workers_may_exit) { + if (apr_atomic_read32(&workers_may_exit)) { break; } rv = ap_queue_pop_socket(worker_queue, &csd, &ptrans); @@ -817,17 +882,19 @@ static void * APR_THREAD_FUNC worker_thread(apr_thread_t *thd, void * dummy) goto worker_pop; } /* We got some other error. */ - else if (!workers_may_exit) { + else if (!apr_atomic_read32(&workers_may_exit)) { ap_log_error(APLOG_MARK, APLOG_CRIT, rv, ap_server_conf, APLOGNO(03139) "ap_queue_pop_socket failed"); } continue; } is_idle = 0; - worker_sockets[thread_slot] = csd; bucket_alloc = apr_bucket_alloc_create(ptrans); + /* Publishes csd once the connection is created (worker_sockets), + * and unpublishes it before closing it or clearing ptrans (see + * worker_lingering_close()) + */ process_socket(thd, ptrans, csd, process_slot, thread_slot, bucket_alloc); - worker_sockets[thread_slot] = NULL; requests_this_child--; apr_pool_clear(ptrans); last_ptrans = ptrans; @@ -952,8 +1019,8 @@ static void setup_threads_runtime(void) lr->accept_func = ap_unixd_accept; } - worker_sockets = apr_pcalloc(pruntime, threads_per_child * - sizeof(apr_socket_t *)); + ap_worker_sockets_create(&worker_sockets, threads_per_child, + &workers_may_exit, pruntime); } /* XXX under some circumstances not understood, children can get stuck @@ -1011,7 +1078,9 @@ static void * APR_THREAD_FUNC start_threads(apr_thread_t *thd, void *dummy) resource_shortage = 1; signal_threads(ST_GRACEFUL); if (!listener_started) { - workers_may_exit = 1; + /* No ap_worker_sockets_shutdown() to wait for */ + apr_atomic_set32(&workers_may_exit, + AP_WORKERS_STOPPED); ap_queue_term(worker_queue); /* wake up main POD thread too */ kill(ap_my_pid, SIGTERM); diff --git a/server/mpm_fdqueue.c b/server/mpm_fdqueue.c index 3697ca722f6..79b72445985 100644 --- a/server/mpm_fdqueue.c +++ b/server/mpm_fdqueue.c @@ -531,4 +531,70 @@ apr_status_t ap_queue_term(fd_queue_t *queue) return queue_interrupt(queue, 1, 1); } +struct ap_worker_sockets_t +{ + apr_socket_t **slots; /* one per worker thread, atomic */ + int nslots; + apr_uint32_t *may_exit; /* the MPM's workers_may_exit, atomic */ +}; + +apr_status_t ap_worker_sockets_create(ap_worker_sockets_t **pws, + int nslots, apr_uint32_t *may_exit, + apr_pool_t *p) +{ + ap_worker_sockets_t *ws = apr_pcalloc(p, sizeof(*ws)); + + ws->slots = apr_pcalloc(p, nslots * sizeof(apr_socket_t *)); + ws->nslots = nslots; + ws->may_exit = may_exit; + *pws = ws; + return APR_SUCCESS; +} + +void ap_worker_sockets_publish(ap_worker_sockets_t *ws, int slot, + apr_socket_t *csd) +{ + apr_socket_t *prev; + + prev = apr_atomic_xchgptr((void *)&ws->slots[slot], csd); + AP_DEBUG_ASSERT(prev == NULL); + (void)prev; + /* After the publication (ap_worker_sockets_shutdown() may have scanned + * the slots before it) + */ + if (apr_atomic_read32(ws->may_exit)) { + apr_socket_shutdown(csd, APR_SHUTDOWN_READWRITE); + } +} + +int ap_worker_sockets_unpublish(ap_worker_sockets_t *ws, int slot, + apr_socket_t *csd) +{ + if (apr_atomic_casptr((void *)&ws->slots[slot], NULL, csd) == csd) { + return 1; + } + /* ap_worker_sockets_shutdown() took it: wait until it's done with it, + * which doesn't take long (a shutdown() per published socket, waiting + * for no worker) + */ + while (apr_atomic_read32(ws->may_exit) == AP_WORKERS_STOPPING) { + apr_sleep(apr_time_from_msec(1)); + } + return 0; +} + +void ap_worker_sockets_shutdown(ap_worker_sockets_t *ws) +{ + int i; + + AP_DEBUG_ASSERT(apr_atomic_read32(ws->may_exit) == AP_WORKERS_STOPPING); + for (i = 0; i < ws->nslots; i++) { + apr_socket_t *csd = apr_atomic_xchgptr((void *)&ws->slots[i], NULL); + if (csd) { + apr_socket_shutdown(csd, APR_SHUTDOWN_READWRITE); + } + } + apr_atomic_set32(ws->may_exit, AP_WORKERS_STOPPED); +} + #endif /* APR_HAS_THREADS */ diff --git a/server/mpm_fdqueue.h b/server/mpm_fdqueue.h index 0dd558b938a..d7c73ebee48 100644 --- a/server/mpm_fdqueue.h +++ b/server/mpm_fdqueue.h @@ -104,6 +104,53 @@ AP_DECLARE(apr_status_t) ap_queue_interrupt_all(fd_queue_t *queue); AP_DECLARE(apr_status_t) ap_queue_interrupt_one(fd_queue_t *queue); AP_DECLARE(apr_status_t) ap_queue_term(fd_queue_t *queue); +/* The workers' sockets, for the child's main thread to shut them down on an + * ungraceful stop. A worker publishes its connection's socket in its slot + * (NULL otherwise) and unpublishes it (CAS socket -> NULL) before closing it, + * clearing its pool or handing the connection over. One thread at a time owns + * the socket: + * - on an ungraceful stop the MPM stores AP_WORKERS_STOPPING in its + * workers_may_exit (passed at create), then the main thread takes each + * published socket (xchg) and shuts it down, never closing it nor touching + * its pool, then stores AP_WORKERS_STOPPED (ap_worker_sockets_shutdown()); + * - a worker whose unpublish fails waits for AP_WORKERS_STOPPED, then owns + * the socket and pool again; + * - after publishing, a worker re-reads the flag and shuts its socket down if + * set, so a socket published after the main thread's scan is shut down too + * (the main thread stores the flag before swapping the slot, the worker + * swaps the slot before re-reading the flag: their swaps on the same slot + * order them, so at least one sees the other). + */ +typedef struct ap_worker_sockets_t ap_worker_sockets_t; /* opaque */ + +/* The values of an MPM's workers_may_exit (an apr_uint32_t, read and written + * with apr_atomic_*32()): nonzero once the workers must exit; on an + * ungraceful stop, AP_WORKERS_STOPPING while ap_worker_sockets_shutdown() + * shuts the published sockets down, then AP_WORKERS_STOPPED. + */ +#define AP_WORKERS_RUNNING 0 +#define AP_WORKERS_STOPPING 1 +#define AP_WORKERS_STOPPED 2 + +/* nslots worker slots, ordered with the MPM's workers_may_exit */ +AP_DECLARE(apr_status_t) ap_worker_sockets_create(ap_worker_sockets_t **ws, + int nslots, + apr_uint32_t *may_exit, + apr_pool_t *p); +/* Publish the connection's socket in the worker's slot; if the workers may + * exit already, shut it down so that the connection ends at once */ +AP_DECLARE(void) ap_worker_sockets_publish(ap_worker_sockets_t *ws, int slot, + apr_socket_t *csd); +/* Unpublish it before closing it or handing its connection over; if + * ap_worker_sockets_shutdown() took it, wait until it's done with it (not + * long) and return 0: the child is exiting */ +AP_DECLARE(int) ap_worker_sockets_unpublish(ap_worker_sockets_t *ws, int slot, + apr_socket_t *csd); +/* On an ungraceful stop, after the MPM set *may_exit to AP_WORKERS_STOPPING: + * shut the published sockets down (never closing them), then set + * AP_WORKERS_STOPPED */ +AP_DECLARE(void) ap_worker_sockets_shutdown(ap_worker_sockets_t *ws); + #endif /* APR_HAS_THREADS */ #endif /* MPM_FDQUEUE_H */ From a32cb60aa5f9536dc0d7f95d0e28be5c12a07a47 Mon Sep 17 00:00:00 2001 From: ylavic Date: Mon, 16 Nov 2026 15:30:07 +0100 Subject: [PATCH 02/10] mpm_event, mpm_worker: free what the child still holds before its pools go in one-process mode. On an ungraceful stop the listener exits at once, closing none of the connections it holds (timeout queues and pollset, deferred lingering closes, the workers' queue): their pools, children of pconf, go only with pconf. In one-process mode (-X) that is at exit, after pchild was destroyed, so the modules' cleanups of these connections may use what pchild freed: mod_http2's session cleanup crashes in h2_workers_join() for an HTTP/2 connection waiting for I/O at SIGTERM. Once the workers may exit, make the workers close the connection rather than hand it over to the listener, and once the threads are joined make the main thread close what is left in clean_child_exit(), before pchild goes. Do it in one-process mode only, where the crash is (a forked child never destroys pconf), and only if the listener was joined, so that no MPM thread uses the queues or the pollset anymore. Nothing is left on a graceful stop. A suspended connection resumed after this (or whose resume timer is dropped here) still goes with pconf. The idle (recycled) connection pools are children of pconf, created after pruntime, so pconf destroys them before pruntime's cleanup of the queue info frees its idle list, which destroys them again: "double free or corruption" at exit with -X, the only mode where pconf is destroyed (by ap_terminate() from atexit()). The listener frees the idle list when it stops, but a worker can push its pool concurrently: an intermittent abort. Free the idle pools in clean_child_exit() in one-process mode too, in both MPMs. * server/mpm/event/event.c: (process_socket, process_lingering_close): Close the connection rather than hand it over to the listener once the workers may exit. (join_workers): Return whether the listener was joined. (just_close, close_held_connections): New. (listener_joined): New, set by child_main() from join_workers(). (clean_child_exit): In one-process mode, call close_held_connections() when the listener was joined, and free the idle pools. (child_main): Set listener_joined from join_workers(). * server/mpm/worker/worker.c: (clean_child_exit): Free the idle pools in one-process mode. --- server/mpm/event/event.c | 112 +++++++++++++++++++++++++++++++++---- server/mpm/worker/worker.c | 8 +++ 2 files changed, 110 insertions(+), 10 deletions(-) diff --git a/server/mpm/event/event.c b/server/mpm/event/event.c index 07d05755d67..5d8994abbfb 100644 --- a/server/mpm/event/event.c +++ b/server/mpm/event/event.c @@ -608,8 +608,8 @@ static void close_socket_nonblocking_(apr_socket_t *csd, * recording the slot, and unpublishes it before closing the connection * (close_connection()) or handing it over (to the listener's queues and * pollset, or to the module suspending it); if ap_worker_sockets_shutdown() - * took it, the worker closes the connection rather than hand it over to the - * listener. + * took it, or once the workers may exit, the worker closes the connection + * rather than hand it over to the listener (see close_held_connections()). */ static void publish_worker_socket(event_conn_state_t *cs, int tslot) { @@ -829,6 +829,11 @@ static const char *event_get_name(void) return "event"; } +/* Set by child_main() once join_workers() joined the listener */ +static int listener_joined = 0; + +static void close_held_connections(void); + /* a clean exit from a child with proper cleanup */ static void clean_child_exit(int code) __attribute__ ((noreturn)); static void clean_child_exit(int code) @@ -839,6 +844,17 @@ static void clean_child_exit(int code) } if (pchild) { + if (one_process) { + /* What pconf's destruction would otherwise get wrong at exit */ + if (listener_joined) { + close_held_connections(); + } + if (worker_queue_info) { + /* the idle pools would be destroyed twice: as pconf's children, + * then from the queue info's cleanup */ + ap_queue_info_free_idle_pools(worker_queue_info); + } + } ap_run_child_stopped(pchild, terminate_mode == ST_GRACEFUL); apr_pool_destroy(pchild); } @@ -1271,7 +1287,8 @@ static void process_socket(apr_thread_t *thd, apr_pool_t * p, apr_socket_t * soc * the default is CONN_SENSE_WANT_READ still. */ update_reqevents_from_sense(cs, CONN_SENSE_WANT_READ); - if (!unpublish_worker_socket(cs)) { + if (!unpublish_worker_socket(cs) + || apr_atomic_read32(&workers_may_exit)) { /* The child is exiting, close rather than hand it over */ close_connection(cs); return; @@ -1319,7 +1336,8 @@ static void process_socket(apr_thread_t *thd, apr_pool_t * p, apr_socket_t * soc /* Add work to pollset. */ cs->pub.sense = CONN_SENSE_DEFAULT; update_reqevents_from_sense(cs, CONN_SENSE_WANT_WRITE); - if (!unpublish_worker_socket(cs)) { + if (!unpublish_worker_socket(cs) + || apr_atomic_read32(&workers_may_exit)) { /* The child is exiting, close rather than hand it over */ close_connection(cs); return; @@ -1375,7 +1393,8 @@ static void process_socket(apr_thread_t *thd, apr_pool_t * p, apr_socket_t * soc /* Add work to pollset. */ cs->pub.sense = CONN_SENSE_DEFAULT; update_reqevents_from_sense(cs, CONN_SENSE_WANT_READ); - if (!unpublish_worker_socket(cs)) { + if (!unpublish_worker_socket(cs) + || apr_atomic_read32(&workers_may_exit)) { /* The child is exiting, close rather than hand it over */ close_connection(cs); return; @@ -1910,7 +1929,8 @@ static void process_lingering_close(event_conn_state_t *cs) /* (Re)queue the connection to come back when readable */ update_reqevents_from_sense(cs, CONN_SENSE_WANT_READ); q = (cs->pub.state == CONN_STATE_LINGER_SHORT) ? short_linger_q : linger_q; - if (!unpublish_worker_socket(cs)) { + if (!unpublish_worker_socket(cs) + || apr_atomic_read32(&workers_may_exit)) { /* The child is exiting, close rather than hand it over */ close_connection(cs); return; @@ -2966,9 +2986,12 @@ static void *APR_THREAD_FUNC start_threads(apr_thread_t * thd, void *dummy) return NULL; } -static void join_workers(apr_thread_t * listener, apr_thread_t ** threads) +/* Join the listener and the workers. Returns nonzero if the listener was + * joined, zero if it never started or join_workers() gave up on it. + */ +static int join_workers(apr_thread_t * listener, apr_thread_t ** threads) { - int i; + int i, joined = 0; apr_status_t rv, thread_rv; if (listener) { @@ -3003,6 +3026,9 @@ static void join_workers(apr_thread_t * listener, apr_thread_t ** threads) ap_log_error(APLOG_MARK, APLOG_CRIT, rv, ap_server_conf, APLOGNO(00476) "apr_thread_join: unable to join listener thread"); } + else { + joined = 1; + } } } @@ -3016,6 +3042,72 @@ static void join_workers(apr_thread_t * listener, apr_thread_t ** threads) } } } + + return joined; +} + +/* process_timeout_queue() callback: close the connection */ +static int just_close(event_conn_state_t *cs) +{ + close_connection(cs); + return 1; +} + +/* In one-process mode, once the listener and the workers are joined, close + * the connections the listener left on an ungraceful stop (timeout queues, + * deferred lingering closes, the workers' queue), so that their cleanups run + * while pchild is alive rather than with pconf after it. timeout_mutex is + * still taken since a module may resume a suspended connection from its own + * thread; a connection resumed after this is not closed here. + */ +static void close_held_connections(void) +{ + event_conn_state_t *cs; + apr_socket_t *csd; + apr_pool_t *ptrans; + timer_event_t *te; + + apr_thread_mutex_lock(timeout_mutex); + process_timeout_queue(waitio_q, 0, just_close); + process_timeout_queue(write_completion_q, 0, just_close); + process_timeout_queue(keepalive_q, 0, just_close); + process_timeout_queue(linger_q, 0, just_close); + process_timeout_queue(short_linger_q, 0, just_close); + apr_thread_mutex_unlock(timeout_mutex); + + cs = apr_atomic_xchgptr((void *)&defer_linger_chain, NULL); + while (cs) { + event_conn_state_t *next = cs->chain; + cs->chain = NULL; + close_connection(cs); + cs = next; + } + + /* Terminated (a noop if the listener did it when exiting), so this does + * not block: APR_EOF once empty. + */ + ap_queue_term(worker_queue); + while (ap_queue_pop_something(worker_queue, &csd, (void **)&cs, + &ptrans, &te) == APR_SUCCESS) { + if (te) { + /* Dropped, its callback not called */ + apr_thread_mutex_lock(g_timer_skiplist_mtx); + APR_RING_INSERT_TAIL(&timer_free_ring.link, te, timer_event_t, + link); + apr_thread_mutex_unlock(g_timer_skiplist_mtx); + } + else if (cs) { + close_connection(cs); + } + else { + if (csd) { + close_socket_nonblocking(csd); + } + if (ptrans) { + ap_queue_info_push_pool(worker_queue_info, ptrans); + } + } + } } static void join_start_thread(apr_thread_t * start_thread_id) @@ -3175,7 +3267,7 @@ static void child_main(int child_num_arg, int child_bucket) * If the worker hasn't exited, then this blocks until * they have (then cleans up). */ - join_workers(ts->listener, threads); + listener_joined = join_workers(ts->listener, threads); } else { /* !one_process */ /* remove SIGTERM from the set of blocked signals... if one of @@ -3219,7 +3311,7 @@ static void child_main(int child_num_arg, int child_bucket) ap_log_error(APLOG_MARK, APLOG_TRACE1, 0, ap_server_conf, "%s termination received, joining workers", rv == AP_MPM_PODX_GRACEFUL ? "graceful" : "ungraceful"); - join_workers(ts->listener, threads); + listener_joined = join_workers(ts->listener, threads); ap_log_error(APLOG_MARK, APLOG_TRACE1, 0, ap_server_conf, "%s termination, workers joined, exiting", rv == AP_MPM_PODX_GRACEFUL ? "graceful" : "ungraceful"); diff --git a/server/mpm/worker/worker.c b/server/mpm/worker/worker.c index baf59b13846..e25de4e42f9 100644 --- a/server/mpm/worker/worker.c +++ b/server/mpm/worker/worker.c @@ -513,6 +513,14 @@ static void clean_child_exit(int code) } if (pchild) { + if (one_process) { + /* What pconf's destruction would otherwise get wrong at exit */ + if (worker_queue_info) { + /* the idle pools would be destroyed twice: as pconf's children, + * then from the queue info's cleanup */ + ap_queue_info_free_idle_pools(worker_queue_info); + } + } ap_run_child_stopped(pchild, terminate_mode == ST_GRACEFUL); apr_pool_destroy(pchild); } From 67d98f310279b66a4f65e9987ec2066df7e1f29e Mon Sep 17 00:00:00 2001 From: ylavic Date: Mon, 16 Nov 2026 17:39:23 +0100 Subject: [PATCH 03/10] mpm_event, mpm_worker: _exit() forked children. Call _exit() rather than exit() in forked children, so that the atexit() handlers inherited from the parent (e.g. OpenSSL's, registered by default) don't run there, possibly while threads the child could not join or modules' threads still use what they clean up. Nor is any atexit() handler a safe place for a child's own cleanup (after pchild, other threads maybe running): pchild's cleanups and the child_stopped hooks still run. This also skips gcov's and LeakSanitizer's handlers in the children (-X still calls exit()) and stdio flushing, which httpd's logs don't use. Keep exit() with -X, which runs ap_terminate() from atexit(). * server/mpm/event/event.c: (clean_child_exit): Call _exit() unless in ONE_PROCESS mode. * server/mpm/worker/worker.c: (clean_child_exit): Call _exit() unless in ONE_PROCESS mode. --- server/mpm/event/event.c | 7 ++++++- server/mpm/worker/worker.c | 7 ++++++- 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/server/mpm/event/event.c b/server/mpm/event/event.c index 5d8994abbfb..85287e52f92 100644 --- a/server/mpm/event/event.c +++ b/server/mpm/event/event.c @@ -861,9 +861,14 @@ static void clean_child_exit(int code) if (one_process) { event_note_child_stopped(/* slot */ 0, 0, 0); + /* Run ap_terminate() as an atexit() handler. */ + exit(code); } - exit(code); + /* A forked child does not run the atexit() handlers inherited from the + * parent process (e.g. OpenSSL's, registered by default). + */ + _exit(code); } /* just_die() is the SIGTERM handler installed in the child immediately diff --git a/server/mpm/worker/worker.c b/server/mpm/worker/worker.c index e25de4e42f9..53ae4781935 100644 --- a/server/mpm/worker/worker.c +++ b/server/mpm/worker/worker.c @@ -527,9 +527,14 @@ static void clean_child_exit(int code) if (one_process) { worker_note_child_killed(/* slot */ 0, 0, 0); + /* Run ap_terminate() as an atexit() handler. */ + exit(code); } - exit(code); + /* A forked child does not run the atexit() handlers inherited from the + * parent process (e.g. OpenSSL's, registered by default). + */ + _exit(code); } static void just_die(int sig) From cd4bdbcc219015a1724099aadc3d94080709b9dd Mon Sep 17 00:00:00 2001 From: ylavic Date: Tue, 27 Oct 2026 10:53:56 +0100 Subject: [PATCH 04/10] mpm_event: wake up the listener out of the timers' and the timeout queues' mutexes. Waking up the listener is a syscall, which should not be made under a mutex taken for every timer registered (g_timer_skiplist_mtx) or every connection added to a timeout queue (timeout_mutex). Decide it under the mutex and wake up the listener after releasing it (once woken up, the listener re-reads the next expiry all the same). Rename the existing wakeup_listener(), which makes the listener exit, to shutdown_listener(), so that wakeup_listener() names the plain poll() wake-up of the listener. * server/mpm/event/event.c: (event_get_timer_event): Call apr_pollset_wakeup() after unlocking g_timer_skiplist_mtx. (TO_QUEUE_APPEND): Return whether the listener has to be woken up. (wakeup_listener): New, wake up the listener if it's poll()ing. (shutdown_listener): Renamed from wakeup_listener(), callers updated. (process_socket, event_resume_suspended, process_lingering_close): Wake up the listener after releasing timeout_mutex when TO_QUEUE_APPEND() says so. --- server/mpm/event/event.c | 69 +++++++++++++++++++++++++++++----------- 1 file changed, 51 insertions(+), 18 deletions(-) diff --git a/server/mpm/event/event.c b/server/mpm/event/event.c index 85287e52f92..3809fb4c517 100644 --- a/server/mpm/event/event.c +++ b/server/mpm/event/event.c @@ -299,8 +299,11 @@ static volatile apr_time_t queues_next_expiry; /* * Macros for accessing struct timeout_queue. * For TO_QUEUE_APPEND and TO_QUEUE_REMOVE, timeout_mutex must be held. + * TO_QUEUE_APPEND returns whether the listener has to be woken up to update + * its poll()ing timeout, for the caller to do it once the mutex released + * (wakeup_listener()). */ -static void TO_QUEUE_APPEND(struct timeout_queue *q, event_conn_state_t *el) +static int TO_QUEUE_APPEND(struct timeout_queue *q, event_conn_state_t *el) { apr_time_t elem_expiry; apr_time_t next_expiry; @@ -317,10 +320,17 @@ static void TO_QUEUE_APPEND(struct timeout_queue *q, event_conn_state_t *el) next_expiry = queues_next_expiry; if (!next_expiry || next_expiry > elem_expiry + TIMEOUT_FUDGE_FACTOR) { queues_next_expiry = elem_expiry; - /* Unblock the poll()ing listener for it to update its timeout. */ - if (listener_is_wakeable) { - apr_pollset_wakeup(event_pollset); - } + return 1; + } + return 0; +} + +/* Wake up the listener if it's poll()ing, for it to see what changed (its + * timeout) */ +static APR_INLINE void wakeup_listener(void) +{ + if (listener_is_wakeable) { + apr_pollset_wakeup(event_pollset); } } @@ -634,10 +644,10 @@ static int unpublish_worker_socket(event_conn_state_t *cs) cs->pfd.desc.s); } -static void wakeup_listener(void) +static void shutdown_listener(void) { ap_log_error(APLOG_MARK, APLOG_DEBUG, 0, ap_server_conf, - "wake up listener%s", listener_may_exit ? " again" : ""); + "shutting down listener%s", listener_may_exit ? " again" : ""); listener_may_exit = 1; disable_listensocks(); @@ -689,7 +699,7 @@ static void signal_threads(int mode) /* in case we weren't called from the listener thread, wake up the * listener thread */ - wakeup_listener(); + shutdown_listener(); /* for ungraceful termination, let the workers exit now; * for graceful termination, the listener thread will notify the @@ -1130,6 +1140,7 @@ static void process_socket(apr_thread_t *thd, apr_pool_t * p, apr_socket_t * soc int clogging = 0, from_wc_q = 0; apr_status_t rv; int rc = OK; + int wakeup; if (cs == NULL) { /* This is a new connection */ listener_poll_type *pt = apr_pcalloc(p, sizeof(*pt)); @@ -1299,7 +1310,7 @@ static void process_socket(apr_thread_t *thd, apr_pool_t * p, apr_socket_t * soc return; } apr_thread_mutex_lock(timeout_mutex); - TO_QUEUE_APPEND(cs->sc->io_q, cs); + wakeup = TO_QUEUE_APPEND(cs->sc->io_q, cs); rv = apr_pollset_add(event_pollset, &cs->pfd); if (rv != APR_SUCCESS && !APR_STATUS_IS_EEXIST(rv)) { AP_DEBUG_ASSERT(0); @@ -1313,6 +1324,9 @@ static void process_socket(apr_thread_t *thd, apr_pool_t * p, apr_socket_t * soc } else { apr_thread_mutex_unlock(timeout_mutex); + if (wakeup) { + wakeup_listener(); + } } return; } @@ -1348,7 +1362,7 @@ static void process_socket(apr_thread_t *thd, apr_pool_t * p, apr_socket_t * soc return; } apr_thread_mutex_lock(timeout_mutex); - TO_QUEUE_APPEND(cs->sc->wc_q, cs); + wakeup = TO_QUEUE_APPEND(cs->sc->wc_q, cs); rv = apr_pollset_add(event_pollset, &cs->pfd); if (rv != APR_SUCCESS && !APR_STATUS_IS_EEXIST(rv)) { AP_DEBUG_ASSERT(0); @@ -1362,6 +1376,9 @@ static void process_socket(apr_thread_t *thd, apr_pool_t * p, apr_socket_t * soc } else { apr_thread_mutex_unlock(timeout_mutex); + if (wakeup) { + wakeup_listener(); + } } return; } @@ -1405,7 +1422,7 @@ static void process_socket(apr_thread_t *thd, apr_pool_t * p, apr_socket_t * soc return; } apr_thread_mutex_lock(timeout_mutex); - TO_QUEUE_APPEND(cs->sc->ka_q, cs); + wakeup = TO_QUEUE_APPEND(cs->sc->ka_q, cs); rv = apr_pollset_add(event_pollset, &cs->pfd); if (rv != APR_SUCCESS && !APR_STATUS_IS_EEXIST(rv)) { AP_DEBUG_ASSERT(0); @@ -1419,6 +1436,9 @@ static void process_socket(apr_thread_t *thd, apr_pool_t * p, apr_socket_t * soc } else { apr_thread_mutex_unlock(timeout_mutex); + if (wakeup) { + wakeup_listener(); + } } return; } @@ -1444,6 +1464,7 @@ static void process_socket(apr_thread_t *thd, apr_pool_t * p, apr_socket_t * soc static apr_status_t event_resume_suspended (conn_rec *c) { event_conn_state_t* cs = (event_conn_state_t*) c->suspended_baton; + int wakeup; if (cs == NULL) { ap_log_cerror (APLOG_MARK, LOG_WARNING, 0, c, APLOGNO(02615) "event_resume_suspended: suspended_baton is NULL"); @@ -1464,9 +1485,12 @@ static apr_status_t event_resume_suspended (conn_rec *c) cs->pub.state = CONN_STATE_WRITE_COMPLETION; update_reqevents_from_sense(cs, CONN_SENSE_WANT_WRITE); apr_thread_mutex_lock(timeout_mutex); - TO_QUEUE_APPEND(cs->sc->wc_q, cs); + wakeup = TO_QUEUE_APPEND(cs->sc->wc_q, cs); apr_pollset_add(event_pollset, &cs->pfd); apr_thread_mutex_unlock(timeout_mutex); + if (wakeup) { + wakeup_listener(); + } } else { process_lingering_close(cs); @@ -1710,6 +1734,7 @@ static timer_event_t * event_get_timer_event(apr_time_t t, { timer_event_t *te; apr_time_t now = (t < 0) ? 0 : apr_time_now(); + int wakeup = 0; /* oh yeah, and make locking smarter/fine grained. */ @@ -1742,14 +1767,18 @@ static timer_event_t * event_get_timer_event(apr_time_t t, next_expiry = timers_next_expiry; if (!next_expiry || next_expiry > te->when + EVENT_FUDGE_FACTOR) { timers_next_expiry = te->when; - /* Unblock the poll()ing listener for it to update its timeout. */ - if (listener_is_wakeable) { - apr_pollset_wakeup(event_pollset); - } + wakeup = listener_is_wakeable; } } apr_thread_mutex_unlock(g_timer_skiplist_mtx); + /* Unblock the poll()ing listener for it to update its timeout, out of the + * mutex (a syscall). + */ + if (wakeup) { + apr_pollset_wakeup(event_pollset); + } + return te; } @@ -1872,6 +1901,7 @@ static void process_lingering_close(event_conn_state_t *cs) apr_size_t nbytes; apr_status_t rv; struct timeout_queue *q; + int wakeup; ap_log_cerror(APLOG_MARK, APLOG_TRACE6, 0, cs->c, "lingering close from state %i", (int)cs->pub.state); @@ -1941,7 +1971,7 @@ static void process_lingering_close(event_conn_state_t *cs) return; } apr_thread_mutex_lock(timeout_mutex); - TO_QUEUE_APPEND(q, cs); + wakeup = TO_QUEUE_APPEND(q, cs); rv = apr_pollset_add(event_pollset, &cs->pfd); if (rv != APR_SUCCESS && !APR_STATUS_IS_EEXIST(rv)) { AP_DEBUG_ASSERT(0); @@ -1954,6 +1984,9 @@ static void process_lingering_close(event_conn_state_t *cs) return; } apr_thread_mutex_unlock(timeout_mutex); + if (wakeup) { + wakeup_listener(); + } } /* call 'func' for all elements of 'q' above 'expiry'. @@ -3019,7 +3052,7 @@ static int join_workers(apr_thread_t * listener, apr_thread_t ** threads) /* listener has not stopped accepting yet */ ap_log_error(APLOG_MARK, APLOG_TRACE1, 0, ap_server_conf, "listener has not stopped accepting yet (%d iter)", iter); - wakeup_listener(); + shutdown_listener(); } if (iter > 10) { ap_log_error(APLOG_MARK, APLOG_DEBUG, 0, ap_server_conf, APLOGNO(00475) From 91b3d17b0345ba956658f00aa4be88dd512b866b Mon Sep 17 00:00:00 2001 From: ylavic Date: Mon, 16 Nov 2026 16:17:06 +0100 Subject: [PATCH 05/10] mpm_worker: rename wakeup_listener() to shutdown_listener(), as in mpm_event. mpm_worker's wakeup_listener() makes the listener exit, like mpm_event's former one now named shutdown_listener(): use the same name in both MPMs. * server/mpm/worker/worker.c: (shutdown_listener): Renamed from wakeup_listener(), callers updated. --- server/mpm/worker/worker.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/server/mpm/worker/worker.c b/server/mpm/worker/worker.c index 53ae4781935..7f946f3ec57 100644 --- a/server/mpm/worker/worker.c +++ b/server/mpm/worker/worker.c @@ -336,7 +336,7 @@ static void worker_lingering_close(conn_rec *c, apr_socket_t *sock, int slot) apr_socket_close(csd); } -static void wakeup_listener(void) +static void shutdown_listener(void) { listener_may_exit = 1; @@ -385,7 +385,7 @@ static void signal_threads(int mode) /* in case we weren't called from the listener thread, wake up the * listener thread */ - wakeup_listener(); + shutdown_listener(); /* for ungraceful termination, let the workers exit now; * for graceful termination, the listener thread will notify the @@ -1167,7 +1167,7 @@ static void join_workers(apr_thread_t *listener, apr_thread_t **threads, /* listener has not stopped accepting yet */ ap_log_error(APLOG_MARK, APLOG_TRACE1, 0, ap_server_conf, "listener has not stopped accepting yet (%d iter)", iter); - wakeup_listener(); + shutdown_listener(); } if (iter > 10) { ap_log_error(APLOG_MARK, APLOG_DEBUG, 0, ap_server_conf, APLOGNO(00276) From 53ad0535c70d47a6a43917636fedbefc1bb3fe69 Mon Sep 17 00:00:00 2001 From: ylavic Date: Mon, 16 Nov 2026 00:16:49 +0100 Subject: [PATCH 06/10] mpm_event: terminate the workers' queue in join_workers() too, failing pushes to it. The listener terminates the workers' queue when it exits, which lets the idle workers leave ap_queue_pop_something() on a graceful stop. join_workers() gives up on a listener that has not stopped accepting after 5 seconds (AH00475), which now happens only when a bug blocks it (in a module or in its loop), then joins the workers anyway (shutdown_listener() terminates only the queue info): the idle workers wait forever and the child never exits, keeping its scoreboard slot (the parent waits for it on a graceful stop without GracefulShutdownTimeout). Terminate the queue before joining the workers, a noop if the listener did it already: the forever-hang becomes an exit. Queued connections are still handled before APR_EOF. The stuck listener could still push to the terminated queue, which ap_queue_push_socket() and ap_queue_push_timer() AP_DEBUG_ASSERT()ed against (the element queued anyway without maintainer mode). Return APR_EOF instead, checked under the queue's mutex, the element not queued. The callers drop the connection or the timer event quietly (a debug log), the child stopping already: mpm_event's push2worker() closes the connection at once (close_connection(), not shutdown_connection() whose deferred lingering close no worker would take), without signaling a graceful stop; push_timer2worker() puts the timer event back on timer_free_ring, its callback not called, on any push failure (its callers ignored the status, losing the event); mpm_worker's listener closes the socket, as on any push failure. * server/mpm_fdqueue.c: (ap_queue_push_socket, ap_queue_push_timer): Return APR_EOF if the queue is terminated rather than assert it is not. * server/mpm/event/event.c: (push2worker): On APR_EOF, log at debug level, close an existing connection at once and don't signal a graceful stop. (push_timer2worker): Move after the timers' declarations, add a locked flag (g_timer_skiplist_mtx held), drop the timer event if it can't be queued. (listener_thread): Pass the locked flag to push_timer2worker(). (join_workers): Call ap_queue_term() before joining the workers. * server/mpm/worker/worker.c: (listener_thread): Log at debug level on APR_EOF. --- server/mpm/event/event.c | 62 +++++++++++++++++++++++++++++++------- server/mpm/worker/worker.c | 14 +++++++-- server/mpm_fdqueue.c | 13 ++++++-- 3 files changed, 74 insertions(+), 15 deletions(-) diff --git a/server/mpm/event/event.c b/server/mpm/event/event.c index 3809fb4c517..658f590b65b 100644 --- a/server/mpm/event/event.c +++ b/server/mpm/event/event.c @@ -1610,11 +1610,6 @@ static void init_serf(apr_pool_t *p) } #endif -static apr_status_t push_timer2worker(timer_event_t* te) -{ - return ap_queue_push_timer(worker_queue, te); -} - /* * Pre-condition: cs is neither in event_pollset nor a timeout queue * this function may only be called by the listener @@ -1630,13 +1625,29 @@ static apr_status_t push2worker(event_conn_state_t *cs, apr_socket_t *csd, } rc = ap_queue_push_socket(worker_queue, csd, cs, ptrans); if (rc != APR_SUCCESS) { - ap_log_error(APLOG_MARK, APLOG_CRIT, rc, ap_server_conf, APLOGNO(00471) - "push2worker: ap_queue_push_socket failed"); + int stopping = APR_STATUS_IS_EOF(rc); + if (stopping) { + /* The workers' queue is terminated, the child is stopping */ + ap_log_error(APLOG_MARK, APLOG_DEBUG, rc, ap_server_conf, + "push2worker: workers' queue terminated, " + "closing the connection"); + } + else { + ap_log_error(APLOG_MARK, APLOG_CRIT, rc, ap_server_conf, + APLOGNO(00471) "push2worker: " + "ap_queue_push_socket failed"); + } /* trash the connection; we couldn't queue the connected * socket to a worker */ if (cs) { - shutdown_connection(cs); + if (stopping) { + /* No worker would take a deferred lingering close */ + close_connection(cs); + } + else { + shutdown_connection(cs); + } } else { if (csd) { @@ -1646,7 +1657,9 @@ static apr_status_t push2worker(event_conn_state_t *cs, apr_socket_t *csd, ap_queue_info_push_pool(worker_queue_info, ptrans); } } - signal_threads(ST_GRACEFUL); + if (!stopping) { + signal_threads(ST_GRACEFUL); + } } return rc; @@ -1726,6 +1739,27 @@ static int timer_comp(void *a, void *b) static apr_thread_mutex_t *g_timer_skiplist_mtx; +/* Push a timer event to a worker, or drop it if it can't be queued, e.g. + * once the workers' queue is terminated (the child is stopping): back to + * timer_free_ring, its callback not called. Pre-condition: the caller holds + * g_timer_skiplist_mtx if locked is set. + */ +static void push_timer2worker(timer_event_t* te, int locked) +{ + apr_status_t rc = ap_queue_push_timer(worker_queue, te); + if (rc != APR_SUCCESS) { + ap_log_error(APLOG_MARK, APLOG_DEBUG, rc, ap_server_conf, + "push_timer2worker: timer event dropped"); + if (!locked) { + apr_thread_mutex_lock(g_timer_skiplist_mtx); + } + APR_RING_INSERT_TAIL(&timer_free_ring.link, te, timer_event_t, link); + if (!locked) { + apr_thread_mutex_unlock(g_timer_skiplist_mtx); + } + } +} + static timer_event_t * event_get_timer_event(apr_time_t t, ap_mpm_callback_fn_t *cbfn, void *baton, @@ -2193,7 +2227,7 @@ static void * APR_THREAD_FUNC listener_thread(apr_thread_t * thd, void *dummy) apr_pool_cleanup_run(te->pfds->pool, te->pfds, event_cleanup_poll_callback); } - push_timer2worker(te); + push_timer2worker(te, 1); } else { APR_RING_INSERT_TAIL(&timer_free_ring.link, te, @@ -2468,7 +2502,7 @@ static void * APR_THREAD_FUNC listener_thread(apr_thread_t * thd, void *dummy) baton->user_baton, 0, /* don't insert it */ NULL /* no associated socket callback */); - push_timer2worker(te); + push_timer2worker(te, 0); } /* We process the timeout queues here only when the global @@ -3070,6 +3104,12 @@ static int join_workers(apr_thread_t * listener, apr_thread_t ** threads) } } + /* The listener terminates the queue when it exits, but if it is stuck + * (not joined above) the workers would wait for it forever on a graceful + * stop. A noop if the listener did it already. + */ + ap_queue_term(worker_queue); + for (i = 0; i < threads_per_child; i++) { if (threads[i]) { /* if we ever created this thread */ rv = apr_thread_join(&thread_rv, threads[i]); diff --git a/server/mpm/worker/worker.c b/server/mpm/worker/worker.c index 7f946f3ec57..2a0fd523319 100644 --- a/server/mpm/worker/worker.c +++ b/server/mpm/worker/worker.c @@ -780,8 +780,18 @@ static void * APR_THREAD_FUNC listener_thread(apr_thread_t *thd, void * dummy) * socket to a worker */ apr_socket_close(csd); - ap_log_error(APLOG_MARK, APLOG_CRIT, rv, ap_server_conf, APLOGNO(03138) - "ap_queue_push_socket failed"); + if (APR_STATUS_IS_EOF(rv)) { + /* The workers' queue is terminated (stopping) */ + ap_log_error(APLOG_MARK, APLOG_DEBUG, rv, + ap_server_conf, + "workers' queue terminated, " + "connection closed"); + } + else { + ap_log_error(APLOG_MARK, APLOG_CRIT, rv, + ap_server_conf, APLOGNO(03138) + "ap_queue_push_socket failed"); + } } else { have_idle_worker = 0; diff --git a/server/mpm_fdqueue.c b/server/mpm_fdqueue.c index 79b72445985..e0cebfe4c60 100644 --- a/server/mpm_fdqueue.c +++ b/server/mpm_fdqueue.c @@ -378,6 +378,7 @@ apr_status_t ap_queue_create(fd_queue_t **pqueue, int capacity, apr_pool_t *p) * * precondition: ap_queue_info_wait_for_idler has already been called * to reserve an idle worker thread + * Returns APR_EOF if the queue is terminated, the socket is not queued. */ apr_status_t ap_queue_push_socket(fd_queue_t *queue, apr_socket_t *sd, void *sd_baton, @@ -390,7 +391,11 @@ apr_status_t ap_queue_push_socket(fd_queue_t *queue, return rv; } - AP_DEBUG_ASSERT(!queue->terminated); + /* Terminated (stopping), the caller drops the socket */ + if (queue->terminated) { + apr_thread_mutex_unlock(queue->one_big_mutex); + return APR_EOF; + } AP_DEBUG_ASSERT(!ap_queue_full(queue)); elem = &queue->data[queue->in++]; @@ -414,7 +419,11 @@ apr_status_t ap_queue_push_timer(fd_queue_t *queue, timer_event_t *te) return rv; } - AP_DEBUG_ASSERT(!queue->terminated); + /* Terminated (stopping), the caller drops the timer */ + if (queue->terminated) { + apr_thread_mutex_unlock(queue->one_big_mutex); + return APR_EOF; + } APR_RING_INSERT_TAIL(&queue->timers, te, timer_event_t, link); From befc33b0e2fe2cf7d8028752d121a7dd89738ff4 Mon Sep 17 00:00:00 2001 From: ylavic Date: Sun, 15 Nov 2026 21:14:30 +0100 Subject: [PATCH 07/10] mpm_event: allocate the threads' start info and the thread starter from pruntime, not malloc(). Each thread's start info is malloc()ed by the start thread and free()d by the thread it starts (leaked if that can't be created), the threads array is freed only at child_main()'s end (not on the clean_child_exit() paths), and the thread starter, its attributes and the start thread come from pchild after the child_init() hooks, which may have started threads using pchild (why pruntime exists), and are destroyed in clean_child_exit() while a listener join_workers() gave up on may still run. Allocate them all from pruntime, the MPM's pool for its threads, so they live as long as the threads using them. child_main() allocates from it only until it creates the start thread, the only one allocating from it then. join_workers() takes the thread starter. * server/mpm/event/event.c: (listener_thread, worker_thread): Don't free() the start info. (create_listener_thread, start_threads): Allocate it from pruntime. (join_workers): Take the thread starter. (child_main): Allocate the thread starter, its threads array and the thread attributes from pruntime, create the start thread on pruntime. --- server/mpm/event/event.c | 47 ++++++++++++++++------------------------ 1 file changed, 19 insertions(+), 28 deletions(-) diff --git a/server/mpm/event/event.c b/server/mpm/event/event.c index 658f590b65b..faef20c3bee 100644 --- a/server/mpm/event/event.c +++ b/server/mpm/event/event.c @@ -2130,7 +2130,6 @@ static void * APR_THREAD_FUNC listener_thread(apr_thread_t * thd, void *dummy) apr_time_t last_log; last_log = apr_time_now(); - free(ti); #if HAVE_SERF init_serf(apr_thread_pool_get(thd)); @@ -2640,8 +2639,6 @@ static void *APR_THREAD_FUNC worker_thread(apr_thread_t * thd, void *dummy) apr_status_t rv; int is_idle = 0; - free(ti); - ap_scoreboard_image->servers[process_slot][thread_slot].pid = ap_my_pid; ap_scoreboard_image->servers[process_slot][thread_slot].tid = apr_os_thread_current(); ap_scoreboard_image->servers[process_slot][thread_slot].generation = retained->mpm->my_generation; @@ -2778,7 +2775,7 @@ static void create_listener_thread(thread_starter * ts) proc_info *my_info; apr_status_t rv; - my_info = (proc_info *) ap_malloc(sizeof(proc_info)); + my_info = apr_palloc(pruntime, sizeof(proc_info)); my_info->pslot = my_child_num; my_info->tslot = -1; /* listener thread doesn't have a thread slot */ rv = ap_thread_create(&ts->listener, thread_attr, listener_thread, @@ -2976,7 +2973,7 @@ static void *APR_THREAD_FUNC start_threads(apr_thread_t * thd, void *dummy) continue; } - my_info = (proc_info *) ap_malloc(sizeof(proc_info)); + my_info = apr_palloc(pruntime, sizeof(proc_info)); my_info->pslot = my_child_num; my_info->tslot = i; @@ -3061,12 +3058,12 @@ static void *APR_THREAD_FUNC start_threads(apr_thread_t * thd, void *dummy) /* Join the listener and the workers. Returns nonzero if the listener was * joined, zero if it never started or join_workers() gave up on it. */ -static int join_workers(apr_thread_t * listener, apr_thread_t ** threads) +static int join_workers(thread_starter *ts) { int i, joined = 0; apr_status_t rv, thread_rv; - if (listener) { + if (ts->listener) { int iter; /* deal with a rare timing window which affects waking up the @@ -3093,7 +3090,7 @@ static int join_workers(apr_thread_t * listener, apr_thread_t ** threads) "the listener thread didn't stop accepting"); } else { - rv = apr_thread_join(&thread_rv, listener); + rv = apr_thread_join(&thread_rv, ts->listener); if (rv != APR_SUCCESS) { ap_log_error(APLOG_MARK, APLOG_CRIT, rv, ap_server_conf, APLOGNO(00476) "apr_thread_join: unable to join listener thread"); @@ -3111,8 +3108,8 @@ static int join_workers(apr_thread_t * listener, apr_thread_t ** threads) ap_queue_term(worker_queue); for (i = 0; i < threads_per_child; i++) { - if (threads[i]) { /* if we ever created this thread */ - rv = apr_thread_join(&thread_rv, threads[i]); + if (ts->threads[i]) { /* if we ever created this thread */ + rv = apr_thread_join(&thread_rv, ts->threads[i]); if (rv != APR_SUCCESS) { ap_log_error(APLOG_MARK, APLOG_CRIT, rv, ap_server_conf, APLOGNO(00477) "apr_thread_join: unable to join worker " @@ -3205,10 +3202,8 @@ static void join_start_thread(apr_thread_t * start_thread_id) static void child_main(int child_num_arg, int child_bucket) { - apr_thread_t **threads; apr_status_t rv; thread_starter *ts; - apr_threadattr_t *thread_attr; apr_thread_t *start_thread_id; int i; @@ -3286,15 +3281,17 @@ static void child_main(int child_num_arg, int child_bucket) /* clear the storage; we may not create all our threads immediately, * and we want a 0 entry to indicate a thread which was not created */ - threads = ap_calloc(threads_per_child, sizeof(apr_thread_t *)); - ts = apr_palloc(pchild, sizeof(*ts)); + ts = apr_pcalloc(pruntime, sizeof(*ts)); + ts->threads = apr_pcalloc(pruntime, + threads_per_child * sizeof(apr_thread_t *)); + ts->child_num_arg = child_num_arg; - apr_threadattr_create(&thread_attr, pchild); + apr_threadattr_create(&ts->threadattr, pruntime); /* 0 means PTHREAD_CREATE_JOINABLE */ - apr_threadattr_detach_set(thread_attr, 0); + apr_threadattr_detach_set(ts->threadattr, 0); if (ap_thread_stacksize != 0) { - rv = apr_threadattr_stacksize_set(thread_attr, ap_thread_stacksize); + rv = apr_threadattr_stacksize_set(ts->threadattr, ap_thread_stacksize); if (rv != APR_SUCCESS && rv != APR_ENOTIMPL) { ap_log_error(APLOG_MARK, APLOG_WARNING, rv, ap_server_conf, APLOGNO(02436) "WARNING: ThreadStackSize of %" APR_SIZE_T_FMT " is " @@ -3303,13 +3300,9 @@ static void child_main(int child_num_arg, int child_bucket) } } - ts->threads = threads; - ts->listener = NULL; - ts->child_num_arg = child_num_arg; - ts->threadattr = thread_attr; - - rv = ap_thread_create(&start_thread_id, thread_attr, start_threads, - ts, pchild); + /* The start thread allocates from pruntime, don't use it after here. */ + rv = ap_thread_create(&start_thread_id, ts->threadattr, start_threads, + ts, pruntime); if (rv != APR_SUCCESS) { ap_log_error(APLOG_MARK, APLOG_ALERT, rv, ap_server_conf, APLOGNO(00480) "ap_thread_create: unable to create worker thread"); @@ -3345,7 +3338,7 @@ static void child_main(int child_num_arg, int child_bucket) * If the worker hasn't exited, then this blocks until * they have (then cleans up). */ - listener_joined = join_workers(ts->listener, threads); + listener_joined = join_workers(ts); } else { /* !one_process */ /* remove SIGTERM from the set of blocked signals... if one of @@ -3389,7 +3382,7 @@ static void child_main(int child_num_arg, int child_bucket) ap_log_error(APLOG_MARK, APLOG_TRACE1, 0, ap_server_conf, "%s termination received, joining workers", rv == AP_MPM_PODX_GRACEFUL ? "graceful" : "ungraceful"); - listener_joined = join_workers(ts->listener, threads); + listener_joined = join_workers(ts); ap_log_error(APLOG_MARK, APLOG_TRACE1, 0, ap_server_conf, "%s termination, workers joined, exiting", rv == AP_MPM_PODX_GRACEFUL ? "graceful" : "ungraceful"); @@ -3399,8 +3392,6 @@ static void child_main(int child_num_arg, int child_bucket) ap_mpm_wait_for_extra_connections(); } - free(threads); - clean_child_exit(resource_shortage ? APEXIT_CHILDSICK : 0); } From 741ec9b3e2aedf2988fdb6a944be2f82fa1cf667 Mon Sep 17 00:00:00 2001 From: ylavic Date: Sun, 15 Nov 2026 21:56:02 +0100 Subject: [PATCH 08/10] mpm_event: initialize the child's main thread, scoreboard and privileges on pconf, not pchild. Nothing the child keeps until its exit should live in pchild, which clean_child_exit() destroys before the process ends: create the child's main thread, reopen the scoreboard and drop privileges on pconf, as the threads' runtime uses pruntime. No change of behavior is intended. * server/mpm/event/event.c: (child_main): Pass pconf rather than pchild to ap_thread_main_create(), ap_reopen_scoreboard() and ap_run_drop_privileges(). --- server/mpm/event/event.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/server/mpm/event/event.c b/server/mpm/event/event.c index faef20c3bee..7cb5992cf57 100644 --- a/server/mpm/event/event.c +++ b/server/mpm/event/event.c @@ -3223,7 +3223,7 @@ static void child_main(int child_num_arg, int child_bucket) #if AP_HAS_THREAD_LOCAL if (!one_process) { apr_thread_t *thd = NULL; - if ((rv = ap_thread_main_create(&thd, pchild))) { + if ((rv = ap_thread_main_create(&thd, pconf))) { ap_log_error(APLOG_MARK, APLOG_EMERG, rv, ap_server_conf, APLOGNO(10377) "Couldn't initialize child main thread"); clean_child_exit(APEXIT_CHILDFATAL); @@ -3240,10 +3240,10 @@ static void child_main(int child_num_arg, int child_bucket) } /*stuff to do before we switch id's, so we have permissions. */ - ap_reopen_scoreboard(pchild, NULL, 0); + ap_reopen_scoreboard(pconf, NULL, 0); /* done with init critical section */ - if (ap_run_drop_privileges(pchild, ap_server_conf)) { + if (ap_run_drop_privileges(pconf, ap_server_conf)) { clean_child_exit(APEXIT_CHILDFATAL); } From 2c4a119cf0957aa7264cb09130ea880496ab9acb Mon Sep 17 00:00:00 2001 From: ylavic Date: Mon, 16 Nov 2026 01:55:11 +0100 Subject: [PATCH 09/10] test: check ungraceful stops and restarts with busy or lingering connections, and the -X exit. Add tests stopping or restarting the server ungracefully while its workers are busy, checking that it does not wait for the connections (the time taken, and no "still did not exit" from the parent, which resends SIGTERM and SIGKILL) and serves again afterwards; and that with -X it exits with status 0 on SIGTERM, nothing crashing. * test/modules/core/test_008_restart.py: (_Sockets): New, client connections closed whatever happens. (TestUngracefulExit): New. (test_core_008_04_busy): New, partial requests and slow readers (skipped with mpm_motorz, which doesn't wake a child reading a request). (test_core_008_05_lingering): New, connections in their lingering close. (test_core_008_06_one_process): New, -X started and stopped by SIGTERM. --- test/modules/core/test_008_restart.py | 278 ++++++++++++++++++++++++++ 1 file changed, 278 insertions(+) diff --git a/test/modules/core/test_008_restart.py b/test/modules/core/test_008_restart.py index 5c414c61fa5..bd6733d61fc 100644 --- a/test/modules/core/test_008_restart.py +++ b/test/modules/core/test_008_restart.py @@ -1,8 +1,17 @@ +import os +import signal +import socket +import subprocess import sys +import tempfile +import threading +import time import pytest from pyhttpd.conf import HttpdConf +from pyhttpd.instance import port_open, signal_group +from pyhttpd.log import HttpdErrorLog class TestRestart: @@ -58,3 +67,272 @@ def test_core_008_03_repeated(self, env): for _ in range(2): assert env.apache_hard_restart() == 0 self.get(env, "after repeated restarts") + + +# How long an ungraceful restart or stop with busy connections may take as a +# whole (reading the configuration again and the new generation included). +# The parent's log tells whether the children were woken up (it resends +# SIGTERM after 3 seconds, "still did not exit"); this only bounds the rest, +# below the SIGKILL after 9 seconds, with room for slow runners. +EXIT_BOUND = 8.0 + +# Whatever the parent logs when a child does not exit when asked to +RE_STILL_RUNNING = "still did not exit" + + +def _log_since(env, pos): + with open(env.httpd_error_log.path, errors='replace') as fd: + fd.seek(pos) + return fd.read() + + +class _Sockets: + """Client connections of a test, closed whatever happens.""" + + def __init__(self, env): + self.env = env + self.host = f"test1.{env.http_tld}" + self.socks = [] + self.threads = [] + self.done = threading.Event() + + def connect(self, rcvbuf=None): + s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + self.socks.append(s) + if rcvbuf: + # before connect(), so that the window stays small and the + # server blocks writing the response + s.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, rcvbuf) + s.settimeout(10) + s.connect(('127.0.0.1', self.env.http_port)) + return s + + def partial_request(self): + """A request whose headers never end: a worker waits for the rest.""" + s = self.connect() + s.sendall(f"GET / HTTP/1.1\r\nHost: {self.host}\r\n".encode()) + return s + + def slow_reader(self): + """A large response read no further than its first bytes: a worker + is stuck writing the rest.""" + s = self.connect(rcvbuf=4096) + s.sendall(f"GET /big.bin HTTP/1.1\r\nHost: {self.host}\r\n\r\n".encode()) + first = s.recv(1024) + assert first.startswith(b"HTTP/1.1 200"), f"bad response: {first!r}" + return s + + def lingering(self): + """A client which goes on sending after a response telling it the + connection is closed: the server keeps reading, in its lingering + close, for as long as it does.""" + s = self.connect() + s.sendall(f"GET / HTTP/1.1\r\nHost: {self.host}\r\n" + f"Connection: close\r\n\r\n".encode()) + s.settimeout(0.1) + + def run(): + while not self.done.is_set(): + try: + s.sendall(b"x" * 16) + except OSError: + return + try: + s.recv(65536) + except (socket.timeout, BlockingIOError): + pass + except OSError: + return + time.sleep(0.1) + + t = threading.Thread(target=run, daemon=True) + t.start() + self.threads.append(t) + return s + + def __enter__(self): + return self + + def __exit__(self, *exc): + self.done.set() + for t in self.threads: + t.join(timeout=5) + for s in self.socks: + try: + s.close() + except OSError: + pass + return False + + +@pytest.mark.skipif(sys.platform == "win32", reason="signals and process groups") +class TestUngracefulExit: + """An ungraceful restart or stop does not wait for busy connections. + + The children are told to exit at once, and the connections their workers + are busy with have to be cut for them to do so: with mpm_worker a request + being read, a response being written or a lingering close each hold a + worker; with mpm_event the request being read does (the responses and + the lingering closes are mostly the listener's then). What is checked is + the parent's own log: it resends SIGTERM and SIGKILL to children which + still did not exit ("still did not exit"), and the test has no use for a + server which got there that way; and, loosely, the time it all takes. + """ + + @pytest.fixture(autouse=True, scope='class') + @classmethod + def _class_scope(cls, env): + threads = """ + StartServers 2 + ServerLimit 4 + ThreadLimit 4 + ThreadsPerChild 4 + MinSpareThreads 4 + MaxSpareThreads 12 + MaxRequestWorkers 16 + """ + conf = HttpdConf(env, extras={ + 'base': f""" + + {threads} + + + {threads} + + """, + }) + conf.add_vhost_test1() + conf.install() + env.httpd_error_log.add_ignored_lognos(['AH01239', 'AH01243']) + # a large file the server cannot write at once; sparse + big = os.path.join(env.server_docs_dir, 'test1', 'big.bin') + with open(big, 'wb') as fd: + fd.truncate(64 * 1024 * 1024) + assert env.apache_restart() == 0 + yield + os.unlink(big) + + @pytest.fixture(autouse=True) + def _serving(self, env): + # each test starts and leaves a running server + if not env.instance.is_running(): + assert env.apache_restart() == 0 + yield + if not env.instance.is_running(): + assert env.apache_restart() == 0 + + def get(self, env, when): + r = env.curl_get(env.mkurl("http", "test1", "/"), + options=['--max-time', '10']) + assert r.exit_code == 0, f"no answer {when}: {r.stderr}" + assert r.response['status'] == 200, f"bad status {when}" + + def check_log(self, env, pos, what): + text = _log_since(env, pos) + assert RE_STILL_RUNNING not in text, \ + f"{what}: the parent resent signals:\n{text}" + for line in text.splitlines(): + assert not HttpdErrorLog.RE_ERRLOG_CRASH.match(line), \ + f"{what}: crash: {line}" + assert not HttpdErrorLog.RE_ERRLOG_ASAN.match(line), \ + f"{what}: sanitizer: {line}" + + def ungraceful(self, env, how): + """Restart or stop the server, ungracefully, and time it. A restart + is over when the new generation is serving, which the parent logs + once the old children are gone.""" + pos = env.httpd_error_log.current_pos() + t0 = time.monotonic() + if how == 'restart': + assert env.instance.restart(graceful=False, timeout=20), \ + "no restart" + else: + assert env.instance.stop(timeout=20), "the server did not stop" + elapsed = time.monotonic() - t0 + return pos, elapsed + + def after(self, env, how, pos, elapsed, pid): + what = f"ungraceful {how}" + self.check_log(env, pos, what) + assert elapsed < EXIT_BOUND, \ + f"{what}: took {elapsed:.1f}s (bound {EXIT_BOUND}s)" + if how == 'restart': + assert env.read_pid_file() == pid, "the restart replaced the parent" + else: + assert not env.instance.is_running() + assert not port_open(env.http_port), "the port is still open" + assert env.apache_restart() == 0 + self.get(env, f"after an {what}") + + @pytest.mark.parametrize("how", ["restart", "stop"]) + def test_core_008_04_busy(self, env, how): + """Busy connections: a request not yet complete (holding a worker + with either MPM), a response being written (holding a worker with + mpm_worker, mostly in write completion with mpm_event).""" + if env.mpm_module == 'mpm_motorz': + pytest.skip("mpm_motorz doesn't wake a child reading a request " + "on an ungraceful stop or restart") + pid = env.read_pid_file() + with _Sockets(env) as c: + c.partial_request() + c.partial_request() + c.slow_reader() + c.slow_reader() + time.sleep(1) # for the server to be on them + pos, elapsed = self.ungraceful(env, how) + self.after(env, how, pos, elapsed, pid) + + @pytest.mark.parametrize("how", ["restart", "stop"]) + def test_core_008_05_lingering(self, env, how): + """Connections in their lingering close (a worker's with mpm_worker, + mostly the listener's with mpm_event).""" + pid = env.read_pid_file() + with _Sockets(env) as c: + c.lingering() + c.lingering() + time.sleep(1) # for the responses to be sent and read on + pos, elapsed = self.ungraceful(env, how) + self.after(env, how, pos, elapsed, pid) + + def test_core_008_06_one_process(self, env): + """The server exits cleanly from SIGTERM in one-process mode (-X): + the status is 0 and nothing crashes while it cleans up.""" + inst = env.instance + assert env.apache_stop() == 0 + pos = env.httpd_error_log.current_pos() + with tempfile.TemporaryFile() as err: + proc = subprocess.Popen( + inst.argv('-X'), env=inst.env, stdin=subprocess.DEVNULL, + stdout=subprocess.DEVNULL, stderr=err, start_new_session=True) + try: + deadline = time.monotonic() + 30 + ready = False + while time.monotonic() < deadline and proc.poll() is None: + # no "resuming normal operations" in this mode + if inst.all_ports_open(): + ready = True + break + time.sleep(0.1) + err.seek(0) + assert ready, f"-X did not come up: {proc.poll()}: {err.read()!r}" + for _ in range(3): + self.get(env, "in one-process mode") + proc.send_signal(signal.SIGTERM) + try: + rc = proc.wait(timeout=15) + except subprocess.TimeoutExpired: + pytest.fail("-X did not exit on SIGTERM") + finally: + if proc.poll() is None: + signal_group(proc.pid, signal.SIGKILL) + proc.wait() + # whatever it started and outlived it (e.g. mod_cgid's + # daemon) would hold the ports for the next start + inst._reap_group(proc.pid) + err.seek(0) + stderr = err.read().decode(errors='replace') + assert rc == 0, f"-X exited with {rc}: {stderr}" + for bad in ('double free', 'corruption', 'exit signal', + 'Segmentation', 'Abort', 'Sanitizer'): + assert bad not in stderr, f"-X: {bad!r} on stderr: {stderr}" + self.check_log(env, pos, "-X") From 92c001d67e0859f4e7a45f463b06c90042eabd51 Mon Sep 17 00:00:00 2001 From: ylavic Date: Mon, 16 Nov 2026 17:13:12 +0100 Subject: [PATCH 10/10] changes-entries: add the entry for mpm_event's and mpm_worker's exit fixes. One entry for the previous commits' fixes. * changes-entries/event-worker-child-exit.txt: New. --- changes-entries/event-worker-child-exit.txt | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100644 changes-entries/event-worker-child-exit.txt diff --git a/changes-entries/event-worker-child-exit.txt b/changes-entries/event-worker-child-exit.txt new file mode 100644 index 00000000000..851f26f3f73 --- /dev/null +++ b/changes-entries/event-worker-child-exit.txt @@ -0,0 +1,6 @@ + *) mpm_event, mpm_worker: Fix possible crashes of child processes on an + ungraceful stop or restart, or at exit in one-process mode (-X), and a + child never exiting on a graceful stop when its listener thread is stuck + (mpm_event). A child now exits at once on an ungraceful stop even when + its connections are busy, and no longer runs the parent process' exit + handlers (e.g. OpenSSL's). [Yann Ylavic]