Repository navigation
Conversation
…e lsphp Every httpd worker process keeps its own pool of connections to an external application, sized by maxConns to the whole application. A self-managed lsphp is shared by all of them, and each open LSAPI connection holds one of its PHP_LSAPI_CHILDREN children. recycleConn() and processPending() parked connections with no queued request, and idle LSAPI connections were only closed after pcKeepAliveTimeout, INT_MAX by default. After a burst, or under steady concurrency above the number of children, connections parked in processes that did not need them could hold every child, while requests in the other processes waited in lsphp's listen queue until LSAPI's idle timeout (300 s) freed one. With LSAPI_AVOID_FORK, as in the default config, LSAPI allows no extra children and reclaims no idle ones, so that is a stall of several minutes with 5xx after 180 s. Return connections with no request to the pool through reuseConn(), which closes them when the worker does not keep idle connections; the pool keeps the object and the next request reconnects it. LsapiWorker::keepIdleConn() returns 0 for a self-managed application running with LSAPI_AVOID_FORK when httpdWorkers > 1. Its children persist and accept from the shared listener themselves, so a finished child goes straight back to it and reconnecting costs one connect(). Without LSAPI_AVOID_FORK, LSAPI forks a child for each new connection and reclaims idle children itself, so connections are kept there, as they are for a single httpd worker and every other handler type.
|
Reproduced #523 on OpenLiteSpeed 1.9.3 (vendor binary built from e11d64a) with Remi PHP 8.5.11 on AlmaLinux 10.2, 12 cores, 32 GB RAM. WordPress runs in one detached PHP process group: With Adding only The median rises because no request is left waiting until the load ends. Sequential requests to the uncached homepage show no difference beyond run-to-run variation. A trivial PHP script went from a 0.11 ms median to 0.13–0.16 ms across three runs. On the PR itself: Could |
With more than one httpd worker process, idle connections parked in the processes' pools can hold every
PHP_LSAPI_CHILDRENchild of a self-managed lsphp. Requests in the other processes then wait in lsphp'slisten queue until LSAPI's idle timeout frees a child, 300 s by default. With the stock
extProcessor lsphp(LSAPI_AVOID_FORK) that is a full stall:Details, evidence and a reproduction are in #523.
Cause
maxConnsto the wholeapplication. All of them share one lsphp and its
PHP_LSAPI_CHILDRENchildren.ExtWorker::recycleConn()andprocessPending()park aconnection with no queued request, and idle LSAPI connections are only closed after
pcKeepAliveTimeout, which defaults toINT_MAX.So connections held by processes that don't need them can pin every child, while the processes that do
need one queue new connections that no child will accept. With
LSAPI_AVOID_FORK, LSAPI allows no extrachildren and reclaims no idle ones, so nothing breaks the deadlock until the idle timeout.
Change
ExtWorker::reuseConn()is now the one place that returns a connection with no request to thepool. If the worker doesn't keep idle connections, it closes the connection first. The pool keeps the
object, and the next request connects it again through the existing reconnect path in
assignReq().ExtWorker::keepIdleConn()is a new virtual, 1 by default.LsapiWorker::keepIdleConn()returns 0 only for a self-managed application running with
LSAPI_AVOID_FORKwhenhttpdWorkers> 1.child goes straight back to it, and reconnecting costs one
connect().LSAPI_AVOID_FORK, LSAPI's parent forks a child for each new connection, so closing wouldmean a fork per request. There LSAPI reclaims idle children itself, and connections are kept as
before.
Unchanged:
LSAPI_AVOID_FORK;Alternatives I measured and rejected:
pcKeepAliveTimeout 1) fixes a single burst, but not steady load. Theprocesses still hold more connections than there are children: 49 req/s at 40 clients on the stock
config.
maxConnscaps a process at 1–2 connections with the stock 10 childrenand 8 workers. Bursts often arrive in one process, over one CDN connection.
LSAPI_AVOID_FORKitmade LSAPI fork a child per request: about 1,100 req/s for one client instead of 6,600.
Testing
Built from master with
./build.sh -o OFF -l OFF -p OFFon Ubuntu 24.04. For each run, the binary of afresh stock 1.9.2 install from repo.litespeed.sh was replaced with the build. Everything else was stock:
extProcessor lsphpuntouched, 8 httpd worker processes (a 12-CPU host), lsphp83 8.3.33. "master" belowis the same build without this commit. The scripts are in #523.
A burst of 40 requests to a PHP page taking 0.7 s:
3 s is the floor: 40 × 0.7 s ÷ 10 children.
httpdWorkers 1gives the same on master.Five bursts of 40, 20 s apart, over steady traffic of about 5 req/s, for 2 minutes:
Throughput of a trivial PHP page (
<?php echo "ok";),ab -n 10000, in req/s:LSAPI_AVOID_FORK)abtimeout)LSAPI_AVOID_FORKremovedThe cost is the reconnect: about 22 µs per request for a single client on a page that does nothing,
5,757 req/s against 6,611. Real PHP pages take milliseconds, and with concurrency the change is a large
gain rather than a cost.
No change where it shouldn't engage:
httpdWorkers 1: the fix never engaged. Idle connections were kept (10 still open after theburst), and the burst took ≤ 2.8 s.
LSAPI_AVOID_FORKremoved: the fix never engaged, and all 40 were served in ≤ 8.8 s.With the patch,
error.logshowed no connection errors in any run.Unit tests:
ols_unittest, in a Debug build with AddressSanitizer as the repo's CMake sets it up,passes 177 of 177, the same as master. AddressSanitizer reported nothing. None of those tests exercise
ExtWorker,LsapiWorkeror the connection pool, which is why the tests above ran against the realserver.
Not addressed here
Without
LSAPI_AVOID_FORKthe pinning still happens, but LSAPI's own reclaim keeps it from stallingoutright. Under sustained concurrency it still costs a lot: 17 to 1,333 req/s at 40 clients across
runs of the trivial page, against about 6,600 for one client. Fixing that cheaply needs LSAPI's help,
for example a child idle on a connection giving it up when others are waiting in the listen queue. That
belongs in
sapi/litespeed, and is described in #523.Fixes #523