Repository navigation
L4 health check reports UP/L4OK for refused connections on macOS (sock_conn_check() trusts a second connect() returning EISCONN) #3513
Description
Title: L4 health check reports UP/L4OK for refused connections on macOS (sock_conn_check() trusts a second connect() returning EISCONN)
Detailed Description of the Problem
On macOS, a plain layer-4 health check (server ... check with no tcp-check / httpchk rules) reports the server as UP with L4OK when the target port actively refuses the connection (TCP RST, ECONNREFUSED). Even server closed_port 127.0.0.1:1 check, where nothing listens on port 1, is shown as UP L4OK 0ms.
The behaviour is identical with the default kqueue poller and with -dk (poll). Reproduced with HAProxy 3.4.5 and 3.2.1 (both Homebrew builds, TARGET=osx) on macOS 27.0 (arm64).
In production this means that a backend whose service is down, but whose host or router still answers with RST, stays UP and keeps receiving traffic. Each request then fails as soon as HAProxy tries to send on the connection, goes through the retries and ends in a 503.
Expected Behavior
The server should be marked DOWN with L4CON ("layer 1-4 connection problem, for example 'Connection refused' (tcp rst)", as documented for check_status in the stats CSV).
Steps to Reproduce the Behavior
- Save this configuration as
repro.cfg:
global
log stdout format raw local0
defaults
mode tcp
timeout connect 5s
timeout client 5s
timeout server 5s
frontend stats
bind 127.0.0.1:18305
mode http
stats enable
stats uri /stats
backend plain_check
# nothing listens on 127.0.0.1:1 -> connect() is refused (RST)
server closed_port 127.0.0.1:1 check
backend tcp_check_send
option tcp-check
tcp-check connect
tcp-check send "\r\n"
server closed_port 127.0.0.1:1 check
backend http_check
option httpchk GET /
server closed_port 127.0.0.1:1 check
- Run
haproxy -f repro.cfg -db(orhaproxy -f repro.cfg -db -dkto use poll instead of kqueue). - After a few seconds run
curl -s 'http://127.0.0.1:18305/stats;csv' | cut -d, -f1,2,18,37,39.
Observed with 3.4.5 (identical with -dk):
# pxname,svname,status,check_status,check_duration
plain_check,closed_port,UP,L4OK,0
tcp_check_send,closed_port,DOWN,SOCKERR,0
http_check,closed_port,DOWN,SOCKERR,0
Log:
[WARNING] (48880) : Server tcp_check_send/closed_port is DOWN, reason: Socket error, info: "Poller reported POLLERR (Broken pipe) at step 2 of tcp-check (send)", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] (48880) : Server http_check/closed_port is DOWN, reason: Socket error, info: "ECONNREFUSED returned by OS (Connection refused)", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
So the kernel does report the refusal, and checks that send something detect it, but the pure L4 connect check declares success.
Do you have any idea what may have caused this?
sock_conn_check() in src/sock.c (3.4.5; the same logic is in 3.2.1 and current master) validates the connection by calling connect() a second time and treats EISCONN as "connected":
/* Use connect() to check the state of the socket. This has the double
* advantage of *not* clearing the error (so that health checks can
* still use getsockopt(SO_ERROR)) and giving us the following info :
* - error
* - connecting (EALREADY, EINPROGRESS)
* - connected (EISCONN, 0)
*/
...
if (connect(fd, (const struct sockaddr *)addr, get_addr_len(addr)) == -1) {
if (errno == EALREADY || errno == EINPROGRESS)
goto wait;
if (errno && errno != EISCONN) {
conn_set_errno(conn, errno);
conn_report_term_evt(conn, tevt_loc_fd, fd_tevt_type_connect_err);
goto out_error;
}
}
done:On Linux the second connect() reports the pending ECONNREFUSED, which is what this code relies on. On macOS (XNU) the second connect() on a socket whose connection attempt was refused returns EISCONN, so the function reaches done and the check passes.
Measured on the same machine with this script (non-blocking connect() to 127.0.0.1:1, wait with the poller, then a second connect() and getsockopt(SO_ERROR)):
import errno, select, socket
def name(e):
return errno.errorcode.get(e, str(e))
def probe(poller):
s = socket.socket()
s.setblocking(False)
try:
s.connect(("127.0.0.1", 1)) # nothing listens on port 1
except OSError as e:
print(f"[{poller}] first connect(): {name(e.errno)}")
if poller == "kqueue":
kq = select.kqueue()
ev = select.kevent(s.fileno(), select.KQ_FILTER_WRITE, select.KQ_EV_ADD | select.KQ_EV_ENABLE)
for r in kq.control([ev], 1, 3.0):
print(f"[{poller}] EVFILT_WRITE flags=0x{r.flags:x} EV_EOF={bool(r.flags & select.KQ_EV_EOF)} "
f"fflags={r.fflags} ({name(r.fflags)})")
kq.close()
else:
p = select.poll()
p.register(s.fileno(), select.POLLOUT)
for _, ev in p.poll(3000):
print(f"[{poller}] revents=0x{ev:x} POLLOUT={bool(ev & select.POLLOUT)} "
f"POLLERR={bool(ev & select.POLLERR)} POLLHUP={bool(ev & select.POLLHUP)}")
try:
s.connect(("127.0.0.1", 1))
print(f"[{poller}] second connect(): 0 (success)")
except OSError as e:
print(f"[{poller}] second connect(): {name(e.errno)}")
print(f"[{poller}] SO_ERROR: {name(s.getsockopt(socket.SOL_SOCKET, socket.SO_ERROR))}")
s.close()
probe("kqueue")
probe("poll")Output on macOS 27.0:
[kqueue] first connect(): EINPROGRESS
[kqueue] EVFILT_WRITE flags=0x8005 EV_EOF=True fflags=61 (ECONNREFUSED)
[kqueue] second connect(): EISCONN
[kqueue] SO_ERROR: ECONNREFUSED
[poll] first connect(): EINPROGRESS
[poll] revents=0x10 POLLOUT=False POLLERR=False POLLHUP=True
[poll] second connect(): EISCONN
[poll] SO_ERROR: ECONNREFUSED
- kqueue path:
EV_EOFonEVFILT_WRITEis turned intoFD_POLL_ERRbyev_kqueue.c, so the function falls through to theconnect()double-check, which returnsEISCONNand leads todone. - poll path: macOS reports
POLLHUPwithoutPOLLERR, so the workaround for EPOLLHUP or EPOLLRDHUP during connection establishment leads to a wrong decision that a backend is down #1863 ("Removing FD_POLL_HUP if no FD_POLL_ERR to let connect() decide") strips the HUP flag, and the sameconnect()double-check returnsEISCONNand leads todone.
In both cases getsockopt(SO_ERROR) still holds ECONNREFUSED at that moment, and the kqueue event also carries the errno in fflags.
Do you have an idea how to solve the issue?
When the poller reported an error or hangup (FD_POLL_ERR / FD_POLL_HUP was set) and the connect() double-check answers EISCONN (or 0), confirm with getsockopt(SO_ERROR) before going to done; if it returns a non-zero error, set errno to it and take the out_error path so that chk_report_conn_err() still sees the right errno. Alternatively the kqueue poller could keep the fflags errno of an EV_EOF write event and expose it to sock_conn_check().
Workaround used meanwhile: option tcp-check + tcp-check connect + tcp-check send "\r\n" in defaults, which makes a refused connection fail at the send step.
What is your configuration?
See the minimal configuration under "Steps to Reproduce".
Output of haproxy -vv
HAProxy version 3.4.5-c28c410 2026/09/24 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2031.
Known bugs: http://www.haproxy.org/bugs/bugs-3.4.5.html
Running on: Darwin 27.0.0 Darwin Kernel Version 27.0.0: Tue Aug 11 21:20:40 PDT 2026; root:xnu-13432.1.9~1/RELEASE_ARM64_T6020 arm64
Build options :
TARGET = osx
CC = cc
CFLAGS = -O2 -g -fwrapv
OPTIONS = USE_OPENSSL=1 USE_ZLIB=1 USE_QUIC=1 USE_PROMEX=1 USE_PCRE2=1 USE_PCRE2_JIT=1
DEBUG =
Feature list : -51DEGREES -ACCEPT4 +ACME -BACKTRACE -CLOSEFROM +CPU_AFFINITY -CRYPT_H -DEVICEATLAS -DL -ECH -ENGINE -EPOLL -EVPORTS +GETADDRINFO +KQUEUE -KTLS -LIBATOMIC +LIBCRYPT -LINUX_CAP -LINUX_SPLICE -LINUX_TPROXY -LUA -MATH -MEMORY_PROFILING -NETFILTER -NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL -PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION +QUIC -QUIC_OPENSSL_COMPAT -RT -SHM_OPEN -SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 -TFO +THREAD -THREAD_DUMP +TPROXY +TRACE -WURFL +ZLIB
Detected feature list :
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_TGROUPS=32, MAX_THREADS=1024, default=12).
Built with SSL library version : OpenSSL 4.0.2 25 Aug 2026
Running on SSL library version : OpenSSL 4.0.2 25 Aug 2026
SSL library supports TLS extensions : yes
SSL library supports SNI : yes
SSL library default verify directory : /opt/homebrew/etc/openssl@4/certs
SSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
OpenSSL providers loaded : default
QUIC: connection sock-per-conn mode support : no
QUIC: GSO emission support : no
Built with the Prometheus exporter as a service
Built with zlib version : 1.2.12
Running on zlib version : 1.2.12
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using:
Built with PCRE2 version : 10.48 2026-08-31
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with clang compiler version 21.0.0 (clang-2100.3.34.2)
Available polling systems :
kqueue : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use kqueue.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
quic : mode=HTTP side=FE|BE mux=QUIC flags=HTX|NO_UPG|FRAMED
qmux : mode=HTTP side=FE|BE mux=QMUX flags=HTX|NO_UPG
h2 : mode=HTTP side=FE|BE mux=H2 flags=HTX|HOL_RISK|NO_UPG
h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO_UPG
<default> : mode=HTTP side=FE|BE mux=H1 flags=HTX
fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG
spop : mode=SPOP side=BE mux=SPOP flags=HOL_RISK|NO_UPG
<default> : mode=SPOP side=BE mux=SPOP flags=HOL_RISK|NO_UPG
none : mode=TCP side=FE|BE mux=PASS flags=NO_UPG
<default> : mode=TCP side=FE|BE mux=PASS flags=
Available services : prometheus-exporter
Available filters :
[BWLIM] bwlim-in
[BWLIM] bwlim-out
[CACHE] cache
[COMP] comp-req
[COMP] comp-res
[COMP] compression
[FCGI] fcgi-app
[SPOE] spoe
[TRACE] trace
Last Outputs and Backtraces
$ haproxy -f repro.cfg -db
[WARNING] (48880) : Server tcp_check_send/closed_port is DOWN, reason: Socket error, info: "Poller reported POLLERR (Broken pipe) at step 2 of tcp-check (send)", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] (48880) : Server http_check/closed_port is DOWN, reason: Socket error, info: "ECONNREFUSED returned by OS (Connection refused)", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
$ curl -s 'http://127.0.0.1:18305/stats;csv' | cut -d, -f1,2,18,37,39
# pxname,svname,status,check_status,check_duration
plain_check,closed_port,UP,L4OK,0
tcp_check_send,closed_port,DOWN,SOCKERR,0
http_check,closed_port,DOWN,SOCKERR,0
No crash, no backtrace.
Additional Information
- macOS 27.0 (build 26A428), Darwin 27.0.0 (xnu-13432.1.9~1), arm64 (Apple silicon). HAProxy installed with Homebrew.
- Also reproduced with HAProxy 3.2.1-f4d1a4e from Homebrew (also
TARGET=osx, built without QUIC/PROMEX), so this is not a recent regression. - Regular traffic towards a refused server still ends with a 503 (
SC--) after the retries, because the error is caught on the first send. The visible impact is the health check status. - This report was drafted with Claude Code (Anthropic Claude Fable 5.1). The reproduction, the measurements and the source reading were done on the reporter's machine, and the text was reviewed by the reporter before filing.
ai-assisted claude-code macos health-check kqueue
Activity
wtarreau commentedon Sep 28, 2026
Very interesting analysis, thanks for the details. Based on what you observed, I guess we could inded use POLL_ERR/POLL_HUP to decide to check SO_ERROR. That might be sufficient. I don't have a MacOS at hand for testing and I want the changes to remain minimal in order not to break other platforms!
cognet commentedon Sep 28, 2026
I can't reproduce that on my mac, running macOS 26, even your script reports ECONNREFUSED as expected, which version are you running?
cognet commentedon Sep 28, 2026
https://ml.ruby-lang.org/archives/list/ruby-core@ml.ruby-lang.org/thread/URCYEKYRXHE6LHV4H3YGTR3UKXOP5EVN/
Seems it is a change of behavior in recent macOS.
wtarreau commentedon Sep 28, 2026
Oh... I love OS bugs as much as compiler bugs, we usually have to maintain workarounds for decades :-( Sadly it was released two weeks ago so it might be too late to hope for a fix now.