{{ message }}
Repository navigation
>=3.2-dev8: Hanging stats socket with cpu-policy group-by-cluster #2951
Description
Detailed Description of the Problem
Regarding https://github.com/orgs/haproxy/discussions/2901#discussioncomment-12934325
Our config and systems were unmodified and are running very stable for quite some time with 3.1.6 (except one rare but already fixed issue in 3.1.7) and 3.2-dev (currently -dev11, started with dev0) and never had this problem.
When adding cpu-policy group-by-cluster to the global section, HAProxy becomes unresponsive / blocking after some time but it seems only for the stats socket. It seems to still serve traffic. Reload doesn't help, only restart.
It is unclear yet what's triggering this issue. It's not immediately and it takes some minutes (usually <10m). A reload isn't required to trigger it it seems.
Expected Behavior
Working stats socket
Steps to Reproduce the Behavior
- 3.2-dev11
- cpu-policy group-by-cluster in the global section
- some magic
- echo help | socat /path/to/socket stdio
- Get no response and/or blocking connections to the socket (not socat tho) and high CPU usage
Do you have any idea what may have caused this?
Somehow related to cpu-policy group-by-cluster
Do you have an idea how to solve the issue?
No response
What is your configuration?
N/AOutput of haproxy -vv
HAProxy version 3.2-dev11-acd372d 2025/04/18 - https://haproxy.org/
Status: development branch - not safe for use in production.
Known bugs: https://github.com/haproxy/haproxy/issues?q=is:issue+is:open
Running on: Linux 6.1.0-33-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.133-1 (2025-04-10) x86_64
Build options :
TARGET = linux-glibc
CC = cc
CFLAGS = -O2 -g -march=x86-64-v3 -mtune=znver1 -fwrapv
OPTIONS = USE_LIBCRYPT=1 USE_OPENSSL_AWSLC=1 USE_LUA=1 USE_ZLIB= USE_SLZ=1 USE_NS= USE_PROMEX=1 USE_PCRE= USE_PCRE_JIT= USE_PCRE2=1 USE_PCRE2_JIT=
DEBUG = -DDEBUG_GLITCHES=1
Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER -NS -OBSOLETE_LINKER +OPENSSL +OPENSSL_AWSLC -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 -PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION -QUIC -QUIC_OPENSSL_COMPAT +RT +SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=4).
Built with SSL library version : OpenSSL 1.1.1 (compatible; AWS-LC 1.49.1)
Running on SSL library version : AWS-LC 1.49.1
SSL library supports TLS extensions : yes
SSL library supports SNI : yes
SSL library FIPS mode : no
SSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.6
Built with the Prometheus exporter as a service
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.42 2022-12-11
PCRE2 library supports JIT : no (USE_PCRE2_JIT not set)
Encrypted password support via crypt(3): yes
Built with gcc compiler version 12.2.0
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
h2 : mode=HTTP side=FE|BE mux=H2 flags=HTX|HOL_RISK|NO_UPG
<default> : mode=HTTP side=FE|BE mux=H1 flags=HTX
h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO_UPG
fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG
<default> : mode=SPOP side=BE mux=SPOP flags=HOL_RISK|NO_UPG
spop : mode=SPOP side=BE mux=SPOP flags=HOL_RISK|NO_UPG
<default> : mode=TCP side=FE|BE mux=PASS flags=
none : mode=TCP side=FE|BE mux=PASS flags=NO_UPG
Available services : prometheus-exporter
Available filters :
[BWLIM] bwlim-in
[BWLIM] bwlim-out
[CACHE] cache
[COMP] compression
[FCGI] fcgi-app
[SPOE] spoe
[TRACE] trace
Last Outputs and Backtraces
Additional Information
No response
Activity
idl0r commentedon Apr 25, 2025
@wtarreau I'm trying to reproduce it with -dev8 right now. I'll also try to get you some gdb output / coredump or something.
If dev8 is also affected, I can also try to bisect it further but it might be ending up in the commit that adds
cpu-policy group-by-cluster. So not sure if that will be that helpful then.idl0r commentedon Apr 25, 2025
It's doing a show stat and a show info for parsing some data. We have like 15 of those checks and they're issued around every 5m. The script then parses the output and is looking for a specific proxy
idl0r commentedon Apr 25, 2025
Can be triggered and reproduced with -dev8!. Hope that helps already somewhat.
wtarreau commentedon Apr 25, 2025
OK thanks for the info. So if you say it usually takes less than 10mn, we could imagine that the python script passes at most once or twice before it happens, so maybe it manages to trigger the issue by dumping such stats and info.
Before you spend a lot of time bisecting, I'd suggest to instead try to manually configure the groups like the cpu-policy does it:
Logically it should reproduce the issue. My fear is that we could have a deeper bug somewhere in file descriptor management or any such thing when it comes to thread groups, and that could be more easily triggered by your setup than others.
idl0r commentedon Apr 25, 2025
Can do!
I wanted to do that on that semi productive node so the mappings above will not work. This one seems totally different:
wtarreau commentedon Apr 25, 2025
Could you also please run a "perf top" so that we have an idea where it's looping ? In case it would be related to a task looping over itself, maybe I'll also ask you to enable task profiling:
wtarreau commentedon Apr 25, 2025
Wow, cool, you manage to trigger it with only 4 single-core clusters ? Nice, this should help the local reproduction. I'm currently trying on a 64-core EPYC like you were ;-)
wtarreau commentedon Apr 25, 2025
Also does your script connect directly to the worker's socket or is it passing the request through the master's socket ?
idl0r commentedon Apr 25, 2025
idl0r commentedon Apr 25, 2025
Worker socket. The configured one, not /run/haproxy-master.sock
idl0r commentedon Apr 25, 2025
Still on -dev8:
wtarreau commentedon Apr 25, 2025
OK that allows us to narrow it quite a bit more. I really suspect that an event is being ignored and I don't understand why. And seeing fd_grab_tgid() appear here doesn't amuse me. It's one of the super-sensitive parts of the thread-group code. I'm suspecting that the sharing of outgoing connections between thread groups could have woken up a subtle race. Could you please try with
tune.idle-pool.shared offin the global section ? It will stop sharing idle conns between servers and will make sure FDs don't switch from one thread to the other. It's indeed possible that a precise timing happens between the FD used (or released?) by the CLI and one used by an idle conn that changed thread.idl0r commentedon Apr 25, 2025
Had it running till it occurs and then:
28 remaining items
rnsanchez commentedon Apr 25, 2025
Hi Willy,
Any chance this patch might help #2937? We have a few instances (maybe most!) that use shards.
wtarreau commentedon Apr 25, 2025
Hi Ricardo!
Maybe I misunderstood it but my understanding was that it was wasting CPU in the stick-table index update. Olivier is currently working on this to offload the lock to a task and the gains he observed are already very good (but we suspect we'll then discover the net bottleneck); hopefully early next week we'll merge his patches. However if you're experiencing cases where the CLI freezes, that's different, but I have not understood that in the description.
rnsanchez commentedon Apr 25, 2025
My question is mostly because during the degradation (loop), we also lose observability:
#2937 (comment)
Given our rate of CLI interactions (stats monitoring which does not keep reconnecting, and our socket-watchdog that reconnects every 500ms), there is the possibility that we could get both issues to trigger, together. Or not. :-)
wtarreau commentedon Apr 25, 2025
I got it but I didn't understand if that was caused by the long time to get the stick-table lock or by anything else. In the traces at least the lock was keeping something like half of the threads busy, which can already be a concern. But you could be right, maybe you were hit by the same bug! Socat has a default timeout of 0.5s so if haproxy was stuck in a CPU loop for that long, socat could have timed out. Using -t you can increase it (socat -t 5 ...), and it will confirm whether it was a lock or the CLI's listener. The lock cannot hold much more than one second without triggering the watchdog, so that would help sort the issue.
In any case, the bug above is horrible, and if you're running with thread groups, I strongly encourage you to pick the patch (or just to update to latest master, because aside this bug it's getting really clean).
idl0r commentedon Apr 28, 2025
Oh, neat! Thanks! Will try that patch today.
idl0r commentedon Apr 28, 2025
Looks very good at the first glance, @wtarreau :)
idl0r commentedon Apr 28, 2025
Running fine for some hours already. So yeah, I think that's it.
BUG/MAJOR: listeners: transfer connection accounting when switching l…
BUG/MAJOR: listeners: transfer connection accounting when switching l…
BUG/MAJOR: listeners: transfer connection accounting when switching l…