{{ message }}
Repository navigation
HAP 2.7.1 crashed under load #2011
Description
Detailed Description of the Problem
We started to upgrade our current hap to 2.7.1 and on machines with low to moderate traffic no problems were detected. I did a roll out on a system which can have more load and after the upgrade it crashed. I restarted and it was fine for couple of days and then under higher load it crashed again. I have core dump from the crash.
Expected Behavior
(≖_≖ )
Steps to Reproduce the Behavior
I don't have a way to reproduce it.
Do you have any idea what may have caused this?
No response
Do you have an idea how to solve the issue?
No response
What is your configuration?
global
nbthread 36
setenv TUNED_WEIGHT 98
setenv val1_WEIGHT 50
setenv val2_WEIGHT 20
setenv val4_WEIGHT 30
setenv val3_WEIGHT 10
listen dummy
timeout connect 10ms
timeout client 10ms
timeout server 10ms
bind 0.0.0.0:0
peers machines
peer x x.x.x:x
peer x x.x.x:x
peer x x.x.x:x
peer x x.x.x:x
table peer-static type ip size 1m expire xs store gpc0,gpt0,gpc0_rate(xs),http_req_rate(xs),http_err_rate(xs),http_err_cnt,conn_cur,conn_rate(xs),sess_cnt,sess_rate(xs)
table peer-netrep type ip size 1m expire xs store gpc0,gpt0,gpc0_rate(xs)
log 0.0.0.0 local1 notice alert
log 0.0.0.0 local2 info alert
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 250000
user haproxy
group haproxy
daemon
set-dumpable
maxsslconn 500000
tune.ssl.cachesize 5000000
tune.ssl.default-dh-param 2048
ssl-default-bind-options no-tls-tickets ssl-min-ver TLSv1.2
ssl-default-bind-ciphers x
stats socket /var/lib/haproxy/stats mode 0660 user haproxy group haproxy level admin
tune.maxrewrite x
tune.h2.max-concurrent-streams x
tune.http.logurilen x
ca-base /etc/pki/tls/certs/
defaults
mode http
log global
no option dontlognull
option http-server-close
option redispatch x
option allbackups
retries x
timeout http-request xs
timeout queue xs
timeout connect x
timeout client xs
timeout server xs
timeout client-fin xs
timeout server-fin xs
timeout http-keep-alive xs
timeout check xs
maxconn x
option abortonclose
option tcp-smart-connect
hash-balance-factor x
hash-type consistent
errorfile 408 408_file.http
errorfile 503 503_file.http
default-server ca-file file.crt check check-ssl ssl verify required maxqueue 1
frontend me-https
bind 0.0.0.0:0
bind 0.0.0.0:0 ssl alpn http/1.1 crt file.pem
bind 0.0.0.0:0 ssl alpn http/1.1 crt file.pem accept-proxy
mode http
maxconn x
rate-limit sessions x
log-format "%{+Q}o %{-Q}ci - - [%trg] %r %ST %B \"\" \"\" %cp %ms %ft %b %s %TR %Tw %Tc %Tr %Ta %tsc %ac %fc %bc %sc %rc %sq %bq %CC %CS %hrl %hsl %sslv %sslc %[ssl_fc_session_id,base64] %[ssl_fc_sni] %[var(txn.custom)]"
tcp-request inspect-delay xs
#### ACL AND HEADER SETTINGS
http-request redirect code 301 location https://x if path1
http-request redirect code 301 location https://x if path2
http-request redirect code 301 location https://x
frontend redirect-www-http
bind 0.0.0.0:0
bind 0.0.0.0:0
mode http
log-format "%{+Q}o %{-Q}ci - - [%trg] %r %ST %B \"\" \"\" %cp %ms %ft %b %s %TR %Tw %Tc %Tr %Ta %tsc %ac %fc %bc %sc %rc %sq %bq %CC %CS"
redirect location https://x code 301 if { hdr(host) -i x }
redirect location https://x code 301 if { hdr_end(host) -i x }
redirect location https://x code 301 if { hdr_end(host) -i x }
redirect location https://x code 301 if { hdr_end(host) -i x }
redirect location https://x code 301 if { hdr_end(host) -i x }
redirect location https://x code 301 if { hdr_end(host) -i x }
redirect location https://x code 301 if { hdr_end(host) -i x }
redirect location https://x code 301
frontend com-http
bind 0.0.0.0:0
mode http
log-format "%{+Q}o %{-Q}ci - - [%trg] %r %ST %B \"\" \"\" %cp %ms %ft %b %s %TR %Tw %Tc %Tr %Ta %tsc %ac %fc %bc %sc %rc %sq %bq %CC %CS %[var(txn.custom)]"
## time to get full path
tcp-request inspect-delay xs
## ACL HEADERS etc ###
redirect prefix https://x code 301
frontend com-https
bind 0.0.0.0:0 ssl alpn http/1.1 crt file.pem
bind 0.0.0.0:0 ssl alpn http/1.1 crt file.pem accept-proxy
mode http
timeout http-keep-alive xs
timeout http-request xs
tcp-request inspect-delay xs
log-format "%{+Q}o %{-Q}ci - - [%trg] %r %ST %B \"\" \"\" %cp %ms %ft %b %s %TR %Tw %Tc %Tr %Ta %tsc %ac %fc %bc %sc %rc %sq %bq %CC %CS %hrl %hsl %sslv %sslc %[ssl_fc_session_id,base64] %[ssl_fc_sni] %[var(txn.custom)]"
### ACLs HEADERS etc ##
default_backend com-https-servers
backend com-https-servers
mode http
balance source
option httpchk
http-check connect port x ssl sni x
http-check send meth HEAD uri /x ver HTTP/1.1 hdr Host x
http-check expect status 200-399
http-check comment comment
http-check disable-on-404
default-server sni x check-sni x maxconn 100
server server1 x:443 weight "${val1_WEIGHT}" backup
server server2 x:443 weight "${val1_WEIGHT}" backup
server server3 x:443 weight "${val1_WEIGHT}" backup
server server4 x:443 weight "${val1_WEIGHT}" backup
server server5 x:443 weight "${val1_WEIGHT}" backup
server server6 x:443 weight "${val1_WEIGHT}" backup
server server7 x:443 weight "${val2_WEIGHT}" backup
server server8 x:443 weight "${val1_WEIGHT}" backup
server server9 x:443 weight "${val2_WEIGHT}" backup
server server10 x:443 weight "${val2_WEIGHT}" backup
server server11 x:443 weight "${val2_WEIGHT}" backup
server server12 x:443 weight "${val3_WEIGHT}" backup
server server13 x:443 weight "${val3_WEIGHT}" backup
server server14 x:443 weight "${val4_WEIGHT}" backup
server server15 x:443 weight "${val4_WEIGHT}" backup
server server16 x:443 weight "${val4_WEIGHT}" backup
server server17 x:443 weight "${val4_WEIGHT}" backup
server server18 x:443 weight "${val4_WEIGHT}" backup
server server19 x:443 weight "${val3_WEIGHT}" backup
server server20 x:443 weight "${val3_WEIGHT}" backup
server server21 x:443 weight "${val2_WEIGHT}" backup
server server22 x:443 weight "${val2_WEIGHT}" backup
server server23 x:443 weight "${val3_WEIGHT}" backup
server s-01 x:443 weight "${val1_WEIGHT}" maxconn 1000
server s-02 x:443 weight "${val4_WEIGHT}" maxconn 1000
server s-03 x:443 weight "${val1_WEIGHT}" maxconn 1000
server s-04 x:443 weight "${val4_WEIGHT}" maxconn 1000Output of haproxy -vv
haproxy -vv
HAProxy version 2.7.1-3e4af0e 2022/12/19 - https://haproxy.org/
Status: stable branch - will stop receiving fixes around Q1 2024.
Known bugs: http://www.haproxy.org/bugs/bugs-2.7.1.html
Running on: Linux 4.18.0-372.26.1.el8_6.x86_64 #1 SMP Tue Sep 13 18:09:48 UTC 2022 x86_64
Build options :
TARGET = linux-glibc
CPU = generic
CC = cc
CFLAGS = -O2 -g -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment
OPTIONS = USE_PCRE=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_TFO=1 USE_SYSTEMD=1 USE_OT=1 USE_PROMEX=1
DEBUG = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS
Feature list : +EPOLL -KQUEUE +NETFILTER +PCRE -PCRE_JIT -PCRE2 -PCRE2_JIT +POLL +THREAD -PTHREAD_EMULATION +BACKTRACE -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -ENGINE +GETADDRINFO +OPENSSL -OPENSSL_WOLFSSL +LUA +ACCEPT4 -CLOSEFROM +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL +SYSTEMD -OBSOLETE_LINKER +PRCTL -PROCCTL +THREAD_DUMP -EVPORTS +OT -QUIC +PROMEX -MEMORY_PROFILING +SHM_OPEN
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=2).
Built with OpenSSL version : OpenSSL 1.1.1s 1 Nov 2022
Running on OpenSSL version : OpenSSL 1.1.1s 1 Nov 2022
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.4.4
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with OpenTracing support.
Support for malloc_trim() is enabled.
Built with zlib version : 1.2.7
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE version : 8.32 2012-11-30
Running on PCRE version : 8.42 2018-03-20
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes
Built with gcc compiler version 4.8.5 20150623 (Red Hat 4.8.5-44)
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
h2 : mode=HTTP side=FE|BE mux=H2 flags=HTX|HOL_RISK|NO_UPG
fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG
<default> : mode=HTTP side=FE|BE mux=H1 flags=HTX
h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO_UPG
<default> : mode=TCP side=FE|BE mux=PASS flags=
none : mode=TCP side=FE|BE mux=PASS flags=NO_UPG
Available services : prometheus-exporter
Available filters :
[BWLIM] bwlim-in
[BWLIM] bwlim-out
[CACHE] cache
[COMP] compression
[FCGI] fcgi-app
[ OT] opentracing
[SPOE] spoe
[TRACE] trace
Last Outputs and Backtraces
Hap LOG:
Thread 4 is about to kill the process.
Thread 1
1/1 stuck=0 prof=0 harmless=0 wantrdv=0
cpu_ns
curr_task=0
Thread 2
1/2 stuck=0 prof=0 harmless=0 wantrdv=0
cpu_ns
curr_task=0
Thread 3
1/3 stuck=0 prof=0 harmless=0 wantrdv=0
cpu_ns
curr_task=0
* Thread 4
1/4 stuck=0 prof=0 harmless=0 wantrdv=0
cpu_ns
curr_task=0
Thread 5
1/5 stuck=0 prof=0 harmless=0 wantrdv=0
cpu_ns
curr_task=0
Thread 6
1/6 stuck=0 prof=0 harmless=0 wantrdv=0
cpu_ns
curr_task=0
Thread 7
1/7 stuck=0 prof=0 harmless=0 wantrdv=0
cpu_ns
curr_task=0
Thread 8
1/8 stuck=0 prof=0 harmless=0 wantrdv=0
cpu_ns
curr_task=0
Thread 9
1/9 stuck=0 prof=0 harmless=0 wantrdv=0
cpu_ns
curr_task=0
Thread 10
1/10 stuck=0 prof=0 harmless=0 wantrdv=0
cpu_ns
curr_task=0
Thread 11
1/11 stuck=0 prof=0 harmless=0 wantrdv=0
cpu_ns
curr_task=0
Thread 12
1/12 stuck=0 prof=0 harmless=0 wantrdv=0
cpu_ns
curr_task=0
Thread 13
1/13 stuck=0 prof=0 harmless=0 wantrdv=0
cpu_ns
curr_task=0
Thread 14
1/14 stuck=0 prof=0 harmless=0 wantrdv=0
cpu_ns
curr_task=0
Thread 15
1/15 stuck=0 prof=0 harmless=0 wantrdv=0
cpu_ns
curr_task=0
Thread 16
1/16 stuck=0 prof=0 harmless=0 wantrdv=0
cpu_ns
curr_task=0
Thread 17
1/17 stuck=0 prof=0 harmless=0 wantrdv=0
cpu_ns
curr_task=0
Thread 18
1/18 stuck=0 prof=0 harmless=0 wantrdv=0
cpu_ns
curr_task=0
Thread 19
1/19 stuck=0 prof=0 harmless=0 wantrdv=0
cpu_ns
curr_task=0
Thread 20
1/20 stuck=0 prof=0 harmless=0 wantrdv=0
cpu_ns
curr_task=0
Thread 21
1/21 stuck=0 prof=0 harmless=0 wantrdv=0
cpu_ns
curr_task=0
Thread 22
1/22 stuck=0 prof=0 harmless=0 wantrdv=0
cpu_ns
curr_task=0
Thread 23
1/23 stuck=0 prof=0 harmless=0 wantrdv=0
cpu_ns
curr_task=0x7f6034657c60 (task) calls=2 last=0
fct=0x55698818ffe0(process_stream) ctx=(nil)
Thread 24
1/24 stuck=0 prof=0 harmless=0 wantrdv=0
cpu_ns
curr_task=0
Thread 25
1/25 stuck=0 prof=0 harmless=0 wantrdv=0
cpu_ns
curr_task=0
Thread 26
1/26 stuck=0 prof=0 harmless=0 wantrdv=0
cpu_ns
curr_task=0
Thread 27
1/27 stuck=0 prof=0 harmless=0 wantrdv=0
cpu_ns
curr_task=0
Thread 28
1/28 stuck=0 prof=0 harmless=0 wantrdv=0
cpu_ns
curr_task=0
Thread 29
1/29 stuck=0 prof=0 harmless=0 wantrdv=0
cpu_ns
curr_task=0
Thread 30
1/30 stuck=0 prof=0 harmless=0 wantrdv=0
cpu_ns
curr_task=0
Thread 31
1/31 stuck=0 prof=0 harmless=0 wantrdv=0
cpu_ns
curr_task=0
Thread 32
1/32 stuck=0 prof=0 harmless=0 wantrdv=0
cpu_ns
curr_task=0
Thread 33
1/33 stuck=0 prof=0 harmless=0 wantrdv=0
cpu_ns
curr_task=0
Thread 34
1/34 stuck=0 prof=0 harmless=0 wantrdv=0
cpu_ns
curr_task=0
Thread 35
1/35 stuck=0 prof=0 harmless=0 wantrdv=0
cpu_ns
curr_task=0
Thread 36
1/36 stuck=0 prof=0 harmless=0 wantrdv=0
cpu_ns
curr_task=0
haproxy version is 2.7.1-3e4af0e
path to executable is
Current worker (512411) exited with code 134 (Aborted)
exit-on-failure
All workers exited. Exiting... (134)
(gdb) t a a bt full
Thread 36 (Thread 0x7f60497f2700 (LWP 512446)):
#0 pl_wait_unlock_long (lock=lock@entry=0x556989325fa0, mask=mask@entry=18446744056529682432) at include/import/plock.h:93
loops = 62664
ret = <optimized out>
m = 135842
#1 0x000055698825d6cd in listener_accept (l=0x556989325f90) at src/listener.c:963
__lk_r = 0x556989325fa0
__set_r = 4
__msk_r = 18446744056529682432
count = 9
status = 5
mask = <optimized out>
cli_conn = 0x0
p = 0x556989324470
max_accept = 4
next_conn = 10
next_feconn = <optimized out>
next_actconn = <optimized out>
expire = <optimized out>
ret = <optimized out>
__func__ = "listener_accept"
#2 0x00005569882a7995 in fd_update_events (fd=58, evts=<optimized out>) at src/fd.c:628
locked = <optimized out>
old = 1179939
new = <optimized out>
new_flags = <optimized out>
must_stop = 0
rmask = 12954663424
tmask = <optimized out>
#3 0x00005569880e8942 in _do_poll (p=<optimized out>, exp=-991615768, wake=0) at src/ev_epoll.c:268
n = <optimized out>
e = <optimized out>
status = <optimized out>
fd = <optimized out>
count = <optimized out>
updt_idx = <optimized out>
wait_time = 24
old_fd = <optimized out>
#4 0x0000556988242043 in run_poll_loop () at src/haproxy.c:2983
next = <optimized out>
wake = <optimized out>
__func__ = "run_poll_loop"
#5 0x00005569882426dc in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3101
ptaf = <optimized out>
ptif = 0x5569888806f0 <per_thread_init_list>
ptdf = <optimized out>
ptff = <optimized out>
init_left = 0
init_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}
init_cond = {__data = {__lock = 333, __futex = 0, __total_seq = 279, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x6c, __nwaiters = 0, __broadcast_seq = 0}, __size = "M\001\000\000\000\000\000\000\027\001", '\000' <repeats 22 times>, "l", '\000' <repeats 14 times>, __align = 333}
#6 0x00007f60b8c081cf in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#7 0x00007f60b7439d83 in iopl () from /lib64/libc.so.6
No symbol table info available.
#8 0x0000000000000000 in ?? ()
No symbol table info available.
Thread 35 (Thread 0x7f6049ff3700 (LWP 512445)):
#0 pl_wait_unlock_long (lock=lock@entry=0x556989325fa0, mask=mask@entry=18446744069414584320) at include/import/plock.h:93
loops = 40987
ret = <optimized out>
m = 203765
#1 0x000055698825b525 in limit_listener (l=l@entry=0x556989325f90, list=list@entry=0x556989325bf8) at src/listener.c:582
__lk_r = 0x556989325fa0
__pl_r = <optimized out>
#2 0x000055698825d6f0 in listener_accept (l=0x556989325f90) at src/listener.c:1220
cli_conn = <optimized out>
p = 0x556989324470
max_accept = 31
next_conn = 31
next_feconn = 0
next_actconn = 0
expire = 0
ret = <optimized out>
__func__ = "listener_accept"
#3 0x00005569882a7995 in fd_update_events (fd=58, evts=<optimized out>) at src/fd.c:628
locked = <optimized out>
old = 1179939
new = <optimized out>
new_flags = <optimized out>
must_stop = 0
rmask = 51518504831
tmask = <optimized out>
--Type <RET> for more, q to quit, c to continue without paging--c
#4 0x00005569880e8942 in _do_poll (p=<optimized out>, exp=-991615768, wake=0) at src/ev_epoll.c:268
n = <optimized out>
e = <optimized out>
status = <optimized out>
fd = <optimized out>
count = <optimized out>
updt_idx = <optimized out>
wait_time = 24
old_fd = <optimized out>
#5 0x0000556988242043 in run_poll_loop () at src/haproxy.c:2983
next = <optimized out>
wake = <optimized out>
__func__ = "run_poll_loop"
#6 0x00005569882426dc in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3101
ptaf = <optimized out>
ptif = 0x5569888806f0 <per_thread_init_list>
ptdf = <optimized out>
ptff = <optimized out>
init_left = 0
init_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}
init_cond = {__data = {__lock = 333, __futex = 0, __total_seq = 279, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x6c, __nwaiters = 0, __broadcast_seq = 0}, __size = "M\001\000\000\000\000\000\000\027\001", '\000' <repeats 22 times>, "l", '\000' <repeats 14 times>, __align = 333}
#7 0x00007f60b8c081cf in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#8 0x00007f60b7439d83 in iopl () from /lib64/libc.so.6
No symbol table info available.
#9 0x0000000000000000 in ?? ()
No symbol table info available.
Thread 34 (Thread 0x7f604a7f4700 (LWP 512444)):
#0 pl_wait_unlock_long (lock=lock@entry=0x556989325fa0, mask=mask@entry=18446744056529682432) at include/import/plock.h:93
loops = 45574
ret = <optimized out>
m = 145602
#1 0x000055698825d6cd in listener_accept (l=0x556989325f90) at src/listener.c:963
__lk_r = 0x556989325fa0
__set_r = 4
__msk_r = 18446744056529682432
count = 2
status = 5
mask = <optimized out>
cli_conn = 0x0
p = 0x556989324470
max_accept = 4
next_conn = 3
next_feconn = <optimized out>
next_actconn = <optimized out>
expire = <optimized out>
ret = <optimized out>
__func__ = "listener_accept"
#2 0x00005569882a7995 in fd_update_events (fd=58, evts=<optimized out>) at src/fd.c:628
locked = <optimized out>
old = 1179939
new = <optimized out>
new_flags = <optimized out>
must_stop = 0
rmask = 2109440
tmask = <optimized out>
#3 0x00005569880e8942 in _do_poll (p=<optimized out>, exp=-991615768, wake=0) at src/ev_epoll.c:268
n = <optimized out>
e = <optimized out>
status = <optimized out>
fd = <optimized out>
count = <optimized out>
updt_idx = <optimized out>
wait_time = 22
old_fd = <optimized out>
#4 0x0000556988242043 in run_poll_loop () at src/haproxy.c:2983
next = <optimized out>
wake = <optimized out>
__func__ = "run_poll_loop"
#5 0x00005569882426dc in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3101
ptaf = <optimized out>
ptif = 0x5569888806f0 <per_thread_init_list>
ptdf = <optimized out>
ptff = <optimized out>
init_left = 0
init_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}
init_cond = {__data = {__lock = 333, __futex = 0, __total_seq = 279, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x6c, __nwaiters = 0, __broadcast_seq = 0}, __size = "M\001\000\000\000\000\000\000\027\001", '\000' <repeats 22 times>, "l", '\000' <repeats 14 times>, __align = 333}
#6 0x00007f60b8c081cf in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#7 0x00007f60b7439d83 in iopl () from /lib64/libc.so.6
No symbol table info available.
#8 0x0000000000000000 in ?? ()
No symbol table info available.
Thread 33 (Thread 0x7f604aff5700 (LWP 512443)):
#0 pl_wait_unlock_long (lock=lock@entry=0x556989325fa0, mask=mask@entry=18446744056529682432) at include/import/plock.h:93
loops = 1155
ret = <optimized out>
m = 201593
#1 0x000055698825d6cd in listener_accept (l=0x556989325f90) at src/listener.c:963
__lk_r = 0x556989325fa0
__set_r = 4
__msk_r = 18446744056529682432
count = 6
status = 5
mask = <optimized out>
cli_conn = 0x0
p = 0x556989324470
max_accept = 4
next_conn = 7
next_feconn = <optimized out>
next_actconn = <optimized out>
expire = <optimized out>
ret = <optimized out>
__func__ = "listener_accept"
#2 0x00005569882a7995 in fd_update_events (fd=58, evts=<optimized out>) at src/fd.c:628
locked = <optimized out>
old = 1179939
new = <optimized out>
new_flags = <optimized out>
must_stop = 0
rmask = 8592570880
tmask = <optimized out>
#3 0x00005569880e8942 in _do_poll (p=<optimized out>, exp=-991615768, wake=0) at src/ev_epoll.c:268
n = <optimized out>
e = <optimized out>
status = <optimized out>
fd = <optimized out>
count = <optimized out>
updt_idx = <optimized out>
wait_time = 24
old_fd = <optimized out>
#4 0x0000556988242043 in run_poll_loop () at src/haproxy.c:2983
next = <optimized out>
wake = <optimized out>
__func__ = "run_poll_loop"
#5 0x00005569882426dc in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3101
ptaf = <optimized out>
ptif = 0x5569888806f0 <per_thread_init_list>
ptdf = <optimized out>
ptff = <optimized out>
init_left = 0
init_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}
init_cond = {__data = {__lock = 333, __futex = 0, __total_seq = 279, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x6c, __nwaiters = 0, __broadcast_seq = 0}, __size = "M\001\000\000\000\000\000\000\027\001", '\000' <repeats 22 times>, "l", '\000' <repeats 14 times>, __align = 333}
#6 0x00007f60b8c081cf in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#7 0x00007f60b7439d83 in iopl () from /lib64/libc.so.6
No symbol table info available.
#8 0x0000000000000000 in ?? ()
No symbol table info available.
Thread 32 (Thread 0x7f604b7f6700 (LWP 512442)):
#0 pl_wait_unlock_long (lock=lock@entry=0x556989325fa0, mask=mask@entry=18446744069414584320) at include/import/plock.h:93
loops = 166417
ret = <optimized out>
m = 203765
#1 0x000055698825b525 in limit_listener (l=l@entry=0x556989325f90, list=list@entry=0x556989325bf8) at src/listener.c:582
__lk_r = 0x556989325fa0
__pl_r = <optimized out>
#2 0x000055698825d6f0 in listener_accept (l=0x556989325f90) at src/listener.c:1220
cli_conn = <optimized out>
p = 0x556989324470
max_accept = 11
next_conn = 11
next_feconn = 0
next_actconn = 0
expire = 0
ret = <optimized out>
__func__ = "listener_accept"
#3 0x00005569882a7995 in fd_update_events (fd=58, evts=<optimized out>) at src/fd.c:628
locked = <optimized out>
old = 1179939
new = <optimized out>
new_flags = <optimized out>
must_stop = 0
rmask = 47314401792
tmask = <optimized out>
#4 0x00005569880e8942 in _do_poll (p=<optimized out>, exp=-991615768, wake=0) at src/ev_epoll.c:268
n = <optimized out>
e = <optimized out>
status = <optimized out>
fd = <optimized out>
count = <optimized out>
updt_idx = <optimized out>
wait_time = 24
old_fd = <optimized out>
#5 0x0000556988242043 in run_poll_loop () at src/haproxy.c:2983
next = <optimized out>
wake = <optimized out>
__func__ = "run_poll_loop"
#6 0x00005569882426dc in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3101
ptaf = <optimized out>
ptif = 0x5569888806f0 <per_thread_init_list>
ptdf = <optimized out>
ptff = <optimized out>
init_left = 0
init_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}
init_cond = {__data = {__lock = 333, __futex = 0, __total_seq = 279, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x6c, __nwaiters = 0, __broadcast_seq = 0}, __size = "M\001\000\000\000\000\000\000\027\001", '\000' <repeats 22 times>, "l", '\000' <repeats 14 times>, __align = 333}
#7 0x00007f60b8c081cf in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#8 0x00007f60b7439d83 in iopl () from /lib64/libc.so.6
No symbol table info available.
#9 0x0000000000000000 in ?? ()
No symbol table info available.
Thread 31 (Thread 0x7f604bff7700 (LWP 512441)):
#0 pl_wait_unlock_long (lock=lock@entry=0x556989325fa0, mask=mask@entry=18446744069414584320) at include/import/plock.h:93
loops = 39283
ret = <optimized out>
m = 88631
#1 0x000055698825b525 in limit_listener (l=l@entry=0x556989325f90, list=list@entry=0x556989325bf8) at src/listener.c:582
__lk_r = 0x556989325fa0
__pl_r = <optimized out>
#2 0x000055698825d6f0 in listener_accept (l=0x556989325f90) at src/listener.c:1220
cli_conn = <optimized out>
p = 0x556989324470
max_accept = 14
next_conn = 14
next_feconn = 0
next_actconn = 0
expire = 0
ret = <optimized out>
__func__ = "listener_accept"
#3 0x00005569882a7995 in fd_update_events (fd=58, evts=<optimized out>) at src/fd.c:628
locked = <optimized out>
old = 1179939
new = <optimized out>
new_flags = <optimized out>
must_stop = 0
rmask = 50132974080
tmask = <optimized out>
#4 0x00005569880e8942 in _do_poll (p=<optimized out>, exp=-991615768, wake=0) at src/ev_epoll.c:268
n = <optimized out>
e = <optimized out>
status = <optimized out>
fd = <optimized out>
count = <optimized out>
updt_idx = <optimized out>
wait_time = 24
old_fd = <optimized out>
#5 0x0000556988242043 in run_poll_loop () at src/haproxy.c:2983
next = <optimized out>
wake = <optimized out>
__func__ = "run_poll_loop"
#6 0x00005569882426dc in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3101
ptaf = <optimized out>
ptif = 0x5569888806f0 <per_thread_init_list>
ptdf = <optimized out>
ptff = <optimized out>
init_left = 0
init_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}
init_cond = {__data = {__lock = 333, __futex = 0, __total_seq = 279, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x6c, __nwaiters = 0, __broadcast_seq = 0}, __size = "M\001\000\000\000\000\000\000\027\001", '\000' <repeats 22 times>, "l", '\000' <repeats 14 times>, __align = 333}
#7 0x00007f60b8c081cf in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#8 0x00007f60b7439d83 in iopl () from /lib64/libc.so.6
No symbol table info available.
#9 0x0000000000000000 in ?? ()
No symbol table info available.
Thread 30 (Thread 0x7f604c7f8700 (LWP 512440)):
#0 pl_wait_unlock_long (lock=lock@entry=0x556989325fa0, mask=mask@entry=18446744069414584320) at include/import/plock.h:93
loops = 14294
ret = <optimized out>
m = 135842
#1 0x000055698825b525 in limit_listener (l=l@entry=0x556989325f90, list=list@entry=0x556989325bf8) at src/listener.c:582
__lk_r = 0x556989325fa0
__pl_r = <optimized out>
#2 0x000055698825d6f0 in listener_accept (l=0x556989325f90) at src/listener.c:1220
cli_conn = <optimized out>
p = 0x556989324470
max_accept = 13
next_conn = 13
next_feconn = 0
next_actconn = 0
expire = 0
ret = <optimized out>
__func__ = "listener_accept"
#3 0x00005569882a7995 in fd_update_events (fd=58, evts=<optimized out>) at src/fd.c:628
locked = <optimized out>
old = 1179939
new = <optimized out>
new_flags = <optimized out>
must_stop = 0
rmask = 49596103168
tmask = <optimized out>
#4 0x00005569880e8942 in _do_poll (p=<optimized out>, exp=-991615768, wake=0) at src/ev_epoll.c:268
n = <optimized out>
e = <optimized out>
status = <optimized out>
fd = <optimized out>
count = <optimized out>
updt_idx = <optimized out>
wait_time = 24
old_fd = <optimized out>
#5 0x0000556988242043 in run_poll_loop () at src/haproxy.c:2983
next = <optimized out>
wake = <optimized out>
__func__ = "run_poll_loop"
#6 0x00005569882426dc in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3101
ptaf = <optimized out>
ptif = 0x5569888806f0 <per_thread_init_list>
ptdf = <optimized out>
ptff = <optimized out>
init_left = 0
init_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}
init_cond = {__data = {__lock = 333, __futex = 0, __total_seq = 279, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x6c, __nwaiters = 0, __broadcast_seq = 0}, __size = "M\001\000\000\000\000\000\000\027\001", '\000' <repeats 22 times>, "l", '\000' <repeats 14 times>, __align = 333}
#7 0x00007f60b8c081cf in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#8 0x00007f60b7439d83 in iopl () from /lib64/libc.so.6
No symbol table info available.
#9 0x0000000000000000 in ?? ()
No symbol table info available.
Thread 29 (Thread 0x7f604cff9700 (LWP 512439)):
#0 pl_wait_unlock_long (lock=lock@entry=0x556989325fa0, mask=mask@entry=18446744069414584320) at include/import/plock.h:93
loops = 181717
ret = <optimized out>
m = 218405
#1 0x000055698825b525 in limit_listener (l=l@entry=0x556989325f90, list=list@entry=0x556989325bf8) at src/listener.c:582
__lk_r = 0x556989325fa0
__pl_r = <optimized out>
#2 0x000055698825d6f0 in listener_accept (l=0x556989325f90) at src/listener.c:1220
cli_conn = <optimized out>
p = 0x556989324470
max_accept = 20
next_conn = 20
next_feconn = 0
next_actconn = 0
expire = 0
ret = <optimized out>
__func__ = "listener_accept"
#3 0x00005569882a7995 in fd_update_events (fd=58, evts=<optimized out>) at src/fd.c:628
locked = <optimized out>
old = 1179939
new = <optimized out>
new_flags = <optimized out>
must_stop = 0
rmask = 51216481024
tmask = <optimized out>
#4 0x00005569880e8942 in _do_poll (p=<optimized out>, exp=-991615768, wake=0) at src/ev_epoll.c:268
n = <optimized out>
e = <optimized out>
status = <optimized out>
fd = <optimized out>
count = <optimized out>
updt_idx = <optimized out>
wait_time = 24
old_fd = <optimized out>
#5 0x0000556988242043 in run_poll_loop () at src/haproxy.c:2983
next = <optimized out>
wake = <optimized out>
__func__ = "run_poll_loop"
#6 0x00005569882426dc in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3101
ptaf = <optimized out>
ptif = 0x5569888806f0 <per_thread_init_list>
ptdf = <optimized out>
ptff = <optimized out>
init_left = 0
init_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}
init_cond = {__data = {__lock = 333, __futex = 0, __total_seq = 279, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x6c, __nwaiters = 0, __broadcast_seq = 0}, __size = "M\001\000\000\000\000\000\000\027\001", '\000' <repeats 22 times>, "l", '\000' <repeats 14 times>, __align = 333}
#7 0x00007f60b8c081cf in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#8 0x00007f60b7439d83 in iopl () from /lib64/libc.so.6
No symbol table info available.
#9 0x0000000000000000 in ?? ()
No symbol table info available.
Thread 28 (Thread 0x7f604d7fa700 (LWP 512438)):
#0 pl_wait_unlock_long (lock=lock@entry=0x556989325fa0, mask=mask@entry=18446744069414584320) at include/import/plock.h:93
loops = 3125
ret = <optimized out>
m = 201593
#1 0x000055698825b525 in limit_listener (l=l@entry=0x556989325f90, list=list@entry=0x556989325bf8) at src/listener.c:582
__lk_r = 0x556989325fa0
__pl_r = <optimized out>
#2 0x000055698825d6f0 in listener_accept (l=0x556989325f90) at src/listener.c:1220
cli_conn = <optimized out>
p = 0x556989324470
max_accept = 12
next_conn = 12
next_feconn = 0
next_actconn = 0
expire = 0
ret = <optimized out>
__func__ = "listener_accept"
#3 0x00005569882a7995 in fd_update_events (fd=58, evts=<optimized out>) at src/fd.c:628
locked = <optimized out>
old = 1179939
new = <optimized out>
new_flags = <optimized out>
must_stop = 0
rmask = 49461885440
tmask = <optimized out>
#4 0x00005569880e8942 in _do_poll (p=<optimized out>, exp=-991615768, wake=0) at src/ev_epoll.c:268
n = <optimized out>
e = <optimized out>
status = <optimized out>
fd = <optimized out>
count = <optimized out>
updt_idx = <optimized out>
wait_time = 24
old_fd = <optimized out>
#5 0x0000556988242043 in run_poll_loop () at src/haproxy.c:2983
next = <optimized out>
wake = <optimized out>
__func__ = "run_poll_loop"
#6 0x00005569882426dc in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3101
ptaf = <optimized out>
ptif = 0x5569888806f0 <per_thread_init_list>
ptdf = <optimized out>
ptff = <optimized out>
init_left = 0
init_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}
init_cond = {__data = {__lock = 333, __futex = 0, __total_seq = 279, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x6c, __nwaiters = 0, __broadcast_seq = 0}, __size = "M\001\000\000\000\000\000\000\027\001", '\000' <repeats 22 times>, "l", '\000' <repeats 14 times>, __align = 333}
#7 0x00007f60b8c081cf in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#8 0x00007f60b7439d83 in iopl () from /lib64/libc.so.6
No symbol table info available.
#9 0x0000000000000000 in ?? ()
No symbol table info available.
Thread 27 (Thread 0x7f604dffb700 (LWP 512437)):
#0 pl_wait_unlock_long (lock=lock@entry=0x556989325fa0, mask=mask@entry=18446744056529682432) at include/import/plock.h:93
loops = 16428
ret = <optimized out>
m = 39818
#1 0x000055698825d6cd in listener_accept (l=0x556989325f90) at src/listener.c:963
__lk_r = 0x556989325fa0
__set_r = 4
__msk_r = 18446744056529682432
count = 7
status = 5
mask = <optimized out>
cli_conn = 0x0
p = 0x556989324470
max_accept = 4
next_conn = 8
next_feconn = <optimized out>
next_actconn = <optimized out>
expire = <optimized out>
ret = <optimized out>
__func__ = "listener_accept"
#2 0x00005569882a7995 in fd_update_events (fd=58, evts=<optimized out>) at src/fd.c:628
locked = <optimized out>
old = 1179939
new = <optimized out>
new_flags = <optimized out>
must_stop = 0
rmask = 12887538176
tmask = <optimized out>
#3 0x00005569880e8942 in _do_poll (p=<optimized out>, exp=-991615789, wake=0) at src/ev_epoll.c:268
n = <optimized out>
e = <optimized out>
status = <optimized out>
fd = <optimized out>
count = <optimized out>
updt_idx = <optimized out>
wait_time = 3
old_fd = <optimized out>
#4 0x0000556988242043 in run_poll_loop () at src/haproxy.c:2983
next = <optimized out>
wake = <optimized out>
__func__ = "run_poll_loop"
#5 0x00005569882426dc in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3101
ptaf = <optimized out>
ptif = 0x5569888806f0 <per_thread_init_list>
ptdf = <optimized out>
ptff = <optimized out>
init_left = 0
init_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}
init_cond = {__data = {__lock = 333, __futex = 0, __total_seq = 279, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x6c, __nwaiters = 0, __broadcast_seq = 0}, __size = "M\001\000\000\000\000\000\000\027\001", '\000' <repeats 22 times>, "l", '\000' <repeats 14 times>, __align = 333}
#6 0x00007f60b8c081cf in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#7 0x00007f60b7439d83 in iopl () from /lib64/libc.so.6
No symbol table info available.
#8 0x0000000000000000 in ?? ()
No symbol table info available.
Thread 26 (Thread 0x7f604e7fc700 (LWP 512436)):
#0 pl_wait_unlock_long (lock=lock@entry=0x556989325fa0, mask=mask@entry=18446744069414584320) at include/import/plock.h:94
loops = 33767
ret = <optimized out>
m = 40247
#1 0x000055698825b525 in limit_listener (l=l@entry=0x556989325f90, list=list@entry=0x556989325bf8) at src/listener.c:582
__lk_r = 0x556989325fa0
__pl_r = <optimized out>
#2 0x000055698825d6f0 in listener_accept (l=0x556989325f90) at src/listener.c:1220
cli_conn = <optimized out>
p = 0x556989324470
max_accept = 27
next_conn = 27
next_feconn = 0
next_actconn = 0
expire = 0
ret = <optimized out>
__func__ = "listener_accept"
#3 0x00005569882a7995 in fd_update_events (fd=58, evts=<optimized out>) at src/fd.c:628
locked = <optimized out>
old = 1179939
new = <optimized out>
new_flags = <optimized out>
must_stop = 0
rmask = 51484917595
tmask = <optimized out>
#4 0x00005569880e8942 in _do_poll (p=<optimized out>, exp=-991615768, wake=0) at src/ev_epoll.c:268
n = <optimized out>
e = <optimized out>
status = <optimized out>
fd = <optimized out>
count = <optimized out>
updt_idx = <optimized out>
wait_time = 24
old_fd = <optimized out>
#5 0x0000556988242043 in run_poll_loop () at src/haproxy.c:2983
next = <optimized out>
wake = <optimized out>
__func__ = "run_poll_loop"
#6 0x00005569882426dc in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3101
ptaf = <optimized out>
ptif = 0x5569888806f0 <per_thread_init_list>
ptdf = <optimized out>
ptff = <optimized out>
init_left = 0
init_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}
init_cond = {__data = {__lock = 333, __futex = 0, __total_seq = 279, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x6c, __nwaiters = 0, __broadcast_seq = 0}, __size = "M\001\000\000\000\000\000\000\027\001", '\000' <repeats 22 times>, "l", '\000' <repeats 14 times>, __align = 333}
#7 0x00007f60b8c081cf in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#8 0x00007f60b7439d83 in iopl () from /lib64/libc.so.6
No symbol table info available.
#9 0x0000000000000000 in ?? ()
No symbol table info available.
Thread 25 (Thread 0x7f604effd700 (LWP 512435)):
#0 pl_wait_unlock_long (lock=lock@entry=0x556989325fa0, mask=mask@entry=18446744069414584320) at include/import/plock.h:93
loops = 99891
ret = <optimized out>
m = 233849
#1 0x000055698825b525 in limit_listener (l=l@entry=0x556989325f90, list=list@entry=0x556989325bf8) at src/listener.c:582
__lk_r = 0x556989325fa0
__pl_r = <optimized out>
#2 0x000055698825d6f0 in listener_accept (l=0x556989325f90) at src/listener.c:1220
cli_conn = <optimized out>
p = 0x556989324470
max_accept = 32
next_conn = 32
next_feconn = 0
next_actconn = 0
expire = 0
ret = <optimized out>
__func__ = "listener_accept"
#3 0x00005569882a7995 in fd_update_events (fd=58, evts=<optimized out>) at src/fd.c:628
locked = <optimized out>
old = 1179939
new = <optimized out>
new_flags = <optimized out>
must_stop = 0
rmask = 68698374015
tmask = <optimized out>
#4 0x00005569880e8942 in _do_poll (p=<optimized out>, exp=-991615768, wake=0) at src/ev_epoll.c:268
n = <optimized out>
e = <optimized out>
status = <optimized out>
fd = <optimized out>
count = <optimized out>
updt_idx = <optimized out>
wait_time = 24
old_fd = <optimized out>
#5 0x0000556988242043 in run_poll_loop () at src/haproxy.c:2983
next = <optimized out>
wake = <optimized out>
__func__ = "run_poll_loop"
#6 0x00005569882426dc in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3101
ptaf = <optimized out>
ptif = 0x5569888806f0 <per_thread_init_list>
ptdf = <optimized out>
ptff = <optimized out>
init_left = 0
init_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}
init_cond = {__data = {__lock = 333, __futex = 0, __total_seq = 279, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x6c, __nwaiters = 0, __broadcast_seq = 0}, __size = "M\001\000\000\000\000\000\000\027\001", '\000' <repeats 22 times>, "l", '\000' <repeats 14 times>, __align = 333}
#7 0x00007f60b8c081cf in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#8 0x00007f60b7439d83 in iopl () from /lib64/libc.so.6
No symbol table info available.
#9 0x0000000000000000 in ?? ()
No symbol table info available.
Thread 24 (Thread 0x7f604f7fe700 (LWP 512434)):
#0 pl_wait_unlock_long (lock=lock@entry=0x556989325fa0, mask=mask@entry=18446744069414584320) at include/import/plock.h:93
loops = 106570
ret = <optimized out>
m = 230328
#1 0x000055698825b525 in limit_listener (l=l@entry=0x556989325f90, list=list@entry=0x556989325bf8) at src/listener.c:582
__lk_r = 0x556989325fa0
__pl_r = <optimized out>
#2 0x000055698825d6f0 in listener_accept (l=0x556989325f90) at src/listener.c:1220
cli_conn = <optimized out>
p = 0x556989324470
max_accept = 19
next_conn = 19
next_feconn = 0
next_actconn = 0
expire = 0
ret = <optimized out>
__func__ = "listener_accept"
#3 0x00005569882a7995 in fd_update_events (fd=58, evts=<optimized out>) at src/fd.c:628
locked = <optimized out>
old = 1179939
new = <optimized out>
new_flags = <optimized out>
must_stop = 0
rmask = 51208092416
tmask = <optimized out>
#4 0x00005569880e8942 in _do_poll (p=<optimized out>, exp=-991615768, wake=0) at src/ev_epoll.c:268
n = <optimized out>
e = <optimized out>
status = <optimized out>
fd = <optimized out>
count = <optimized out>
updt_idx = <optimized out>
wait_time = 24
old_fd = <optimized out>
#5 0x0000556988242043 in run_poll_loop () at src/haproxy.c:2983
next = <optimized out>
wake = <optimized out>
__func__ = "run_poll_loop"
#6 0x00005569882426dc in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3101
ptaf = <optimized out>
ptif = 0x5569888806f0 <per_thread_init_list>
ptdf = <optimized out>
ptff = <optimized out>
init_left = 0
init_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}
init_cond = {__data = {__lock = 333, __futex = 0, __total_seq = 279, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x6c, __nwaiters = 0, __broadcast_seq = 0}, __size = "M\001\000\000\000\000\000\000\027\001", '\000' <repeats 22 times>, "l", '\000' <repeats 14 times>, __align = 333}
#7 0x00007f60b8c081cf in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#8 0x00007f60b7439d83 in iopl () from /lib64/libc.so.6
No symbol table info available.
#9 0x0000000000000000 in ?? ()
No symbol table info available.
Thread 23 (Thread 0x7f604ffff700 (LWP 512433)):
#0 pl_wait_unlock_long (lock=lock@entry=0x556989325698, mask=mask@entry=18446744069414584320) at include/import/plock.h:93
loops = 11892
ret = <optimized out>
m = 83350
#1 0x000055698825bbc6 in resume_listener (l=0x556989325f90, lpx=lpx@entry=0) at src/listener.c:520
__lk_r = 0x556989325698
__set_r = 21474836484
__msk_r = 18446744069414584320
__pl_r = <optimized out>
px = 0x556989324470
was_paused = 0
ret = 1
#2 0x000055698825db87 in listener_release (l=<optimized out>) at src/listener.c:1242
fe = 0x556989324470
#3 0x00005569882a359f in session_free (sess=0x7f60182d7930) at src/session.c:71
conn = <optimized out>
conn_back = <optimized out>
srv_list = <optimized out>
srv_list_back = <optimized out>
#4 0x00005569882a39cd in conn_session_free (conn=0x7f60502fa070) at src/session.c:106
No locals.
#5 0x0000556988295cba in mux_pt_destroy (ctx=ctx@entry=0x7f6044156e00) at src/mux_pt.c:229
conn = 0x7f60502fa070
__FUNCTION__ = "mux_pt_destroy"
#6 0x00005569882962d1 in mux_pt_detach (sd=0x7f60400643b0) at src/mux_pt.c:442
conn = 0x7f60502fa070
ctx = 0x7f6044156e00
__FUNCTION__ = "mux_pt_detach"
#7 0x000055698824afb8 in sc_detach_endp (scp=scp@entry=0x7f604ffdbbb8) at src/stconn.c:356
conn = 0x7f60502fa070
sedesc = 0x7f60400643b0
sc = 0x55698aa05e70
#8 0x000055698824b5fa in sc_destroy (sc=0x55698aa05e70) at src/stconn.c:425
No locals.
#9 0x000055698818e963 in stream_free (s=s@entry=0x7f60281ca020) at src/stream.c:721
sess = <optimized out>
fe = 0x556989324470
bref = <optimized out>
back = <optimized out>
i = <optimized out>
#10 0x00005569881933e5 in process_stream (t=<optimized out>, context=0x7f60281ca020, state=<optimized out>) at src/stream.c:2632
srv = <optimized out>
s = 0x7f60281ca020
sess = <optimized out>
rqf_last = 12886114
rpf_last = <optimized out>
rq_prod_last = 10
rq_cons_last = 10
rp_cons_last = 10
rp_prod_last = 10
req_ana_back = 0
req = 0x7f60281ca040
res = 0x7f60281ca0a0
scf = <optimized out>
scb = 0x7f6034110b40
rate = <optimized out>
#11 0x0000556988273f80 in run_tasks_from_lists (budgets=budgets@entry=0x7f604ffdbde0) at src/task.c:632
process = <optimized out>
tl_queues = <optimized out>
t = 0x7f6034657c60
budget_mask = 15 '\017'
profile_entry = 0x0
done = 0
queue = 1
state = 1028
ctx = <optimized out>
__func__ = "run_tasks_from_lists"
#12 0x00005569882747c9 in process_runnable_tasks () at src/task.c:861
tt = 0x556988aa0f00 <ha_thread_ctx+8448>
lrq = <optimized out>
grq = <optimized out>
t = <optimized out>
max = {0, 43, 0, 0}
max_total = <optimized out>
tmp_list = <optimized out>
queue = <optimized out>
max_processed = <optimized out>
lpicked = <optimized out>
gpicked = <optimized out>
heavy_queued = 1
budget = <optimized out>
#13 0x0000556988242083 in run_poll_loop () at src/haproxy.c:2912
next = <optimized out>
wake = <optimized out>
__func__ = "run_poll_loop"
#14 0x00005569882426dc in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3101
ptaf = <optimized out>
ptif = 0x5569888806f0 <per_thread_init_list>
ptdf = <optimized out>
ptff = <optimized out>
init_left = 0
init_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}
init_cond = {__data = {__lock = 333, __futex = 0, __total_seq = 279, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x6c, __nwaiters = 0, __broadcast_seq = 0}, __size = "M\001\000\000\000\000\000\000\027\001", '\000' <repeats 22 times>, "l", '\000' <repeats 14 times>, __align = 333}
#15 0x00007f60b8c081cf in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#16 0x00007f60b7439d83 in iopl () from /lib64/libc.so.6
No symbol table info available.
#17 0x0000000000000000 in ?? ()
No symbol table info available.
Thread 22 (Thread 0x7f6054c46700 (LWP 512432)):
#0 0x00005569882a6778 in fd_add_to_fd_list (list=0x556988ae7480 <update_list>, fd=58) at src/fd.c:134
next = 0
new = <optimized out>
old = 0
last = <optimized out>
#1 0x00005569882a6fe4 in updt_fd_polling (fd=<optimized out>) at src/fd.c:511
update_mask = 0
tgrp = <optimized out>
#2 0x000055698825bb2b in resume_listener (l=0x556989325f90, lpx=lpx@entry=0) at src/listener.c:542
px = 0x556989324470
was_paused = <optimized out>
ret = 1
#3 0x000055698825be72 in dequeue_proxy_listeners (px=px@entry=0x556989324470) at src/listener.c:613
listener = <optimized out>
#4 0x000055698825cf90 in listener_accept (l=<optimized out>) at src/listener.c:1190
cli_conn = <optimized out>
p = 0x556989324470
max_accept = <optimized out>
next_conn = <optimized out>
next_feconn = <optimized out>
next_actconn = <optimized out>
expire = <optimized out>
ret = <optimized out>
__func__ = "listener_accept"
#5 0x00005569882a7995 in fd_update_events (fd=58, evts=<optimized out>) at src/fd.c:628
locked = <optimized out>
old = 1179939
new = <optimized out>
new_flags = <optimized out>
must_stop = 0
rmask = 40869853696
tmask = <optimized out>
#6 0x00005569880e8942 in _do_poll (p=<optimized out>, exp=-991615792, wake=0) at src/ev_epoll.c:268
n = <optimized out>
e = <optimized out>
status = <optimized out>
fd = <optimized out>
count = <optimized out>
updt_idx = <optimized out>
wait_time = 12
old_fd = <optimized out>
Additional Information
No response
Activity
xanaxalan commentedon Jan 25, 2023
Darlelet commentedon Jan 25, 2023
Hi,
From your haproxy -vv output, haproxy reports that it detected 2 available threads:
Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=2).From your config snippet, I can see that you manually set
nbthreadto 36.Is the haproxy -vv output from another machine that the one which suffers from the crash?
If not, any reason for manually setting nbthread to 36?
Thanks
xanaxalan commentedon Jan 25, 2023
@Darlelet
nbthreadis set to 36 in the config to accommodate resources of the machine. A little correction-vvis not from the same machine as the config, I reverted the installation, because of the 2 crashes, this was installed in vm.Darlelet commentedon Jan 26, 2023
Ok thank you, so do you confirm that the machine running the config you posted has >= 36 threads and that you don't have other haproxy processes running on the same threads?
Because from your trace it looks like some lock contention over the listener lock.
Moreover, from your config file:
maxconn xYou intentionally didn't share maxconn info for your global and https frontend sections, are you sure this is correctly tuned for your load?
The trace suggests that nearly all threads are waiting for the listener lock in
limit_listener()function, this function is only being called when either reaching global maxconn or proxy maxconn.Edit: I was wrong, it could also result from
rate-limit sessions,maxconnrate,maxsessrateormaxsslratestatements, but given your config,rate-limit sessionsandmaxconncould be interesting leads.Also, you said you were rolling out to 2.7.1 from an older version when the issue occurred: what is your current haproxy version? (the one that seems to do fine under load using the same config as the one you provided)
This could help pinpoint the major changes in haproxy architecture that could explain why the crash is occurring with 2.7.1 but not with your older version with same config and load.
xanaxalan commentedon Jan 26, 2023
We have other machines with hap 2.4.19 in anycast setup with this configuration and sizing should be fine. It runs without an issue.
Upgrade was done on one of them and as I mentioned it was fine for couple of days until load spike killed it. Version which we are running now below. maxconn in defaults is set to
150000haproxy -vv
Darlelet commentedon Jan 26, 2023
Thank you for the extra information
On your existing setup running 2.4.19, do you appear to witness some log messages similar to this one from time to time?
Proxy X reached the configured maximum connection limit. Please check the global 'maxconn' value.xanaxalan commentedon Jan 26, 2023
No nothing about the
maxconnin the logs.wtarreau commentedon Jan 27, 2023
I confirm that the trace shows that a listener reached some limits, however we don't know if these are per-process limits or per-proxy limits. It could also be a configured rate-limit but I'm not seeing any in the config snippet. Anoter possibility could be that the peers port received some heavy connection attempts, for example if it's open to the whole world, as peers do have per-proxy connection limits as well.
All this doesn't justify such a crash, it looks like there's a race there. But a few very related ones were fixed between 2.7.1 and 2.7.2, and I'm not sure whether they could produce this one or not. In any case it would be worth updating it. If it's not urgent on your side we may emit 2.7.3 next week with a few more fixes (unrelated though).
Darlelet commentedon Jan 27, 2023
Looks like
rate-limit sessionsis in play in @xanaxalan configwtarreau commentedon Jan 27, 2023
Ah you're right, I indeed missed it.
xanaxalan commentedon Jan 27, 2023
I will deploy 7.2 today and monitor it, will see how it behaves. I can pick up 7.3 next week after that. Just out of curiosity, when old version of hap is running and update of pkg is done, should hap survive reload? I am doing restarts but just was wondering if this would be possible :)
xanaxalan commentedon Jan 31, 2023
Unfortunately on 7.2 I've got same behavior. Crashed today.
xanaxalan commentedon Feb 27, 2023
I was testing 2.7.3 and still crashing. Tried on different HW this time and it survived medium load spike just fine which was promising and then after 3 days it crashed without any significant load. Attaching core dump from 2.7.3 [ (gdb) t a a bt full ]
gdb_hap_2.7.3_ssl_1.1.1t.txt
2 remaining items
Darlelet commentedon Feb 27, 2023
Great!
The patch should be applied for fd_rm_from_fd_list() as well, as in @xanaxalan previous trace, the issue occurred in
fd_rm_from_fd_list()The culprit is likely 4d9888c since the "volatile" keyword was lost at multiple places with this commit.
xanaxalan commentedon Feb 27, 2023
Great we have something to test :) thank you!! Will patch and let it run again. Will let you know about the outcome.
xanaxalan commentedon Feb 27, 2023
@Darlelet can you provide me a diff for
fd_rm_from_fd_list()I am looking at it and it's not obvious to me what I should change.wtarreau commentedon Feb 27, 2023
@Darlelet oh indeed, good catch. I didn't remember about that one but it definitely explains what it's new.
@xanaxalan here's the one Aurélien was talking about (not even build-tested but should be straight-forward):
At each place, the change consists in replacing
x=ywithx=HA_ATOMIC_LOAD(&y).wtarreau commentedon Feb 27, 2023
I just check for other parts being affected. The one above was not used for x86_64, but it could have got blocked by another one looping in fd_add_to_fd_list). I've run a quick check on the whole executable and only found the fd_add one on my binary:
It's indeed only present since 2.7, as 2.6 doesn't have it.
wtarreau commentedon Feb 27, 2023
@Darlelet since you diagnosed the problem and its origin, I'll let you handle the final patch if that's OK for you. Thanks!
Darlelet commentedon Feb 27, 2023
You're right Willy, I didn't take the time to properly re-check the original backtrace and remembered I saw the same check being performed in
fd_rm_from_fd_list()while looking at the code a few days ago, that got me confused in thinking that I originally saw it in the backtrace but it was only a side-discovery.So indeed this last one probably doesn't affect @xanaxalan
However, there is one last thing about
fd_rm_from_fd_list():What do you think Willy, is this needed as well (since fdtab is not declared as volatile)?
Or even better:
(this way we make sure we don't miss anything from 4d9888c)
You did help a lot, thank you :) Alright I'm fine with that
wtarreau commentedon Feb 27, 2023
The first one could be a good idea, because indeed, while we're copying a structure, it only contains two fields and could reside in a register pair that the compiler may propagate through the loop if it had enough. The second option however is not correct for 32-bit (single load) and is generally not a good idea in that we prefer to avoid explicit casts as much as possible as they tend to hide problems. And yes I know we do have some a few lines later for the CAS and DWCAS but that's for the good cause :-) You can even use this one which will preserve type checking in case we decide to change the type of fdtab->update in the future:
Darlelet commentedon Feb 27, 2023
Oh yes indeed I missed the double-load for the DWCAS case, good catch
I'll take it, thanks!
wtarreau commentedon Feb 27, 2023
And by the way I could confirm that exactly this could happen there indeed with the following simplified code:
It just keeps a copy of the values in a single 64-bit register and works it it in loops:
Fixing it using the volatile dereference does the job, as expected:
With that said, on my machine fd_rm_from_fd_list() didn't have this problem since it used to store the update struct into rax which is overwritten over and over, so the fdtab[fd].update was always reloaded.
BUG/MEDIUM: fd: avoid infinite loops in fd_add_to_fd_list and fd_rm_f…
BUG/MEDIUM: fd: avoid infinite loops in fd_add_to_fd_list and fd_rm_f…
xanaxalan commentedon Mar 8, 2023
Looks like latest patch fixed the issue. Did not see any crashes since I deployed patched version.
Darlelet commentedon Mar 8, 2023
Great! Thank you for letting us know @xanaxalan