{{ message }}
Repository navigation
Segfault when a backend goes down #3538
Description
Detailed Description of the Problem
We have observed several segmentation faults triggered by a backend going down.
[WARNING] (13) : Server be_kd_dev/backend1dev is DOWN, reason: Layer7 timeout, check dura>
[WARNING] (13) : Server be_kd_dev/backend2dev is DOWN, reason: Layer7 timeout, check dura>
[ALERT] (1) : Current worker (13) exited with code 139 (Segmentation fault)
[WARNING] (1) : A worker process unexpectedly died and this can only be explained by a bug in haproxy o>
Please check that you are running an up to date and maintained version of haproxy and open a bug report.
HAProxy version 3.4.3-80ea565fd 2026/07/29 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2031.
Known bugs: http://www.haproxy.org/bugs/bugs-3.4.3.html
Running on: Linux 5.14.0-687.52.1.el9_8.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Sep 25 13:22:27 UTC 2026 x86_64
[ALERT] (1) : exit-on-failure: killing every processes with SIGTERM
[WARNING] (1) : All workers exited. Exiting... (139)
Coredump shows this bt:
#0 0x000055e953e813ea in set_backend_down (be=0x7f58e4814800) at src/backend.c:3013
#1 0x000055e953dd2de5 in srv_update_status (s=s@entry=0x7f58e4829000, type=type@entry=0, cause=<optimized out>) at src/server.c:7326
#2 0x000055e953dd3c41 in srv_set_stopped (s=0x7f58e4829000, cause=<optimized out>) at src/server.c:2215
#3 0x000055e953dd3e45 in srv_set_stopped (s=<optimized out>, cause=<optimized out>) at src/server.c:2222
#4 0x000055e953ec53b4 in check_notify_failure (check=check@entry=0x7f58e48292e0) at src/check.c:593
#5 0x000055e953ec7cb6 in process_chk_conn (t=<optimized out>, context=0x7f58e48292e0, state=<optimized out>) at src/check.c:1392
#6 0x000055e953f95178 in run_tasks_from_lists (budgets=<optimized out>) at include/haproxy/tinfo.h:127
#7 0x000055e953f95bcf in process_runnable_tasks () at src/task.c:913
#8 0x000055e953ed6356 in run_poll_loop () at src/haproxy.c:3013
#9 0x000055e953ed6a51 in run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:3244
#10 0x000055e953c53692 in main (argc=<optimized out>, argv=0x7ffee7a64668) at src/haproxy.c:4020
It faults on access to shared counters src/backend.c:3013.
Our config has shm-state-file and guids for all frontends, backends, servers, binds.
Expected Behavior
No segfault ;)
Steps to Reproduce the Behavior
Not sure yet. I believe it requires the shm-stats-file, some reloads, and a backend going down.
Do you have any idea what may have caused this?
I would guess it is a problem with a pointer into shared memory. I have also observed that stats counters are zero and no longer counting after some (?) reloads. Probably, when a backend goes down in this situation, it uses a bad pointer.
Do you have an idea how to solve the issue?
No response
What is your configuration?
global
localpeer proxy1dev
maxconn 1000
log 127.0.0.1:514 format rfc5424 local0 info
log-send-hostname
stats socket /run/haproxy/haproxy.socket level admin expose-fd listeners
stats timeout 2m
shm-stats-file /run/haproxy/shm_stats_file
server-state-file /run/haproxy/server_state
hard-stop-after 15m
peers haproxies_dev
peer proxy1dev 10.7.222.51:2201
peer proxy2dev 10.7.222.52:2201
defaults default_http_frontend
mode http
log global
defaults default_http_backend
mode http
timeout server 30s
timeout connect 1s
timeout check 3s
retries 3
load-server-state-from-file global
log global
frontend fe_kd_dev from default_http_frontend
guid fe_kd_dev
bind 10.7.242.18:443 ssl crt "@public/startc" guid-prefix bnd_fe_kd_dev:443
default_backend be_kd_dev
backend be_kd_dev from default_http_backend
guid be_kd_dev
option httpchk HEAD /actuator/health
http-check send ver HTTP/1.1 hdr Host %[bc_dst]:%[bc_dst_port]
default-server check inter 10s fastinter 1s downinter 2s observe layer7 on-error mark-down
server backend1dev 10.7.200.110:8098 verifyhost backend1dev guid srv_be_kd_dev_backend1dev:8098
server backend2dev 10.7.200.210:8098 verifyhost backend2dev guid srv_be_kd_dev_backend2dev:8098Output of haproxy -vv
HAProxy version 3.4.3-80ea565fd 2026/07/29 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2031.
Known bugs: http://www.haproxy.org/bugs/bugs-3.4.3.html
Running on: Linux 5.14.0-687.52.1.el9_8.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Sep 25 13:22:27 UTC 2026 x86_64
Build options :
TARGET = linux-glibc
CC = cc
CFLAGS = -O2 -g -fwrapv -fvect-cost-model=very-cheap
OPTIONS = USE_PTHREAD_EMULATION=1 USE_LINUX_TPROXY=1 USE_GETADDRINFO=1 USE_OPENSSL_AWSLC=1 USE_LUA=1 USE_SLZ=1 USE_TFO=1 USE_QUIC=1 USE_PROMEX=1 USE_PCRE2=1 USE_PCRE2_JIT=1
DEBUG =
Feature list : -51DEGREES +ACCEPT4 +ACME +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ECH -ENGINE +EPOLL -EVPORTS +GETADDRINFO +HAVE_TCP_MD5SIG -KQUEUE +KTLS -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL +OPENSSL_AWSLC -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX +PTHREAD_EMULATION +QUIC -QUIC_OPENSSL_COMPAT +RT +SHM_OPEN +SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 +TFO +THREAD +THREAD_DUMP +TPROXY +TRACE -WURFL -ZLIB
Detected feature list : +HAVE_WORKING_TCP_MD5SIG
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_TGROUPS=32, MAX_THREADS=1024, default=2).
Built with SSL library version : OpenSSL 1.1.1 (compatible; AWS-LC 4.0.0)
Running on SSL library version : AWS-LC 4.0.0
SSL library supports TLS extensions : yes
SSL library supports SNI : yes
SSL library FIPS mode : no
SSL library default verify directory : /etc/ssl/certs
SSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
QUIC: connection sock-per-conn mode support : yes
QUIC: GSO emission support : yes
Built with Lua version : Lua 5.4.7
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.46 2025-08-27
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 14.2.0
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
qmux : mode=HTTP side=FE|BE mux=QMUX flags=HTX|NO_UPG
quic : mode=HTTP side=FE|BE mux=QUIC flags=HTX|NO_UPG|FRAMED
h2 : mode=HTTP side=FE|BE mux=H2 flags=HTX|HOL_RISK|NO_UPG
<default> : mode=HTTP side=FE|BE mux=H1 flags=HTX
h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO_UPG
fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG
<default> : mode=SPOP side=BE mux=SPOP flags=HOL_RISK|NO_UPG
spop : mode=SPOP side=BE mux=SPOP flags=HOL_RISK|NO_UPG
<default> : mode=TCP side=FE|BE mux=PASS flags=
none : mode=TCP side=FE|BE mux=PASS flags=NO_UPG
Available services : prometheus-exporter
Available filters :
[BWLIM] bwlim-in
[BWLIM] bwlim-out
[CACHE] cache
[COMP] comp-req
[COMP] comp-res
[COMP] compression
[FCGI] fcgi-app
[SPOE] spoe
[TRACE] trace
Last Outputs and Backtraces
Additional Information
No response
Activity
wtarreau commentedon Oct 14, 2026
Hello,
your version missed 197 bug fixes, a number of which concern health checks and could very possibly be this issue. Please update, try again and let us know if the issue persists. There's no way one will try to guess with such a number of already fixed bugs still affecting your version: https://www.haproxy.org/bugs/bugs-3.4.3.html
flyingflo commentedon Oct 15, 2026
Thanks @wtarreau good point. I still have no reliable reproducer, too. We will upgrade to the latest 3.4.
wtarreau commentedon Oct 15, 2026
OK. I'm closing for now to help tracking remaining issues, but feel free to reopen if you get the problem again.