You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Tracked server briefly goes UP while its agent remains DOWN #3543
A server configured with both track and agent-check can go UP when the server it tracks recovers, even though its agent is still returning down.
I reproduced this with HAProxy 3.4.7. Once the tracked server recovers, the tracking server becomes available for traffic while its agent health remains zero. It goes DOWN again when its next agent check runs.
The length of the window depends on agent-inter. With a 10-second interval, the server may receive traffic for close to 10 seconds despite its agent continuing to return down.
Expected Behavior
The tracking server should remain DOWN until its own agent reports that it is UP.
The agent-check documentation says that only the agent may revert its own actions, and that only an agent reporting up can bring back a server that the agent marked DOWN. Recovery of a tracked server does not appear to be a documented exception.
This is related to #1176, but is not the same bug. That issue involved an agent UP result overriding a failed regular health check. This case involves tracked-server recovery overriding an agent DOWN result.
Steps to Reproduce the Behavior
Save the configuration below as haproxy.cfg.
Start a TCP server for the health check and traffic target:
This path does not check the tracking server's agent health. It marks the server running while the agent health remains below its rise threshold.
Do you have an idea how to solve the issue?
Automatic tracker propagation should check whether each tracking server's local checks allow the state transition before marking it running.
It may make sense to extract the eligibility checks from check_notify_success() into a helper and use that helper for automatic tracker propagation. I would avoid changing the general behaviour of srv_set_running(), since it is also used by administrative paths that may intentionally force server state.
A regression test could cover this sequence:
The tracked server is DOWN and the tracking server's agent reports DOWN.
The tracked server recovers while the agent remains DOWN.
The tracking server remains DOWN.
The agent changes to UP.
The tracking server becomes UP.
What is your configuration?
global
stats socket /tmp/haproxy-track-agent-repro.sock level admin
defaults
mode tcp timeout connect1s timeout client5s timeout server5s
frontend test bind127.0.0.1:28082 default_backend service
backend checks server source192.0.2.1:28080 check inter 500ms fall1 rise1
backend service server tracking 127.0.0.1:28080 track checks/source agent-check agent-addr127.0.0.1 agent-port28081 agent-inter10s
Output of haproxy -vv
HAProxy version 3.4.7-8872a26 2026/10/14 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2031.
Known bugs: http://www.haproxy.org/bugs/bugs-3.4.7.html
Running on: Darwin 27.0.0 Darwin Kernel Version 27.0.0: Tue Aug 11 21:06:40 PDT 2026; root:xnu-13432.1.9~1/RELEASE_ARM64_T6050 arm64
Build options :
TARGET = osx
CC = cc
CFLAGS = -O2 -g -fwrapv
OPTIONS = USE_OPENSSL=1 USE_ZLIB=1 USE_QUIC=1 USE_PROMEX=1 USE_PCRE2=1 USE_PCRE2_JIT=1
DEBUG =
Feature list : -51DEGREES -ACCEPT4 +ACME -BACKTRACE -CLOSEFROM +CPU_AFFINITY -CRYPT_H -DEVICEATLAS -DL -ECH -ENGINE -EPOLL -EVPORTS +GETADDRINFO +KQUEUE -KTLS -LIBATOMIC +LIBCRYPT -LINUX_CAP -LINUX_SPLICE -LINUX_TPROXY -LUA -MATH -MEMORY_PROFILING -NETFILTER -NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL -PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION +QUIC -QUIC_OPENSSL_COMPAT -RT -SHM_OPEN -SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 -TFO +THREAD -THREAD_DUMP +TPROXY +TRACE -WURFL +ZLIB
Detected feature list :
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_TGROUPS=32, MAX_THREADS=1024, default=18).
Built with SSL library version : OpenSSL 4.0.3 29 Sep 2026
Running on SSL library version : OpenSSL 4.0.3 29 Sep 2026
SSL library supports TLS extensions : yes
SSL library supports SNI : yes
SSL library default verify directory : /opt/homebrew/etc/openssl@4/certs
SSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
OpenSSL providers loaded : default
QUIC: connection sock-per-conn mode support : no
QUIC: GSO emission support : no
Built with the Prometheus exporter as a service
Built with zlib version : 1.2.12
Running on zlib version : 1.2.12
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using:
Built with PCRE2 version : 10.49 2026-09-28
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with clang compiler version 21.0.0 (clang-2100.3.34.2)
Available polling systems :
kqueue : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use kqueue.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
quic : mode=HTTP side=FE|BE mux=QUIC flags=HTX|NO_UPG|FRAMED
qmux : mode=HTTP side=FE|BE mux=QMUX flags=HTX|NO_UPG
h2 : mode=HTTP side=FE|BE mux=H2 flags=HTX|HOL_RISK|NO_UPG
h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO_UPG
<default> : mode=HTTP side=FE|BE mux=H1 flags=HTX
fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG
spop : mode=SPOP side=BE mux=SPOP flags=HOL_RISK|NO_UPG
<default> : mode=SPOP side=BE mux=SPOP flags=HOL_RISK|NO_UPG
none : mode=TCP side=FE|BE mux=PASS flags=NO_UPG
<default> : mode=TCP side=FE|BE mux=PASS flags=
Available services : prometheus-exporter
Available filters :
[BWLIM] bwlim-in
[BWLIM] bwlim-out
[CACHE] cache
[COMP] comp-req
[COMP] comp-res
[COMP] compression
[FCGI] fcgi-app
[SPOE] spoe
[TRACE] trace
Last Outputs and Backtraces
No crash.
Additional Information
I found this while testing whether several servers could share a TCP health check while retaining their own agent checks. The issue only affects an agent response such as down that changes operational health. Administrative states such as maint are preserved during tracked-server recovery.
Activity