Tracked server briefly goes UP while its agent remains DOWN · Issue #3543 · haproxy/haproxy · GitHub
Skip to content

Tracked server briefly goes UP while its agent remains DOWN #3543

Open

Description

@snh

Detailed Description of the Problem

A server configured with both track and agent-check can go UP when the server it tracks recovers, even though its agent is still returning down.

I reproduced this with HAProxy 3.4.7. Once the tracked server recovers, the tracking server becomes available for traffic while its agent health remains zero. It goes DOWN again when its next agent check runs.

The length of the window depends on agent-inter. With a 10-second interval, the server may receive traffic for close to 10 seconds despite its agent continuing to return down.

Expected Behavior

The tracking server should remain DOWN until its own agent reports that it is UP.

The agent-check documentation says that only the agent may revert its own actions, and that only an agent reporting up can bring back a server that the agent marked DOWN. Recovery of a tracked server does not appear to be a documented exception.

This is related to #1176, but is not the same bug. That issue involved an agent UP result overriding a failed regular health check. This case involves tracked-server recovery overriding an agent DOWN result.

Steps to Reproduce the Behavior

  1. Save the configuration below as haproxy.cfg.
  2. Start a TCP server for the health check and traffic target:
socat TCP-LISTEN:28080,bind=127.0.0.1,reuseaddr,fork SYSTEM:'echo ok'
  1. Start an agent that always responds with down:
socat TCP-LISTEN:28081,bind=127.0.0.1,reuseaddr,fork SYSTEM:'echo down'
  1. Start HAProxy:
haproxy -f ./haproxy.cfg
  1. Wait until checks/source is DOWN and service/tracking is DOWN via its agent. The source initially checks the unreachable 192.0.2.1 address.
printf "show servers state\n" | socat stdio /tmp/haproxy-track-agent-repro.sock
  1. Change only the source server's address so that its regular TCP health check can recover:
printf "set server checks/source addr 127.0.0.1 port 28080\n" | socat stdio /tmp/haproxy-track-agent-repro.sock
  1. Inspect the state again immediately:
printf "show servers state\n" | socat stdio /tmp/haproxy-track-agent-repro.sock

checks/source recovers to operational state 2. service/tracking also changes to operational state 2 even though its agent health remains zero.

  1. Send traffic through the frontend during this window:
printf "test\n" | nc -w 1 127.0.0.1 28082

The request reaches service/tracking and returns ok.

  1. After the next agent check, service/tracking goes DOWN again.

This occurs once per recovery of the tracked server. It does not continuously oscillate while the tracked server remains UP.

Do you have any idea what may have caused this?

In HAProxy 3.4.7, check_notify_success() checks the local health check, agent health and tracked-server state before marking a server running:

if ((s->agent.state & CHK_ST_ENABLED) &&
    (s->agent.health < s->agent.rise))
    return;

However, srv_set_running() propagates recovery directly to tracking servers:

for (srv = s->trackers; srv; srv = srv->tracknext) {
    HA_SPIN_LOCK(SERVER_LOCK, &srv->lock);
    srv_set_running(srv, SRV_OP_STCHGC_NONE);
    HA_SPIN_UNLOCK(SERVER_LOCK, &srv->lock);
}

This path does not check the tracking server's agent health. It marks the server running while the agent health remains below its rise threshold.

Do you have an idea how to solve the issue?

Automatic tracker propagation should check whether each tracking server's local checks allow the state transition before marking it running.

It may make sense to extract the eligibility checks from check_notify_success() into a helper and use that helper for automatic tracker propagation. I would avoid changing the general behaviour of srv_set_running(), since it is also used by administrative paths that may intentionally force server state.

A regression test could cover this sequence:

  1. The tracked server is DOWN and the tracking server's agent reports DOWN.
  2. The tracked server recovers while the agent remains DOWN.
  3. The tracking server remains DOWN.
  4. The agent changes to UP.
  5. The tracking server becomes UP.

What is your configuration?

global
    stats socket /tmp/haproxy-track-agent-repro.sock level admin

defaults
    mode tcp
    timeout connect 1s
    timeout client 5s
    timeout server 5s

frontend test
    bind 127.0.0.1:28082
    default_backend service

backend checks
    server source 192.0.2.1:28080 check inter 500ms fall 1 rise 1

backend service
    server tracking 127.0.0.1:28080 track checks/source agent-check agent-addr 127.0.0.1 agent-port 28081 agent-inter 10s

Output of haproxy -vv

HAProxy version 3.4.7-8872a26 2026/10/14 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2031.
Known bugs: http://www.haproxy.org/bugs/bugs-3.4.7.html
Running on: Darwin 27.0.0 Darwin Kernel Version 27.0.0: Tue Aug 11 21:06:40 PDT 2026; root:xnu-13432.1.9~1/RELEASE_ARM64_T6050 arm64
Build options :
  TARGET  = osx
  CC      = cc
  CFLAGS  = -O2 -g -fwrapv
  OPTIONS = USE_OPENSSL=1 USE_ZLIB=1 USE_QUIC=1 USE_PROMEX=1 USE_PCRE2=1 USE_PCRE2_JIT=1
  DEBUG   =

Feature list : -51DEGREES -ACCEPT4 +ACME -BACKTRACE -CLOSEFROM +CPU_AFFINITY -CRYPT_H -DEVICEATLAS -DL -ECH -ENGINE -EPOLL -EVPORTS +GETADDRINFO +KQUEUE -KTLS -LIBATOMIC +LIBCRYPT -LINUX_CAP -LINUX_SPLICE -LINUX_TPROXY -LUA -MATH -MEMORY_PROFILING -NETFILTER -NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL -PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION +QUIC -QUIC_OPENSSL_COMPAT -RT -SHM_OPEN -SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 -TFO +THREAD -THREAD_DUMP +TPROXY +TRACE -WURFL +ZLIB
Detected feature list :

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=32, MAX_THREADS=1024, default=18).
Built with SSL library version : OpenSSL 4.0.3 29 Sep 2026
Running on SSL library version : OpenSSL 4.0.3 29 Sep 2026
SSL library supports TLS extensions : yes
SSL library supports SNI : yes
SSL library default verify directory : /opt/homebrew/etc/openssl@4/certs
SSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
OpenSSL providers loaded : default
QUIC: connection sock-per-conn mode support : no
QUIC: GSO emission support : no
Built with the Prometheus exporter as a service
Built with zlib version : 1.2.12
Running on zlib version : 1.2.12
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using:
Built with PCRE2 version : 10.49 2026-09-28
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with clang compiler version 21.0.0 (clang-2100.3.34.2)

Available polling systems :
     kqueue : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use kqueue.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
       quic : mode=HTTP  side=FE|BE  mux=QUIC  flags=HTX|NO_UPG|FRAMED
       qmux : mode=HTTP  side=FE|BE  mux=QMUX  flags=HTX|NO_UPG
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
       spop : mode=SPOP  side=BE     mux=SPOP  flags=HOL_RISK|NO_UPG
  <default> : mode=SPOP  side=BE     mux=SPOP  flags=HOL_RISK|NO_UPG
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=

Available services : prometheus-exporter
Available filters :
    [BWLIM] bwlim-in
    [BWLIM] bwlim-out
    [CACHE] cache
    [COMP] comp-req
    [COMP] comp-res
    [COMP] compression
    [FCGI] fcgi-app
    [SPOE] spoe
    [TRACE] trace

Last Outputs and Backtraces

No crash.

Additional Information

I found this while testing whether several servers could share a TCP health check while retaining their own agent checks. The issue only affects an agent response such as down that changes operational health. Administrative states such as maint are preserved during tracked-server recovery.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions