Repository navigation
Reconnecting one peer delays heartbeats to another idle peer #3536
Description
Detailed Description of the Problem
When one peer reconnects, HAProxy can delay sending heartbeats to another idle peer in the same peers section. Heartbeats from the affected peer continue to arrive normally.
The test below reproduces it on unmodified HAProxy 3.5-dev8, commit 4e6160c786a025f9050a260a39a1ef99045a2cc2, with an empty stick table. C's heartbeat is due at three seconds. B disconnects at two seconds and reconnects half a second later, making B's next heartbeat due at 5.5 seconds. At four seconds, HAProxy has received four heartbeats from C but sent none back.
Abbreviated show peers output from the failing run:
id=C(remote,active) ... app_state=RUNNING learn_state=NOTASSIGNED last_status=ESTA last_hdshk=4s
reconnect=4s heartbeat=<PAST> confirm=0 tx_hbt=0 rx_hbt=4 no_hbt=0
In a longer local test, our local receiver has a five-second inactivity timeout. The missed heartbeats cause it to disconnect and reconnect repeatedly. I only saw this with idle peers; it didn't reproduce while table updates were flowing.
Expected Behaviour
HAProxy should still send C's heartbeat at three seconds. B reconnecting shouldn't change that.
Steps to Reproduce the Behaviour
- Build unmodified commit
4e6160c786a025f9050a260a39a1ef99045a2cc2. - Save the VTest below as
reg-tests/peers/heartbeat.vtc. - Run
TMPDIR=/tmp VTEST_PROGRAM=/path/to/vtest make reg-tests reg-tests/peers/heartbeat.vtc.
The test only needs VTest and HAProxy. The CLI check fails on the original code because C's tx_hbt stays at zero. It passes with the patch below.
Regression test
vtest "An idle peer keeps its heartbeat deadline when another peer reconnects"
feature ignore_unknown_macro
#REGTEST_TYPE=slow
barrier b_ready cond 2
barrier b_rejoined cond 2
barrier b_done cond 4
# Reserve an address for outgoing attempts. The test peers connect to HAProxy.
server s1 {
barrier b_done sync
} -start
haproxy h1 -arg "-L A" -conf {
global
.if feature(THREAD)
nbthread 1
.endif
defaults
timeout client "${HAPROXY_TEST_TIMEOUT-5s}"
timeout connect "${HAPROXY_TEST_TIMEOUT-5s}"
timeout server "${HAPROXY_TEST_TIMEOUT-5s}"
peers peers
bind "fd@${A}"
server A
server B ${s1_addr}:${s1_port}
server C ${s1_addr}:${s1_port}
table stkt type ip size 10 store gpc0
} -start
# Finish the initial resync so it cannot cause extra sync-task wakeups.
client c1 -connect ${h1_A_sock} {
send "HAProxyS 2.1\nA\nC 1 1\n"
recv 4
recv 2
sendhex "0001"
recv 2
barrier b_ready sync
loop 4 {
sendhex "0004"
delay 1
}
recv 2
barrier b_done sync
} -start
barrier b_ready sync
client c2 -connect ${h1_A_sock} {
send "HAProxyS 2.1\nA\nB 1 1\n"
recv 4
sendhex "0004"
delay 2
} -run
delay 0.5
# B is processed after C. Its new heartbeat is later than C's existing one.
client c3 -connect ${h1_A_sock} {
send "HAProxyS 2.1\nA\nB 1 1\n"
recv 4
barrier b_rejoined sync
loop 3 {
sendhex "0004"
delay 1
}
barrier b_done sync
} -start
barrier b_rejoined sync
delay 1.5
# C's heartbeat was due at 3s, not at B's new deadline around 5.5s.
haproxy h1 -cli {
send "show peers"
expect ~ "id=C\\(remote,active\\)[^\n]*\n[^\n]*tx_hbt=[1-9][0-9]* rx_hbt=[1-9][0-9]* no_hbt=0"
send "show peers"
expect ~ "id=B\\(remote,active\\)"
}
barrier b_done sync
client c1 -wait
client c3 -wait
server s1 -wait
Do you have any idea what may have caused this?
The shared task's expiry is reset once in process_peer_sync(). Two assignments in __process_running_peer_sync() then overwrite it rather than keeping the earliest deadline:
task->expire = tick_first(peer->reconnect, peer->heartbeat);They are in the pending-update branch and the idle branch. In this test, B is processed after C, so B's later deadline replaces C's.
Do you have an idea how to solve the issue?
Keeping the earlier deadline at both lines fixes the reproduction:
-task->expire = tick_first(peer->reconnect, peer->heartbeat);
+task->expire = tick_first(task->expire, tick_first(peer->reconnect, peer->heartbeat));The regression failed before this change and passed 20 repeated runs afterwards; both existing non-TLS peer-sync tests also passed.
What is your configuration?
This is the configuration used by the VTest:
global
nbthread 1
defaults
timeout client 5s
timeout connect 5s
timeout server 5s
peers peers
bind "fd@${A}"
server A
server B ${s1_addr}:${s1_port}
server C ${s1_addr}:${s1_port}
table stkt type ip size 10 store gpc0Output of haproxy -vv
This is from the patched build. The failing build used the same commit and build options, without the two-line change.
HAProxy version 3.5-dev8 2026/10/01 - https://haproxy.org/
Status: development branch - not safe for use in production.
Known bugs: https://github.com/haproxy/haproxy/issues?q=is:issue+is:open
Running on: Darwin 27.0.0 Darwin Kernel Version 27.0.0: Tue Aug 11 21:06:40 PDT 2026; root:xnu-13432.1.9~1/RELEASE_ARM64_T6050 arm64
Build options :
TARGET = osx
CC = cc
CFLAGS = -O2 -g -fwrapv
OPTIONS =
DEBUG =
Feature list : -51DEGREES -ACCEPT4 -BACKTRACE -CLOSEFROM +CPU_AFFINITY -CRYPT_H -DEVICEATLAS -DL -ECH -ENGINE -EPOLL -EVPORTS +FCGI +GETADDRINFO +H2 +KQUEUE -KTLS -LIBATOMIC +LIBCRYPT -LINUX_CAP -LINUX_SPLICE -LINUX_TPROXY -LUA -MATH -MEMORY_PROFILING -MINIDEBUG -NETFILTER -NS -OBSOLETE_LINKER -OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL -PCRE -PCRE2 -PCRE2_JIT -PCRE_JIT +POLL -PRCTL -PROCCTL -PROMEX -PTHREAD_EMULATION -QUIC -QUIC_OPENSSL_COMPAT -RT -SHM_OPEN +SLZ +SPOE -SSL -STATIC_PCRE -STATIC_PCRE2 -TFO +THREAD -THREAD_DUMP +TPROXY +TRACE -WURFL -ZLIB
Detected feature list :
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_TGROUPS=32, MAX_THREADS=1024, default=18).
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using:
Built without PCRE or PCRE2 support (using libc's regex instead)
Encrypted password support via crypt(3): yes
Built with clang compiler version 21.0.0 (clang-2100.3.34.2)
Available polling systems :
kqueue : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use kqueue.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
h2 : mode=HTTP side=FE|BE mux=H2 flags=HTX|HOL_RISK|NO_UPG
fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG
spop : mode=SPOP side=BE mux=SPOP flags=HOL_RISK|NO_UPG
<default> : mode=SPOP side=BE mux=SPOP flags=HOL_RISK|NO_UPG
h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO_UPG
<default> : mode=HTTP side=FE|BE mux=H1 flags=HTX
none : mode=TCP side=FE|BE mux=PASS flags=NO_UPG
<default> : mode=TCP side=FE|BE mux=PASS flags=
Available services : none
Available filter classes :
trace, cache-store, decomp-req, decomp-res, spoe, http-comp-req, http-comp-res, bwlim-in, bwlim-out, fcgi
Last Outputs and Backtraces
Additional Information
/cc @awlx
Activity
haproxyFred commentedon Oct 14, 2026
Thanks for the report. I’m currently busy with another task, but I’ll take a look at it asap.
haproxyFred commentedon Oct 15, 2026
Your analysis and proposed fix are correct. The VTC shows that reconnecting peer B through client c3 after c2 disconnects overwrites the shared task’s earlier deadline for peer C, emulated by c1, delaying heartbeats sent to C.
However, the VTC relies on precise timing. For instance, under heavy load, the CLI assertion may run after the delayed heartbeat has already been sent, allowing the test to pass even without the fix. We therefore cannot use it as a reliable regression test in its current form.
Could you submit a patch following the guidelines in the CONTRIBUTING file?
BUG/MEDIUM: peers: keep the earliest sync task deadline
haproxyFred commentedon Oct 16, 2026
@snh I didn't know if you were planning to fix it, so I did it myself.
brenc commentedon Oct 19, 2026
FYI: I reported this a few weeks ago (#3520) along w/ several other peers related bugs. I've been running w/ my patched binary for several weeks. Today I rolled out 3.4.7 (plus patches) which is how I noticed this issue.
haproxyFred commentedon Oct 20, 2026
@brenc thank you for this reminder. I have just close #3520 which is a duplicate of this one.
BUG/MEDIUM: peers: keep the earliest sync task deadline