Haproxy 3.3.X: peer server not responding - timeout · Issue #3234 · haproxy/haproxy · GitHub
Skip to content

Haproxy 3.3.X: peer server not responding - timeout #3234

Closed
Closed
Haproxy 3.3.X: peer server not responding - timeout#3234
Labels
3.3The issue affects the HAProxy 3.3 stable branch.status: fixedThis issue is a now-fixed bug.type: bugThis issue describes a bug.

Description

@mougams
mougams
opened on Jan 2, 2026

Detailed Description of the Problem

Using HAProxy 3.3.X, with following peer declared in our conf

peers netsvc-lb-consul-aggregator
    peer ip-10-6-69-250.kitchen.criteo.preprod 0.0.0.0:8082
    peer 127.0.0.1_10004 127.0.0.1:10004

After some times, despite the port being bind by a process

$ ss -tupln | grep 8082
tcp   LISTEN 0      6              0.0.0.0:8082       0.0.0.0:*

A tcp check goes on timeout

$ curl --connect-timeout 10 -vv 127.0.0.1:8082
* Rebuilt URL to: 127.0.0.1:8082/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connection timed out after 10001 milliseconds
* Closing connection 0
curl: (28) Connection timed out after 10001 milliseconds

Expected Behavior

Following should always respond as below, and not go on timeout

$ curl --connect-timeout 10 -vv 127.0.0.1:8082     
* Rebuilt URL to: 127.0.0.1:8082/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 8082 (#0)
> GET / HTTP/1.1
> Host: 127.0.0.1:8082
> User-Agent: curl/7.61.1
> Accept: */*
> 
501
* Closing connection 0

$ nc 127.0.0.1 8082

501

^C

Steps to Reproduce the Behavior

  1. Install HAProxy 3.3.X (here we installed HAProxy 3.3.1)
  2. Use the simple configuration attached below and start HAProxy
  3. Try to curl 127.0.0.1:8082 - after some some time, it should go on timeout

Do you have any idea what may have caused this?

No response

Do you have an idea how to solve the issue?

Revert back to HAProxy 3.2.X (3.2.9 in our case)

What is your configuration?

global
    log 127.0.0.1 format rfc5424 local0 info
    daemon
    stats socket /var/lib/haproxy/stats level admin mode 600 user haproxy group haproxy expose-fd listeners
    stats timeout 2m
    maxconn 1048576
    user haproxy
    group haproxy
    set-dumpable
    tune.bufsize 33792
    tune.runqueue-depth 1200
    tune.sched.low-latency on
    tune.ssl.cachesize 0
    tune.h2.be.initial-window-size 1048576
    tune.h2.fe.initial-window-size 8388608
    ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11
    ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384
    ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384
    ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11
    ssl-default-server-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES256-SHA
    ssl-default-server-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384
    ssl-server-verify none
    ssl-load-extra-files none
    issuers-chain-path /etc/haproxy/tls/ica
    hard-stop-after 14421s
    close-spread-time 291s
    nbthread 1
    no-memory-trimming
    cpu-map auto:1/1-1 0-0
    spread-checks 5
    chroot /etc/haproxy/chroot
    strict-limits
    localpeer ip-10-6-69-250.kitchen.criteo.preprod
    # server-state-file /etc/haproxy/server-states.txt
    profiling.tasks on

defaults
    mode http
    log global
    option abortonclose
    option log-separate-errors
    option httplog
    option dontlognull
    option http-keep-alive
    option forwardfor except 127.0.0.0/8
    option redispatch 1
    option http-ignore-probes
    option http-buffer-request
    retries 2
    retry-on conn-failure 0rtt-rejected
    timeout http-request 10s
    timeout queue 1s
    timeout connect 10s
    timeout client 900s
    timeout server 900s
    timeout http-keep-alive 10s
    timeout check 5s
    maxconn 1048576
    balance random
    http-reuse always
    http-check expect rstatus ^(200|429)
    compression algo gzip
    compression type text/css text/html text/javascript application/javascript text/plain text/xml application/json application/wasm

listen stats
    bind *:8080
    stats enable
    stats uri /haproxy_stats
    http-request use-service prometheus-exporter if { path /metrics }
    monitor-uri /
    errorfile 200 /etc/haproxy/errorfiles/200.http

resolvers system-dns-tcp
    nameserver local tcp@10.1.8.153:53
    timeout resolve 30s
    hold valid 60s
    accepted_payload_size 8192

resolvers system-dns-udp
    nameserver local 10.1.8.153:53
    timeout resolve 3s
    hold valid 60s
    accepted_payload_size 8192

# Monitoring listener
listen monitoring
    bind 127.0.0.1:80
    # Backend health monitoring (/backend_health?name=<backend>)
    http-request set-var(req.backend) urlp(name) if { path /backend_health } { urlp(name) -m found }
    http-request set-var(req.min_healthy_servers) urlp_val(min_healthy_servers) if { path /backend_health } { urlp(min_healthy_servers) -m found }
    http-request deny deny_status 200 if { var(req.backend),nbsrv,sub(req.min_healthy_servers) ge 0 }
    # General purpose process monitoring
    monitor-uri /delivery/monitor/lb-check
    errorfile 200 /etc/haproxy/errorfiles/200.http

peers netsvc-lb-consul-aggregator
    peer ip-10-6-69-250.kitchen.criteo.preprod 0.0.0.0:8082
    peer 127.0.0.1_10004 127.0.0.1:10004

Output of haproxy -vv

$ haproxy -vv
HAProxy version 3.3.1-9c24c11-crto378.el8 2025/12/19 - https://haproxy.org/
Status: stable branch - will stop receiving fixes around Q1 2027.
Known bugs: http://www.haproxy.org/bugs/bugs-3.3.1.html
Running on: Linux 6.12.17-crto125.el8.x86_64 #1 SMP Tue Mar 11 13:58:01 UTC 2025 x86_64
Build options : 
  TARGET  = linux-glibc
  CC      = cc
  CFLAGS  = -O2 -g -fwrapv -fvect-cost-model=very-cheap -DTCP_USER_TIMEOUT=18
  OPTIONS = USE_BACKTRACE=1 USE_LINUX_TPROXY=1 USE_GETADDRINFO=1 USE_OPENSSL_AWSLC=1 USE_ZLIB=1 USE_TFO=1 USE_NS=1 USE_QUIC=1 USE_PROMEX=1 USE_PCRE=1 USE_PCRE_JIT=1 USE_MAXMIND_GEOIP2=1
  DEBUG   = 

Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ECH -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE +KTLS -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY -LUA -MATH +MAXMIND_GEOIP2 -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL +OPENSSL_AWSLC -OPENSSL_WOLFSSL -OT +PCRE -PCRE2 -PCRE2_JIT +PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION +QUIC -QUIC_OPENSSL_COMPAT +RT +SHM_OPEN -SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL +ZLIB +ACME

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=32, MAX_THREADS=1024, default=1).
Built with SSL library version : OpenSSL 1.1.1 (compatible; AWS-LC 1.62.1)
Running on SSL library version : AWS-LC 1.62.1
SSL library supports TLS extensions : yes
SSL library supports SNI : yes
SSL library FIPS mode : no
SSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
QUIC: connection sock-per-conn mode support : yes
QUIC: GSO emission support : no
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE version : 8.42 2018-03-20
Running on PCRE version : 8.42 2018-03-20
PCRE library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 14.2.1 20250110 (Red Hat 14.2.1-7)

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
       quic : mode=HTTP  side=FE|BE  mux=QUIC  flags=HTX|NO_UPG|FRAMED
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=SPOP  side=BE     mux=SPOP  flags=HOL_RISK|NO_UPG
       spop : mode=SPOP  side=BE     mux=SPOP  flags=HOL_RISK|NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : prometheus-exporter
Available filters :
        [BWLIM] bwlim-in
        [BWLIM] bwlim-out
        [CACHE] cache
        [COMP] compression
        [FCGI] fcgi-app
        [SPOE] spoe
        [TRACE] trace

Last Outputs and Backtraces

Additional Information

No response

Activity

  1. mougams
    added
    type: bugThis issue describes a bug.
    status: needs-triageThis issue needs to be triaged.
    on Jan 2, 2026

  2. wtarreau commented on Jan 2, 2026

    @wtarreau
    wtarreau
    on Jan 2, 2026
    Member

    Hi Basha!

    That's not cool at all, but thanks for the first report of the year! Could you please issue a "show peers" on the CLI so that we get a snapshot of internal states etc ? I suspect that applets are stuck for whatever reasons I currently ignore :-/

  3. mougams commented on Jan 2, 2026

    @mougams
    mougams
    on Jan 2, 2026
    ContributorAuthor

    You're welcome 😄

    Here is the command output

    $ echo "show peers" | sudo socat stdio /var/lib/haproxy/stats
0x561047d4c800: [19/Feb/1970:17:02:26] id=netsvc-lb-consul-aggregator disabled=0 flags=0x22100003 resync_timeout=<PAST> task_calls=1916
  0x561047d4efc0: id=127.0.0.1_10004(remote,active) addr=127.0.0.1:10004 app_state=RUNNING learn_state=NOTASSIGNED last_status=ESTA last_hdshk=47m52s
        reconnect=4s heartbeat=2s confirm=0 tx_hbt=957 rx_hbt=957 no_hbt=0 new_conn=1 proto_err=0 coll=0
        flags=0x20 appctx:0x561047ede790 st0=7 st1=0 task_calls=2879 state=EST
  0x561047d6ba10: id=ip-10-6-69-250.kitchen.criteo.preprod(local,inactive) addr=0.0.0.0:8082 app_state=STOPPED learn_state=NOTASSIGNED last_status=NONE last_hdshk=<NEVER>
        reconnect=<NEVER> heartbeat=<NEVER> confirm=0 tx_hbt=0 rx_hbt=0 no_hbt=0 new_conn=0 proto_err=0 coll=0
        flags=0x0

$ echo "show peers state" | sudo socat stdio /var/lib/haproxy/stats
No such peers

    I reproduced it in a test environement, but this also happens in preprod, where there are more than 1 peer.

    And this works all the time on HAProxy 3.2.X - I ran a loop curl for 30mn w/o issues.

  4. wtarreau commented on Jan 2, 2026

    @wtarreau
    wtarreau
    on Jan 2, 2026
    Member

    Thanks! I must confess I'm not fluent in peers :-) Since you say you reproduced it in test environment, how did you proceed to observe this ? And was it quick to reproduce ? (i.e. do you think we have some hope to bisect it ?).

  5. mougams commented on Jan 2, 2026

    @mougams
    mougams
    on Jan 2, 2026
    ContributorAuthor

    First I noticed it in our preprod in our HealthCheck dashboard, all our healthcheck for peers were red.

    In my test environment, with HAProxy started with the conf pasted above, I simply curled in a loop

    while true; do date; curl --connect-timeout 10 -vv 127.0.0.1:8082; echo ""; sleep 2; done

    And I started to observe multiple timeout after some time. FYI, at startup, the endpoint responds correctly.

  6. mougams commented on Jan 2, 2026

    @mougams
    mougams
    on Jan 2, 2026
    ContributorAuthor

    You can for sure bisect it since it works just fine on HAProxy 3.2.9, but fails on HAProxy 3.3.1. I haven't tested it, but I'm pretty sure it will fail on 3.3.0 too.

  7. wtarreau commented on Jan 2, 2026

    @wtarreau
    wtarreau
    on Jan 2, 2026
    Member

    OK thanks, I'll try with this, that's exactly what I needed.

  8. wtarreau commented on Jan 2, 2026

    @wtarreau
    wtarreau
    on Jan 2, 2026
    Member

    In my case I cannot trigger it, in your case does curl immediately return with an error ? Here's what I'm having:

    Fri Jan  2 18:45:24 CET 2026
18:45:24.826337 [0-0] * [SETUP] added
18:45:24.826367 [0-0] *   Trying 127.0.0.1:8082...
18:45:24.826414 [0-0] * [SETUP] Curl_conn_connect(block=0) -> 0, done=0
18:45:24.826428 [0-0] * [SETUP] Curl_conn_connect(block=0) -> 0, done=1
18:45:24.826437 [0-0] * Established connection to 127.0.0.1 (127.0.0.1 port 8082) from 127.0.0.1 port 38098
18:45:24.826454 [0-0] * [SETUP] query ALPN
18:45:24.826460 [0-0] * using HTTP/1.x
18:45:24.826481 [0-0] > GET / HTTP/1.1
18:45:24.826481 [0-0] > Host: 127.0.0.1:8082
18:45:24.826481 [0-0] > User-Agent: curl/8.17.0
18:45:24.826481 [0-0] > Accept: */*
18:45:24.826481 [0-0] >
18:45:24.826516 [0-0] * Request completely sent off
18:45:24.826639 [0-0] * Received HTTP/0.9 when not allowed
18:45:24.826648 [0-0] * closing connection #0
curl: (1) Received HTTP/0.9 when not allowed

    Even if I reduce the sleep delay, after hundreds of attempts it continues to run flawlessly. That's why I'm suspecting a race condition, which might not be easy to spot.

  9. wtarreau commented on Jan 2, 2026

    @wtarreau
    wtarreau
    on Jan 2, 2026
    Member

    I also tried adding a table attached to these peers to try to have updates to be sent, but to no avail for now.

  10. mougams commented on Jan 2, 2026

    @mougams
    mougams
    on Jan 2, 2026
    ContributorAuthor

    In my case I cannot trigger it, in your case does curl immediately return with an error ?

    Yes, I reduced the curl connect to 2s, and it started to fail after 2/3mn of HAProxy process being up
    while true; do date; curl --connect-timeout 2 -vv 127.0.0.1:8082; echo ""; sleep 2; done

    That's why I'm suspecting a race condition, which might not be easy to spot.

    Yes, I suspected something like that too ... But in my env, I'm able to reproduce this all the time on HAProxy 3.3.1.

    Let me know if you think of something I could enable or logs (or maybe add a custom patch to enable some) that could help pin point the issue. Thx!

  11. wtarreau commented on Jan 2, 2026

    @wtarreau
    wtarreau
    on Jan 2, 2026
    Member

    Just to be sure, this is with the same config as above (i.e. no stick-table), and in absence of any traffic, right ?

  12. mougams commented on Jan 2, 2026

    @mougams
    mougams
    on Jan 2, 2026
    ContributorAuthor

    Yep, that's correct.
    And we have a stk aggregator (hapee-extras-stktagg-1.0-44.95.rhel-8.x86_64.rpm) listening on port 10004 on this host.

  13. mougams commented on Jan 2, 2026

    @mougams
    mougams
    on Jan 2, 2026
    ContributorAuthor

    And just saw that a newest version of stk aggregator has been released recently in december

    hapee-extras-stktagg-1.0-44.96.rhel-8.x86_64.rpm 2025-12-11 09:35 61K

    I'll bump it next week.

  14. wtarreau commented on Jan 2, 2026

    @wtarreau
    wtarreau
    on Jan 2, 2026
    Member

    OK so maybe I need to have another peer listening on port 10004 as well, in case it makes any difference. Still not stuck after an hour or so.

  15. capflam commented on Jan 5, 2026

    @capflam
    capflam
    on Jan 5, 2026
    Member

    @mougams, are you able to reproduce the issue by commenting the 2nd peer line (peer 127.0.0.1_10004 127.0.0.1:10004) ? In that case, you may enable the peers and applet traces.

  16. capflam commented on Jan 5, 2026

    @capflam
    capflam
    on Jan 5, 2026
    Member

    FYI, I found an issue with client timeout handling in the peer applet in 3.3. I don't know if it is related to your issue, but I'm digging.

  17. mougams commented on Jan 5, 2026

    @mougams
    mougams
    on Jan 5, 2026
    ContributorAuthor

    Hello @capflam - even by commenting the line

    # peer 127.0.0.1_10004 127.0.0.1:10004

    after some time, I got some timeout

    * Rebuilt URL to: 127.0.0.1:8082/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connection timed out after 2001 milliseconds
* Closing connection 0
curl: (28) Connection timed out after 2001 milliseconds

    I'll activate more logs and dig more on my side. Will let you know.

  18. capflam commented on Jan 5, 2026

    @capflam
    capflam
    on Jan 5, 2026
    Member

    @mougams, here is a patch fixing the issue I found on the peer applet. It can be applied on top of the 3.3.1. Could you test it ?

    0001-BUG-MEDIUM-peers-Properly-handle-shutdown-when-tryin.patch.txt

  19. mougams commented on Jan 5, 2026

    @mougams
    mougams
    on Jan 5, 2026
    ContributorAuthor

    Nice catch @capflam ! I confirm that this fixes the issue 😄
    It's been running w/o issue for almost 1 hour.

  20. haproxy-mirror
    added a commit that references this issue on Jan 5, 2026

    BUG/MEDIUM: peers: Properly handle shutdown when trying to get a line

    0b29b76

  21. capflam commented on Jan 5, 2026

    @capflam
    capflam
    on Jan 5, 2026
    Member

    Thanks for the test. Now merged !

  22. capflam
    added
    status: fixedThis issue is a now-fixed bug.
    3.3The issue affects the HAProxy 3.3 stable branch.
    and removed
    status: needs-triageThis issue needs to be triaged.
    on Jan 5, 2026
  23. mougams
    closed this as completedon Jan 5, 2026
  24. FireBurn
    added a commit that references this issue on Jan 10, 2026

    BUG/MEDIUM: peers: Properly handle shutdown when trying to get a line

    c7961a0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.3The issue affects the HAProxy 3.3 stable branch.status: fixedThis issue is a now-fixed bug.type: bugThis issue describes a bug.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions