shm-stats-file: backend's own counters alternate between two slots across reloads (frontend and server counters are preserved)

Detailed Description of the Problem

With shm-stats-file set and GUIDs on every object, a reload preserves the counters of frontends and servers as documented. The backend's own row (svname=BACKEND in show stat) does not: after each reload it shows the values of the process before the previous one, as if the backend were attached to the other of two slots in the file on every reload.

Observed stot over three reloads, sending one request after each reload (the frontend and the server row are correct throughout):

start, 3 requests : fe/FRONTEND=3 be/s1=3 be/BACKEND=3
reload 1, +1 : fe/FRONTEND=4 be/s1=4 be/BACKEND=1
reload 2, +1 : fe/FRONTEND=5 be/s1=5 be/BACKEND=4
reload 3, +1 : fe/FRONTEND=6 be/s1=6 be/BACKEND=2

The same pattern holds for every cumulative backend counter we looked at (stot, bin, bout, hrsp_2xx, req_tot, lbtot). The file contains two entries for the backend's GUID after the first reload (strings on the file shows the backend GUID twice, the frontend and server GUIDs once each). It happens with or without a haproxy -c run between reloads.

Reproduced on two separate hosts with the same package.

Expected Behavior

The backend's own counters are preserved across a reload like the frontend's and the servers', or, if backend counters are not meant to be shared, they start from zero on every reload. With no guid on the backend they do start from zero on every reload, which is consistent; with a guid they show stale figures from two processes back, which is misleading in graphs.

Steps to Reproduce the Behavior

Save the configuration below as shm.cfg and replace /tmp/shmtest with an empty writable directory.
haproxy -W -f /tmp/shmtest/shm.cfg -p /tmp/shmtest/pid &
Send three requests: for i in 1 2 3; do curl -s -o /dev/null http://127.0.0.1:18080/; done
echo "show stat" | socat - /tmp/shmtest/admin.sock | cut -d, -f1,2,8
kill -USR2 $(cat /tmp/shmtest/pid), wait a second, send one request, run step 4 again.
Repeat step 5 a few times. be,BACKEND alternates; fe,FRONTEND and be,s1 keep counting.

Do you have any idea what may have caused this?

Only an observation: the backend's GUID appears twice in the file after the first reload, the frontend's and the server's once. It looks as if the new process cannot attach to the backend's slot while the old process still holds it and takes a second one, then alternates between the two on later reloads.

Possibly related: in #3183 a bisect session noted that "reloading the process multiple times would alternate between one old value and a new one for the frontend", fixed by 3262da8. On 3.4.6 we see that alternation for the backend's own row only; frontends and servers are fine. #3320 (GUID truncation) does not apply: the GUIDs here are under ten characters.

Do you have an idea how to solve the issue?

No. As a workaround we no longer set guid on backends, which makes the backend row start from zero on each reload while frontends and servers are still preserved.

What is your configuration?

global
    shm-stats-file /tmp/shmtest/stats.shm
    stats socket /tmp/shmtest/admin.sock mode 600 level admin

defaults
    mode http
    timeout connect 2s
    timeout client 5s
    timeout server 5s

frontend fe
    guid n3-fe
    bind 127.0.0.1:18080 guid-prefix n3-fe-l
    default_backend be

backend be
    guid n3-be
    server s1 127.0.0.1:18081 guid n3-s1

frontend origin
    guid n3-origin
    bind 127.0.0.1:18081
    http-request return status 200 content-type text/plain string ok

Output of `haproxy -vv`

HAProxy version 3.4.6-1ppa1~resolute 2026/09/30 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2031.
Known bugs: http://www.haproxy.org/bugs/bugs-3.4.6.html
Running on: Linux 7.0.0-38-generic #38-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep  4 09:10:14 UTC 2026 x86_64
Build options : 
  TARGET  = linux-glibc
  CC      = x86_64-linux-gnu-gcc
  CFLAGS  = -O2 -g -fwrapv -fvect-cost-model=very-cheap -g -O2 -Werror=implicit-function-declaration -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -flto=auto -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -fdebug-prefix-map=/build/haproxy-fMvzI9/haproxy-3.4.6=/usr/src/haproxy-3.4.6-1ppa1~resolute -Wdate-time -D_FORTIFY_SOURCE=3
  OPTIONS = USE_OPENSSL=1 USE_LUA=1 USE_SLZ=1 USE_OT=1 USE_QUIC=1 USE_PROMEX=1 USE_PCRE2=1 USE_PCRE2_JIT=1
  DEBUG   = 

Feature list : -51DEGREES +ACCEPT4 +ACME +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ECH -ENGINE +EPOLL -EVPORTS +GETADDRINFO +HAVE_TCP_MD5SIG -KQUEUE +KTLS -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL +OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION +QUIC -QUIC_OPENSSL_COMPAT +RT +SHM_OPEN +SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 +TFO +THREAD +THREAD_DUMP +TPROXY +TRACE -WURFL -ZLIB
Detected feature list : +HAVE_WORKING_TCP_MD5SIG

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=32, MAX_THREADS=1024, default=8).
Built with SSL library version : OpenSSL 3.5.5 27 Jan 2026
Running on SSL library version : OpenSSL 3.5.5 27 Jan 2026
SSL library supports TLS extensions : yes
SSL library supports SNI : yes
SSL library default verify directory : /usr/lib/ssl/certs
SSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
OpenSSL providers loaded : default
QUIC: connection sock-per-conn mode support : yes
QUIC: GSO emission support : yes
Built with Lua version : Lua 5.4.8
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with OpenTracing support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.46 2025-08-27
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 15.2.0

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
       quic : mode=HTTP  side=FE|BE  mux=QUIC  flags=HTX|NO_UPG|FRAMED
       qmux : mode=HTTP  side=FE|BE  mux=QMUX  flags=HTX|NO_UPG
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
       spop : mode=SPOP  side=BE     mux=SPOP  flags=HOL_RISK|NO_UPG
  <default> : mode=SPOP  side=BE     mux=SPOP  flags=HOL_RISK|NO_UPG
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=

Available services : prometheus-exporter
Available filters :
	[BWLIM] bwlim-in
	[BWLIM] bwlim-out
	[CACHE] cache
	[COMP] comp-req
	[COMP] comp-res
	[COMP] compression
	[FCGI] fcgi-app
	[  OT] opentracing
	[SPOE] spoe
	[TRACE] trace

Last Outputs and Backtraces

Additional Information

Ubuntu 26.04, package 3.4.6-1ppa1~resolute from the vbernat PPA, x86_64. The same pattern with chroot and user haproxy in global, started as root with -W and reloaded with SIGUSR2, on a second host.

Thanks for the detailed report. This is clearly unexpected as the frontend and backend stats preservation should behave the same no matter the proxy type.

Thanks to your description, I may have found an explanation. It looks like it only happens for backend without frontend capability, due to a typo where the frontend shared array is checked instead of the backend one for a backend... I'm very sorry about that.

Could you please try the patch below to see if it solves the issue? If that's not possible I will perform the tests myself. Thanks!

diff --git a/src/stats-file.c b/src/stats-file.c
index 761f0d2cc..07524f336 100644
--- a/src/stats-file.c
+++ b/src/stats-file.c
@@ -813,7 +813,7 @@ static void shm_stats_file_preload(void)
                                        } else if (curr_obj->type == SHM_STATS_FILE_OBJECT_TYPE_BE) {
                                                if (!px->be_counters.shared.tg)
                                                        px->be_counters.shared.tg = calloc(global.nbtgroups, sizeof(*px->be_counters.shared.tg));
-                                               if (px->fe_counters.shared.tg == NULL)
+                                               if (px->be_counters.shared.tg == NULL)
                                                        goto release;
                                                px->be_counters.shared.tg[obj_tgid - 1] = &curr_obj->data.be;
                                        } else

Tested on a separate Debian host: HAProxy 3.4.6 built from the haproxy.org tarball twice with the same flags (make TARGET=linux-glibc), once as released and once with your patch, running the reproducer from the report against each.

Released, backend row (stot) over three reloads with one request after each: 3, 1, 4, 2
Patched: 3, 4, 5, 6

Frontend and server rows count 3, 4, 5, 6 in both builds. So the patch fixes it. Thanks for the quick analysis!

Thanks again for the quick feedback. I'll provide a redacted patch shortly!

Ok so it looks like this is a regression that was introduced in 3.4 with 5495c88, so only 3.4 and dev are affected

BUG/MEDIUM: stats-file: shm-stats-file be counters not working anymore

Sunbelt Computer Software

PL/B Language Development and Support

Navigation Menu

Repository navigation

shm-stats-file: backend's own counters alternate between two slots across reloads (frontend and server counters are preserved) #3527

Detailed Description of the Problem

Expected Behavior

Steps to Reproduce the Behavior

Do you have any idea what may have caused this?

Do you have an idea how to solve the issue?

What is your configuration?

Output of `haproxy -vv`

Last Outputs and Backtraces

Additional Information

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Sunbelt Computer Software

PL/B Language Development and Support

shm-stats-file: backend's own counters alternate between two slots across reloads (frontend and server counters are preserved) #3527

Description

Detailed Description of the Problem

Expected Behavior

Steps to Reproduce the Behavior

Do you have any idea what may have caused this?

Do you have an idea how to solve the issue?

What is your configuration?

Output of haproxy -vv

Last Outputs and Backtraces

Additional Information

Activity

Darlelet commented on Oct 9, 2026

ssavant2 commented on Oct 9, 2026

Darlelet commented on Oct 9, 2026

Darlelet commented on Oct 9, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Output of `haproxy -vv`