shm-stats-file doesn't preserve stats

Detailed Description of the Problem

The shm-stats-file is meant to persist frontend, backend, and server statistics. But it doesn't.

Environment

I'm using the HAProxy 3.3 Docker container. This is my Docker Compose file:

---
networks:
  custom_net:
    driver: bridge
    ipam:
      config:
        - subnet: 172.16.0.0/16
          gateway: 172.16.0.1

services:

  haproxy:
    image: haproxytech/haproxy-ubuntu:s6-3.3
    ports:
      - "80:80"
      - "443:443/tcp"
      - "5555:5555"
      - "9999:9999"
      - "8404:8404"
      - "8405:8405"
    volumes:
      - "./haproxy:/etc/haproxy"
    networks:
      custom_net:
        ipv4_address: 172.16.0.10

  web2:
    image: jmalloc/echo-server
    networks:
      custom_net:
        ipv4_address: 172.16.0.12

I've set in the global section of my configuration:

expose-experimental-directives
shm-stats-file /dev/shm/myshm

I've set guid in the frontend, backend, and on the server line.

I reload HAProxy via this command:

docker compose exec haproxy /package/admin/s6/command/s6-svc -2 /run/s6-rc/servicedirs/haproxy

I can see that the file is created at /dev/shm/myshm.
Statistics aren't preserved. I can see this on the Stats page and via echo "show stat typed" | sudo socat stdio tcp4-connect:127.0.0.1:9999.

Expected Behavior

Stats are preserved.

Steps to Reproduce the Behavior

Use HAProxy 3.3
Add the required global directives expose-experimental-directives and shm-stats-file.
Add guid to the frontend, backend, and server.
Make requests.
Reload HAProxy.

Do you have any idea what may have caused this?

No response

Do you have an idea how to solve the issue?

No response

What is your configuration?

global
  #log /dev/log local0 info
  log stdout format raw local0
  pidfile /var/run/haproxy.pid
  # user haproxy
  # group haproxy
  stats socket /var/run/haproxy.sock mode 660 level admin
  stats socket ipv4@*:9999  level admin
  stats timeout 30s
  ssl-load-extra-del-ext
  ssl-load-extra-files key

  # NEED THIS FOR QUIC TO BACKEND
  # ALSO FOR PERSISTING STATS
  expose-experimental-directives
  shm-stats-file /dev/shm/myshm

defaults
  log global
  mode http
  option httplog
  option dontlognull
  timeout connect 10m
  timeout client  10m
  timeout server  10m

frontend stats
  mode http
  bind :8404
  stats enable
  stats uri /stats
  stats refresh 3s
  stats admin if LOCALHOST

frontend mysite
  guid a88e2a95-547e-47f1-b406-ea82ea47abcc
  bind :80
  use_backend webservers
  
backend webservers
  guid 3db38dc1-4aa8-4172-b7de-affb7f1f51a8
  server web2 172.16.0.12:8080 check maxconn 30 guid 775e29c2-0b97-4f19-9976-dba604b833f4

Output of `haproxy -vv`

HAProxy version 3.3-dev12-b26a6d5 2025/11/08 - https://haproxy.org/
Status: development branch - not safe for use in production.
Known bugs: https://github.com/haproxy/haproxy/issues?q=is:issue+is:open
Running on: Linux 5.15.167.4-microsoft-standard-WSL2 #1 SMP Tue Nov 5 00:21:55 UTC 2024 x86_64
Build options : 
  TARGET  = linux-glibc
  CC      = cc
  CFLAGS  = -O2 -g -fwrapv -fvect-cost-model=very-cheap
  OPTIONS = USE_PTHREAD_EMULATION=1 USE_LINUX_TPROXY=1 USE_GETADDRINFO=1 USE_OPENSSL=1 USE_LUA=1 USE_SLZ=1 USE_TFO=1 USE_QUIC=1 USE_PROMEX=1 USE_PCRE2=1 USE_PCRE2_JIT=1 USE_QUIC_OPENSSL_COMPAT=1
  DEBUG   = 

Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ECH -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE +KTLS -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX +PTHREAD_EMULATION +QUIC +QUIC_OPENSSL_COMPAT +RT +SHM_OPEN +SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB +ACME

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=32, MAX_THREADS=1024, default=20).
Built with SSL library version : OpenSSL 3.0.13 30 Jan 2024
Running on SSL library version : OpenSSL 3.0.13 30 Jan 2024
SSL library supports TLS extensions : yes
SSL library supports SNI : yes
SSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
OpenSSL providers loaded : default
QUIC: connection sock-per-conn mode support : yes
QUIC: GSO emission support : yes
Built with Lua version : Lua 5.4.6
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.42 2022-12-11
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 13.3.0

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
       quic : mode=HTTP  side=FE|BE  mux=QUIC  flags=HTX|NO_UPG|FRAMED
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=SPOP  side=BE     mux=SPOP  flags=HOL_RISK|NO_UPG
       spop : mode=SPOP  side=BE     mux=SPOP  flags=HOL_RISK|NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : prometheus-exporter
Available filters :
        [BWLIM] bwlim-in
        [BWLIM] bwlim-out
        [CACHE] cache
        [COMP] compression
        [FCGI] fcgi-app
        [SPOE] spoe
        [TRACE] trace

Last Outputs and Backtraces

Additional Information

No response

@Darlelet

Indeed, thanks Nick! I could bisect it to this commit before which it would still work fine: 5c299de ("MEDIUM: stats: consider that shared stats pointers may be NULL").

@Darlelet I've seen a very strange behavior starting with this commit during the bisect session, reloading the process multiple times would alternate between one old value and a new one for the frontend.

I'm not seeing anything obviously wrong in the patch. Aurélien, do you think that the loops iterating over thread groups could face a null in the middle of others ? For example here:

@@ -74,7 +74,7 @@ void counters_be_shared_drop(struct be_counters_shared *counters);
        uint64_t __ret = 0;                                                   \
         int it;                                                               \
                                                                               \
-       for (it = 0; it < global.nbtgroups; it++)                             \
+       for (it = 0; (it < global.nbtgroups && scounters[it]); it++)          \
                __ret += rfunc((type *)((char *)scounters[it] + offset));     \
        __ret;                                                                \
 })

Shouldn't it be this instead ?

@@ -74,7 +74,7 @@ void counters_be_shared_drop(struct be_counters_shared *counters);
        uint64_t __ret = 0;                                                   \
        int it;                                                               \
                                                                               \
        for (it = 0; it < global.nbtgroups; it++)                             \
+             if (scounters[it])
                __ret += rfunc((type *)((char *)scounters[it] + offset));     \
        __ret;                                                                \
 })

Maybe we're only reloading some thread groups or shards and not all, which could explain why it alternates on reload ?

It should be fixed with 3262da8

In 5c299de I completely overlooked the fact that shm stats preloading should be considered as init step for the counters and thus the pointers are expected to be NULL there, and the extra check used everywhere else before using the pointer doesn't apply there.

Thanks both for pointing the issue

It was a dumb mistake indeed

Closing now that the fix is merged. Thanks Aurélien!

Sunbelt Computer Software

PL/B Language Development and Support

Navigation Menu

Repository navigation

shm-stats-file doesn't preserve stats #3183

Detailed Description of the Problem

Environment

Expected Behavior

Steps to Reproduce the Behavior

Do you have any idea what may have caused this?

Do you have an idea how to solve the issue?

What is your configuration?

Output of `haproxy -vv`

Last Outputs and Backtraces

Additional Information

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Sunbelt Computer Software

PL/B Language Development and Support

shm-stats-file doesn't preserve stats #3183

Description

Detailed Description of the Problem

Environment

Expected Behavior

Steps to Reproduce the Behavior

Do you have any idea what may have caused this?

Do you have an idea how to solve the issue?

What is your configuration?

Output of haproxy -vv

Last Outputs and Backtraces

Additional Information

Activity

wtarreau commented on Nov 11, 2025

Darlelet commented on Nov 12, 2025

Darlelet commented on Nov 12, 2025

wtarreau commented on Nov 13, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Output of `haproxy -vv`