shm-stats-file: GUID truncation causes unbounded object growth · Issue #3320 · haproxy/haproxy · GitHub
Skip to content

shm-stats-file: GUID truncation causes unbounded object growth #3320

Closed
Closed
shm-stats-file: GUID truncation causes unbounded object growth#3320
Labels
status: fixedThis issue is a now-fixed bug.type: bugThis issue describes a bug.

Description

@phihos
phihos
opened on Apr 3, 2026

Detailed Description of the Problem

The currently experimental shm-stats-file directive causes the stats file to grow on every reload when a backend has two server GUIDs like this:

    server s1  127.0.0.1:1 guid srv:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:SRV_1 disabled
    server s10 127.0.0.1:1 guid srv:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:SRV_10 disabled
  1. First is 126 chars
  2. Second is 127 chars
  3. First is prefix of second

haproxy seems to truncate the 127th character when persisting the GUID to the shm stats file causing a duplication of the first GUID on every reload. This does not cause an error until the shm-stats-file-max-objects is reached.

Expected Behavior

haproxy should not truncate the last character of GUIDs when persiusting into the stats file.

Steps to Reproduce the Behavior

Take the haproxy.cfg from further below and run this script:

rm -f /dev/shm/haproxy-test
haproxy -f haproxy.cfg -W -db &
sleep 2

# dump object count: expect 5 (4 unique + 1 truncation duplicate)
od -A n -t d4 -j 536 -N 4 /dev/shm/haproxy-test

# each reload adds one duplicate object
for i in $(seq 1 10); do kill -USR2 $!; sleep 0.3; done
sleep 2

# expect 15 (4 unique + 11 duplicates)
od -A n -t d4 -j 536 -N 4 /dev/shm/haproxy-test

kill $!

Do you have any idea what may have caused this?

I am no C expert, but it seems that counters.c:112 truncates GUIDs written to the SHM stats file:

snprintf(shm_obj->guid, sizeof(shm_obj->guid) - 1, "%s", guid_get(guid));

Do you have an idea how to solve the issue?

--- a/src/counters.c
+++ b/src/counters.c
@@ -109,7 +109,7 @@
 		else if (!shared->tg[it]) {
 			struct shm_stats_file_object *shm_obj;

 			shm_obj = shm_stats_file_add_object(errmsg);
 			if (shm_obj) {
-				snprintf(shm_obj->guid, sizeof(shm_obj->guid)- 1, "%s", guid_get(guid));
+				snprintf(shm_obj->guid, sizeof(shm_obj->guid), "%s", guid_get(guid));
 				if (is_be) {

What is your configuration?

global
    expose-experimental-directives
    shm-stats-file /dev/shm/haproxy-test
    shm-stats-file-max-objects 5000

defaults
    mode http
    timeout connect 5s
    timeout client 5s
    timeout server 5s

frontend fe
    bind :9876
    guid fe:test
    default_backend be

backend be
    guid be:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    server s1  127.0.0.1:1 guid srv:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:SRV_1 disabled
    server s10 127.0.0.1:1 guid srv:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:SRV_10 disabled

Output of haproxy -vv

HAProxy version 3.3.6-1c05c79a5 2026/03/19 - https://haproxy.org/
Status: stable branch - will stop receiving fixes around Q1 2027.
Known bugs: http://www.haproxy.org/bugs/bugs-3.3.6.html
Running on: Linux 6.19.8-200.fc43.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Mar 13 22:06:06 UTC 2026 x86_64
Build options : 
  TARGET  = linux-glibc
  CC      = cc
  CFLAGS  = -O2 -g -fwrapv -fvect-cost-model=very-cheap
  OPTIONS = USE_OPENSSL=1 USE_ZLIB=1 USE_PROMEX=1 USE_PCRE2=1 USE_PCRE2_JIT=1
  DEBUG   = 

Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ECH -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE +KTLS -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY -LUA -MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION -QUIC -QUIC_OPENSSL_COMPAT +RT +SHM_OPEN -SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL +ZLIB +ACME +HAVE_TCP_MD5SIG

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=32, MAX_THREADS=1024, default=16).
Built with SSL library version : OpenSSL 3.6.1 27 Jan 2026
Running on SSL library version : OpenSSL 3.6.1 27 Jan 2026
SSL library supports TLS extensions : yes
SSL library supports SNI : yes
SSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
OpenSSL providers loaded : default
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with zlib version : 1.3.1.zlib-ng
Running on zlib version : 1.3.1.zlib-ng
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.47 2025-10-21
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 12.3.0

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=SPOP  side=BE     mux=SPOP  flags=HOL_RISK|NO_UPG
       spop : mode=SPOP  side=BE     mux=SPOP  flags=HOL_RISK|NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : prometheus-exporter
Available filters :
	[BWLIM] bwlim-in
	[BWLIM] bwlim-out
	[CACHE] cache
	[COMP] compression
	[FCGI] fcgi-app
	[SPOE] spoe
	[TRACE] trace

Last Outputs and Backtraces

Additional Information

No response

Activity

  1. phihos
    added
    type: bugThis issue describes a bug.
    status: needs-triageThis issue needs to be triaged.
    on Apr 3, 2026

  2. Darlelet commented on Apr 3, 2026

    @Darlelet
    Darlelet
    on Apr 3, 2026
    Contributor

    Thanks for the report @phihos, I will take a look shortly :)

  3. Darlelet commented on Apr 3, 2026

    @Darlelet
    Darlelet
    on Apr 3, 2026
    Contributor

    Indeed your analysis is correct: shm_obj->guid can store GUID_MAX_LEN + 1 (1 extra byte for the terminating NULL byte that is added by snprintf() automatically), the - 1 was added by mistake, I couldn't recall that snprintf() doesn't write past the number of bytes specified to append the terminating NULL byte, so this is not needed indeed, and as you pointed out it causes unexpected behavior because we don't support the full 128 characters while the GUID API supports it.

    Your proposal looks correct to me :)

    Would you like to provide the patch yourself or do you want us to do that for you?

    Thank you very much for all the details and the problem analysis

  4. Darlelet
    added
    devThis issue affects the HAProxy development branch.
    status: reviewedThis issue was reviewed. A fix is required.
    3.3The issue affects the HAProxy 3.3 stable branch.
    and removed
    status: needs-triageThis issue needs to be triaged.
    on Apr 3, 2026

  5. phihos commented on Apr 3, 2026

    @phihos
    phihos
    on Apr 3, 2026
    ContributorAuthor

    Thank you for getting back to me so quickly @Darlelet!

    Would you like to provide the patch yourself or do you want us to do that for you?

    I know you do the whole mailing list + diff file thing from the olden times. I am not really used to it. Before I fumble around with that process I would be happy if you just did it 😃

  6. phihos commented on Apr 3, 2026

    @phihos
    phihos
    on Apr 3, 2026
    ContributorAuthor

    Oh maybe one more remark: New haproxy processes loaded the shm file with the duplicate entries without question. Maybe some additional input validation would be helpful in case similar issues come up in the future. When validation fails haproxy could discard the file and log a warning.

  7. Darlelet commented on Apr 3, 2026

    @Darlelet
    Darlelet
    on Apr 3, 2026
    Contributor

    You're welcome, no worries I can write the patch and credit you if that sounds good for you

  8. Darlelet commented on Apr 3, 2026

    @Darlelet
    Darlelet
    on Apr 3, 2026
    Contributor

    Hmm, I understand but in fact GUID duplicates are intended in the shm-stats-file, because upon config change one can decide to reuse the same GUID for a different object type, while existing (older) processes still refer to the old one.

    Once the old object (with duplicated GUID) is not used anymore the slot will be automatically reused, but indeed in your case it would tend to pile up when reloading often

  9. Darlelet commented on Apr 3, 2026

    @Darlelet
    Darlelet
    on Apr 3, 2026
    Contributor

    Also, for performance reason, when a new process start, it picks the first object matching the GUID and expected object type, it doesn't try to scan the full file if not needed

  10. phihos commented on Apr 3, 2026

    @phihos
    phihos
    on Apr 3, 2026
    ContributorAuthor

    I see. Then there is not much we can do I guess.

    Sorry for jumping to an adjacent topic, but I wanted to ask before creating a feature request: In my case I only generated GUIDs because I wanted continuous Prometheus metrics without having ghost traffic loss in the graphs. So I do not really care about the GUIDs themselves. I just set them to use the shm file. Could they be autogenerated somehow? I currently use some generic schema like <type>:<name> e.g. be:my_backend. Can we have this autogenerated if not set?

  11. haproxy-mirror
    added a commit that references this issue on Apr 4, 2026

    BUG/MINOR: counters: fix unexpected 127 char GUID truncation for shm-…

    ca53ee1
  12. Darlelet
    added
    status: fixedThis issue is a now-fixed bug.
    and removed
    devThis issue affects the HAProxy development branch.
    status: reviewedThis issue was reviewed. A fix is required.
    on Apr 4, 2026

  13. Darlelet commented on Apr 4, 2026

    @Darlelet
    Darlelet
    on Apr 4, 2026
    Contributor

    Regarding the initial issue it is now fixed in master, and will soon be backported to 3.3

    For your last question, although I understand your need I don't have answers to provide for now as I thought the current GUID design was intentional so I'll let other speak for themselves if they have relevant insights

  14. FireBurn
    added a commit that references this issue on Apr 10, 2026

    BUG/MINOR: counters: fix unexpected 127 char GUID truncation for shm-…

    65a8220
  15. capflam
    removed
    3.3The issue affects the HAProxy 3.3 stable branch.
    on Apr 30, 2026
  16. capflam
    closed this as completedon Apr 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    status: fixedThis issue is a now-fixed bug.type: bugThis issue describes a bug.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions