Repository navigation
Web service using Server-Sent Events (SSE) is always closed after 30 seconds #3534
Description
Detailed Description of the Problem
I have a web service which is using Server-Sent Events (SSE) to stream real-time logs from a server to a client.
The client performs a GET request (using the JavaScript EventSource object) and the server responds with the following HTTP headers:
Content-Type: text/event-stream
Cache-Control: no-cache
Connection: keep-alive
The session is then kept open whilst the server streams events to the client, which the client reads via JavaScript:
EventSource.addEventListener('message', (e) => {
console.log(e.data);
});The server ensures it sends data at least every 5 seconds - even when there are no events, as I send a keepalive event.
The client will never send data to the server as SSE is used for one way streaming from server to client only.
Without HAProxy in the setup the session appears to stays open indefinitely, but as soon as I put HAProxy in the middle, the session is closed after 30 seconds.
I have tried to increase timeout tunnel, timeout client, timeout server, timeout http-request and none of them have any effect on this - it is still closed after 30 seconds with the following message in the logs:
192.0.2.1:55301 [12/Oct/2026:18:39:17.987] fe_https~ be_http/default 1/0/0/0/30004 200 696 - - CD-- 1/1/0/0/0 0/0 "GET https://server.domain/_logs HTTP/3.0"
If I perform a tcpdump then I can see the server is sending data back to the client every 5 seconds - without HAProxy in the middle then the server sees the following request headers from the client:
GET /_logs HTTP/1.1
Host: server.domain
Accept: text/event-stream
Cache-Control: no-cache
Connection: keep-alive
However, if I perform the same capture on the back-end side of HAProxy then we can see that HAProxy changes the Connection header to close in the request.
How come HAProxy is closing the session when we have a stream of data from server to client every 5 seconds?
Expected Behavior
As the server is sending data every 5 seconds towards the client I would expect the session to stay open for longer than 30 seconds.
Steps to Reproduce the Behavior
Not that easy to replicate as you need a web service that is serving an event source, as well as some JavaScript that is processing the response. If required then I can write a Python implementation that recreates the issue, but it isn't a few lines of code.
Do you have any idea what may have caused this?
Hopefully a connection timeout I am unaware of how to change.
Do you have an idea how to solve the issue?
Unfortunately not
What is your configuration?
global
log stdout format raw local0
maxconn 4096
user root
tune.listener.default-shards by-thread
ssl-default-bind-ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256
ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256
ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11
ssl-default-bind-curves X25519MLKEM768:X25519:P-256
defaults
log global
option redispatch
option splice-auto
option socket-stats
option log-health-checks
option http-server-close
option httplog
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 3s
mode http
frontend fe_http
bind ipv4@:80 name ipv4
bind ipv6@:80 name ipv6
redirect scheme https code 301
frontend fe_https
ssl-f-use crt /usr/local/etc/haproxy/fullchain.pem
bind ipv4@:443 ssl alpn h2,http/1.1 allow-0rtt tfo name ipv4
bind ipv6@:443 ssl alpn h2,http/1.1 allow-0rtt tfo name ipv6
bind quic4@:443 ssl alpn h3 allow-0rtt name quic4
bind quic6@:443 ssl alpn h3 allow-0rtt name quic6
http-response set-header Strict-Transport-Security 'max-age=63072000; includeSubDomains; preload'
http-response set-header Alt-Svc 'h3=":443"; ma=86400'
http-request set-header X-Forwarded-For %[src]
use_backend be_http
backend be_http
option httpchk GET /ping
http-check expect string OK
server default 127.0.0.1:8080 checkOutput of haproxy -vv
HAProxy version 3.4.6-56332c5 2026/09/28 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2031.
Known bugs: http://www.haproxy.org/bugs/bugs-3.4.6.html
Running on: Linux 5.14.0-686.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Feb 19 10:49:27 UTC 2026 x86_64
Build options :
TARGET = linux-musl
CC = cc
CFLAGS = -O2 -g -fwrapv -fvect-cost-model=very-cheap
OPTIONS = USE_PTHREAD_EMULATION=1 USE_GETADDRINFO=1 USE_OPENSSL=1 USE_LUA=1 USE_QUIC=1 USE_PROMEX=1 USE_PCRE2=1 USE_PCRE2_JIT=1
DEBUG =
Feature list : -51DEGREES +ACCEPT4 +ACME +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ECH -ENGINE +EPOLL -EVPORTS +GETADDRINFO +HAVE_TCP_MD5SIG -KQUEUE +KTLS -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCT L +PROMEX +PTHREAD_EMULATION +QUIC -QUIC_OPENSSL_COMPAT +RT +SHM_OPEN +SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 +TFO +THREAD +THREAD_DUMP +TPROXY +TRACE -WURFL -ZLIB
Detected feature list : +HAVE_WORKING_TCP_MD5SIG
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_TGROUPS=32, MAX_THREADS=1024, default=2).
Built with SSL library version : OpenSSL 3.5.8 25 Aug 2026
Running on SSL library version : OpenSSL 3.5.8 25 Aug 2026
SSL library supports TLS extensions : yes
SSL library supports SNI : yes
SSL library default verify directory : /etc/ssl/certs
SSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
OpenSSL providers loaded : default
QUIC: connection sock-per-conn mode support : yes
QUIC: GSO emission support : yes
Built with Lua version : Lua 5.4.8
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.48 2026-08-31
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 15.2.0
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
qmux : mode=HTTP side=FE|BE mux=QMUX flags=HTX|NO_UPG
quic : mode=HTTP side=FE|BE mux=QUIC flags=HTX|NO_UPG|FRAMED
h2 : mode=HTTP side=FE|BE mux=H2 flags=HTX|HOL_RISK|NO_UPG
<default> : mode=HTTP side=FE|BE mux=H1 flags=HTX
h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO_UPG
fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG
<default> : mode=SPOP side=BE mux=SPOP flags=HOL_RISK|NO_UPG
spop : mode=SPOP side=BE mux=SPOP flags=HOL_RISK|NO_UPG
<default> : mode=TCP side=FE|BE mux=PASS flags=
none : mode=TCP side=FE|BE mux=PASS flags=NO_UPG
Available services : prometheus-exporter
Available filters :
[BWLIM] bwlim-in
[BWLIM] bwlim-out
[CACHE] cache
[COMP] comp-req
[COMP] comp-res
[COMP] compression
[FCGI] fcgi-app
[SPOE] spoe
[TRACE] trace
Last Outputs and Backtraces
Additional Information
No response
Activity
cmason3 commentedon Oct 13, 2026
Disabled HTTP/3 (QUIC) and it no longer times out after 30 seconds - looks like QUIC is enforcing a 30 second session limit somewhere?
a-denoyelle commentedon Oct 13, 2026
There is indeed the QUIC idle timeout which is set by default to 30 seconds. The connection is silently closed if no frame is received after such delay. You could increase it but it is a negotiated value so if the client announces a smaller one it will be picked instead.
cmason3 commentedon Oct 13, 2026
Thanks! Increasing
tune.quic.fe.max-idle-timeoutseems to have fixed the issue. I am struggling to understand how an "idle" timeout is triggering here, as I am sending data every 5 seconds, so the client is receiving data and the server is sending data, which I wouldn't describe as idle?a-denoyelle commentedon Oct 13, 2026
In the scenario you describe, idle timeout must not be triggered or there is a bug. Are you sure that the exchanges are performed over the QUIC connection ? Maybe the client is opening both a QUIC + TCP connections and switch to the later one ?
cmason3 commentedon Oct 13, 2026
The whole session is happening over QUIC - I have just done a tcpdump and all I can see is UDP 443 traffic to/from the source IP - I can see QUIC encrypted packets every 5 seconds from the server.
What debugs or logging can I enable to help?
I have just done a test and can see this session was established at 15:07:08:
Initially we send some data to the client and then at intervals of 5 seconds we send keepalive data:
HAProxy then stops sending data at 15:07:33 - I assume this is when it decides to close the session, but HAProxy doesn't tell me when it actually prints that message (Doh! it is 15:07:08 + 30003 ms, which is when the next packet would have been sent.).
a-denoyelle commentedon Oct 13, 2026
Just to be sure, the network dump only show traffic from haproxy to the client. Is the client never acknowledges it or did you simply remove the packets from the dump ? I expect the latter as haproxy should reemit content if no ACK is received.
If you have only a single connection, you can enable QUIC traces at the developer level which should give precise information about what happens here. One method to enable them is to use the following startup argument
-dt quic:developer:clean.cmason3 commentedon Oct 13, 2026
Yeh sorry.. I omitted the client to server packets for brevity... please find attached the requested outputs - there are two requests - the first is to
/logs(standard html page request), which then makes a call to/_logsas the event source.quic-debug.txt
a-denoyelle commentedon Oct 14, 2026
I forgot that traces on stderr do not have timestamp attached so this renders the debugging complicated. However, I think there is an issue with our QUIC stack and the idle timer management if the client only emits ACK, which is the case you encounter here.
a-denoyelle commentedon Oct 14, 2026
Here is a patch which should fix your issue. If you can, do not hesitate to apply and test it.
0001-BUG-MINOR-quic-rearm-packet-on-packet-received-even-.patch
cmason3 commentedon Oct 14, 2026
Ok.. that took a little longer to test, as I run HAProxy from containers and had to tweak the Dockerfile to include the patch. Running the patch on top of 3.4.6 and it has been running for 10 minutes without timing out, so that looks like it has fixed the issue. Thanks! :)
wtarreau commentedon Oct 14, 2026
BTW since you're using server-sent events, be aware that in 3.5-dev9 and 3.4.9, we've implemented their detection to automatically enable http-no-delay, which avoids the in-kernel buffering latency on TCP (40-200ms pauses typically). This alone is a good reason to update as you will definitely experience a much better interactivity since you don't have the option in your conf. As a short-term improvement (before updating) you can also simply add "option http-no-delay" in your frontend or backend.
cmason3 commentedon Oct 14, 2026
Thanks, that is good to know.. I always update as soon as the new stable version is out :)
BUG/MINOR: quic: rearm packet on packet received even non ack-eliciting