🐛 Bug Report: SSR runtime never clears its per-request timeout timer, so memory grows with requests/s × site timeout · Issue #14474 · appwrite/appwrite · GitHub
You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
🐛 Bug Report: SSR runtime never clears its per-request timeout timer, so memory grows with requests/s × site timeout #14474
Deploy a Next.js (or any Node SSR) site with adapter ssr, runtime node-24, on a self-hosted Appwrite 2.5.0. We used Next.js 15.5.4 with output: 'standalone' and a server-rendered page of 29 KB.
Set the site timeout to 30 s. Delete the runtime container through the executor so it starts fresh.
Send a constant 150 requests/s to the page for 2 minutes (for example oha -z 120s -q 75 -c 60 against each of two API nodes) and sample the RSS of the runtime container every 2 s (docker stats).
Repeat with timeout 5 s and 120 s. Stop the load and keep sampling.
👍 Expected behavior
Memory of the runtime depends on the number of requests in flight (a few MB), not on the site timeout. After the load stops, per-request memory is released right away.
👎 Actual Behavior
RSS of the runtime container follows requests/s × timeout × ≈ 70–130 KB. Same load (150 requests/s, all 200), only the site timeout changed:
Site timeout
RSS after 30 s / 60 s / 95 s / 127 s of load
Shape
After the load stops
5 s
392 / 417 / 430 MB
plateau
back to 67 MB after ≈ 40 s
30 s
694 / 759 / 745 / 724 MB
plateau from ≈ 30 s on (700–790 MB)
RSS stays at 751 MB (V8 does not return it), heapUsed drops from ≈ 290 to 148 MB
120 s (run 1)
705 / 1,037 / 1,385 / 1,674 MB
linear, ≈ 10–12 MB/s, no plateau
1.5–1.7 GB until the timeout expires, then 97–557 MB
120 s (run 2)
696 / 1,134 / 1,529 / 1,773 MB
same
same
The plateau is reached when the first timers start to fire, i.e. after timeout seconds. With 30 s that was 3 of 3 runs; with 120 s, 2 of 2 runs show linear growth within the observed window. process.memoryUsage().external (response buffers) rose to 850–876 MB at 120 s.
If the container has a memory limit (the spec s-1vcpu-512mb gives 512 MB), the 30 s plateau is reached at roughly 100 req/s of such a page (≈ 100 MB base + ≈ 130 KB × rps × 30 s), after which the kernel OOM-kills the runtime. Our test cluster has _APP_COMPUTE_MEMORY=0 (no limit), so we did not observe the kill itself in this series, only the growth. In a separate check, a runtime container killed by the kernel OOM killer (OOMKilled: true) was restarted by Docker (restartPolicy: always) and served requests again; while the memory stayed too small it went into a restart loop and requests in that window failed. So under sustained load above the limit the site keeps getting killed and restarted, and all in-flight requests of each killed process are lost.
Cause (open-runtimes SSR wrapper shared by the Node, Bun and Deno runtimes): for every request that carries x-open-runtimes-timeout (the executor always sends it), the wrapper starts a setTimeout and never clears it, even when the response was long finished (http-injection.mjs L60–L70):
The callback closes over req, res and the request scope, so the timer keeps them (and, with them, buffered response data) alive until it fires. The executor sends the header at Docker.php L958 with the site's timeout (default 30 s, up to 900 s), so a site with a long timeout pays more memory per request than one with a short timeout. The image on our cluster (openruntimes/node:v5-24) contains this file unchanged (setTimeout at line 61, no clearTimeout).
We did not run the runtime with a patched wrapper, so the link between this timer and the growth rests on code reading and on the match of the measured curve (plateau at timeout, release after the timeout, linear growth for 120 s). The Node function server has the same pattern for function timeouts (server.js L308–L312); there the closure is small and we did not measure it.
Related but different: #14419 (Bun runtime, memory growth with function logging enabled).
🎲 Appwrite version
2.5.0 (executor 1.1.2, openruntimes/node:v5-24). The code is the same on open-runtimes main (da714e7).
💻 Operating system
Linux (Debian 12), Docker
🧱 Your Environment
Self-hosted, two API nodes behind a load balancer, executor and runtimes on one 4 vCPU node. Load generator on a separate machine. A single runtime container received all 150 requests/s.
💡 Suggested fix
Keep the timer handle and clear it as soon as the response is done:
close fires for finished, aborted and errored responses, so no path leaves the timer behind. The same for the function server in server.js: clearTimeout after Promise.race returns.
Trade-off: none functionally, the timer only matters while the response is open. timer.unref() alone would not help, it does not release the closure.
Until then, users can lower the site timeout: the plateau shrinks proportionally (30 s → 5 s: ≈ 750 → ≈ 430 MB at 150 requests/s).
👀 Have you spent some time to check if this issue has been raised before?
Activity