🐛 Bug Report: SSR runtime never clears its per-request timeout timer, so memory grows with requests/s × site timeout · Issue #14474 · appwrite/appwrite · GitHub
Skip to content

🐛 Bug Report: SSR runtime never clears its per-request timeout timer, so memory grows with requests/s × site timeout #14474

Open

Description

@JoshiJoshiJoshi

👟 Reproduction steps

  1. Deploy a Next.js (or any Node SSR) site with adapter ssr, runtime node-24, on a self-hosted Appwrite 2.5.0. We used Next.js 15.5.4 with output: 'standalone' and a server-rendered page of 29 KB.
  2. Set the site timeout to 30 s. Delete the runtime container through the executor so it starts fresh.
  3. Send a constant 150 requests/s to the page for 2 minutes (for example oha -z 120s -q 75 -c 60 against each of two API nodes) and sample the RSS of the runtime container every 2 s (docker stats).
  4. Repeat with timeout 5 s and 120 s. Stop the load and keep sampling.

👍 Expected behavior

Memory of the runtime depends on the number of requests in flight (a few MB), not on the site timeout. After the load stops, per-request memory is released right away.

👎 Actual Behavior

RSS of the runtime container follows requests/s × timeout × ≈ 70–130 KB. Same load (150 requests/s, all 200), only the site timeout changed:

Site timeout RSS after 30 s / 60 s / 95 s / 127 s of load Shape After the load stops
5 s 392 / 417 / 430 MB plateau back to 67 MB after ≈ 40 s
30 s 694 / 759 / 745 / 724 MB plateau from ≈ 30 s on (700–790 MB) RSS stays at 751 MB (V8 does not return it), heapUsed drops from ≈ 290 to 148 MB
120 s (run 1) 705 / 1,037 / 1,385 / 1,674 MB linear, ≈ 10–12 MB/s, no plateau 1.5–1.7 GB until the timeout expires, then 97–557 MB
120 s (run 2) 696 / 1,134 / 1,529 / 1,773 MB same same
  • The plateau is reached when the first timers start to fire, i.e. after timeout seconds. With 30 s that was 3 of 3 runs; with 120 s, 2 of 2 runs show linear growth within the observed window. process.memoryUsage().external (response buffers) rose to 850–876 MB at 120 s.
  • If the container has a memory limit (the spec s-1vcpu-512mb gives 512 MB), the 30 s plateau is reached at roughly 100 req/s of such a page (≈ 100 MB base + ≈ 130 KB × rps × 30 s), after which the kernel OOM-kills the runtime. Our test cluster has _APP_COMPUTE_MEMORY=0 (no limit), so we did not observe the kill itself in this series, only the growth. In a separate check, a runtime container killed by the kernel OOM killer (OOMKilled: true) was restarted by Docker (restartPolicy: always) and served requests again; while the memory stayed too small it went into a restart loop and requests in that window failed. So under sustained load above the limit the site keeps getting killed and restarted, and all in-flight requests of each killed process are lost.

Cause (open-runtimes SSR wrapper shared by the Node, Bun and Deno runtimes): for every request that carries x-open-runtimes-timeout (the executor always sends it), the wrapper starts a setTimeout and never clears it, even when the response was long finished (http-injection.mjs L60–L70):

if (req.safeTimeout !== null && req.safeTimeout !== undefined) {
  setTimeout(() => {
    if (!res.headersSent) { /* 500 "Execution timed out." */ }
  }, req.safeTimeout * 1000);
}

The callback closes over req, res and the request scope, so the timer keeps them (and, with them, buffered response data) alive until it fires. The executor sends the header at Docker.php L958 with the site's timeout (default 30 s, up to 900 s), so a site with a long timeout pays more memory per request than one with a short timeout. The image on our cluster (openruntimes/node:v5-24) contains this file unchanged (setTimeout at line 61, no clearTimeout).

We did not run the runtime with a patched wrapper, so the link between this timer and the growth rests on code reading and on the match of the measured curve (plateau at timeout, release after the timeout, linear growth for 120 s). The Node function server has the same pattern for function timeouts (server.js L308–L312); there the closure is small and we did not measure it.

Related but different: #14419 (Bun runtime, memory growth with function logging enabled).

🎲 Appwrite version

2.5.0 (executor 1.1.2, openruntimes/node:v5-24). The code is the same on open-runtimes main (da714e7).

💻 Operating system

Linux (Debian 12), Docker

🧱 Your Environment

Self-hosted, two API nodes behind a load balancer, executor and runtimes on one 4 vCPU node. Load generator on a separate machine. A single runtime container received all 150 requests/s.

💡 Suggested fix

  • Keep the timer handle and clear it as soon as the response is done:

    const timer = setTimeout(/* ... */, req.safeTimeout * 1000);
    res.once("close", () => clearTimeout(timer));

    close fires for finished, aborted and errored responses, so no path leaves the timer behind. The same for the function server in server.js: clearTimeout after Promise.race returns.

  • Trade-off: none functionally, the timer only matters while the response is open. timer.unref() alone would not help, it does not release the closure.

  • Until then, users can lower the site timeout: the plateau shrinks proportionally (30 s → 5 s: ≈ 750 → ≈ 430 MB at 150 requests/s).

👀 Have you spent some time to check if this issue has been raised before?

  • I checked and didn't find a similar issue

🏢 Have you read the Code of Conduct?

  • I have read the Code of Conduct

Activity

  1. added
    product / authFixes and upgrades for the Appwrite Auth / Users / Teams services.
    product / databasesFixes and upgrades for the Appwrite Database.
    product / functionsFixes and upgrades for the Appwrite Functions.
    product / self-hostedIssues only found when self-hosting Appwrite
    product / sitesFixes and upgrades for Appwrite Sites.
    product / storageFixes and upgrades for the Appwrite Storage.
    on Oct 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    product / authFixes and upgrades for the Appwrite Auth / Users / Teams services.product / databasesFixes and upgrades for the Appwrite Database.product / functionsFixes and upgrades for the Appwrite Functions.product / self-hostedIssues only found when self-hosting Appwriteproduct / sitesFixes and upgrades for Appwrite Sites.product / storageFixes and upgrades for the Appwrite Storage.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions