🐛 Bug Report: Static sites check a bcrypt password on every request (~48 ms CPU each, ~75 req/s per cluster) · Issue #14473 · appwrite/appwrite · GitHub
You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
🐛 Bug Report: Static sites check a bcrypt password on every request (~48 ms CPU each, ~75 req/s per cluster) #14473
Deploy a static site (any Vite SPA build with a few dozen small assets, adapter static) on a self-hosted Appwrite 2.5.0.
Load-test one small asset (4 KB JS file) through the router with 20, 80 and 160 connections, for example oha -z 20s -c 80 https://<site-domain>/assets/index-<hash>.js. Watch docker stats of the openruntimes/static:v5-1 runtime container while it runs.
To see the cost in isolation, send requests directly to the runtime container (from another container in the same Docker network, port 3000) and compare the response time of a request without Authorization header and one with a wrong Basic password. Both are rejected with 401.
👍 Expected behavior
A 4 KB static file is served at several hundred requests per second per CPU core, like a plain static-web-server. The API router and the executor are not the bottleneck for static assets.
👎 Actual Behavior
The whole cluster serves about 75 requests/s of a 4 KB asset, no matter how many connections are open. The static runtime container uses 2.1–2.3 CPU cores while the API, executor and database are almost idle.
Connections
Requests/s (2 runs)
p50
p99
Static runtime CPU (avg)
API node CPU (avg)
20
72.2 / 71.0
278 / 282 ms
474 / 491 ms
214 / 211 %
11–12 %
80
74.5 / 75.8
985 / 1,055 ms
2.8 / 2.6 s
219 / 222 %
11–12 %
160
77.4 / 76.6
1.96 / 2.22 s
7.9 / 5.5 s
225 / 229 %
11–15 %
All responses were 200, 6 of 6 runs. Latency grows linearly with the queue (c / 75 s).
For comparison, the same cluster executes 660–800 function executions/s at 80 connections, so a static asset is ≈ 10× slower than a function execution.
Direct measurement against the runtime (40 requests, median): no Authorization header → 401 after 0.10 ms; wrong password → 401 after 47.8 ms. That is the cost of one bcrypt verification. 2.2 cores / 75 requests/s ≈ 30 ms CPU per request, which matches.
A warm 4 KB asset takes ≈ 62 ms end to end, a trivial SSR route on the same cluster ≈ 31 ms. A page with 40 assets occupies the cluster for ≈ 0.5 s (≈ 2 page views/s).
Cause (open-runtimes, not Appwrite's own code):
The executor generates a random runtime secret per runtime (controllers.php L74, passed to the container as OPEN_RUNTIMES_SECRET at L92) and sends it on every request both as Authorization: Basic base64(opr:<secret>) and as x-open-runtimes-secret (Docker.php L955–L956).
The static runtime turns the secret into an htpasswd bcrypt entry with cost 10 at start-up (hooks/server.sh L16: htpasswd -inBC10 "opr") and hands it to static-web-server --basic-auth (L110). static-web-server verifies the hash on every request. The Node, Bun and Deno runtimes compare the header x-open-runtimes-secret with a plain string instead (system.mjs L29–L43).
The secret is 32 random hex characters (128 bit) and only lives inside the Docker network, so a slow password hash protects nothing here: brute-forcing a 128-bit random value is not feasible regardless of the hash cost.
The runtime images on our cluster (openruntimes/static:v5-1, executor 1.1.2) contain exactly this script (same lines). In addition, every site request causes one ClickHouse INSERT into executions (about 1,400 inserts in a 20 s window at 70 requests/s, no batching); we mention it only because it is a second per-asset cost.
🎲 Appwrite version
2.5.0 (executor 1.1.2, openruntimes/static:v5-1, static-web-server 2.39.0). The code is the same on open-runtimes main (da714e7).
💻 Operating system
Linux (Debian 12), Docker
🧱 Your Environment
Self-hosted, two API nodes behind a load balancer, executor and runtimes on one 4 vCPU node, Postgres/Redis/ClickHouse on a separate node. _APP_COMPUTE_MEMORY=0. Load generator on a separate machine, 20 s per run, 2 runs per level.
💡 Suggested fix
Any of these removes the bcrypt cost from the request path; they differ in how much they change:
Lower the bcrypt cost in server.sh (htpasswd -inBC4 instead of -C10). One line. Cost is exponential, so cost 4 is 64× cheaper (we estimate ≈ 0.5–1 ms per request; not measured). It still verifies a hash per request, but the secret is random and 128 bit, so the cost factor adds no security.
Use a cheap hash that static-web-server supports for --basic-auth (to be checked against its documentation; e.g. apr1/SHA1 htpasswd entries) instead of bcrypt. Same effect, no per-request cost worth mentioning, but depends on what the server accepts.
Drop the Basic-Auth check inside static-web-server and check x-open-runtimes-secret like the other runtimes, in a small front process (or an advanced.headers/rewrite rule if the server can match a header). Cleanest, but changes the architecture of the static runtime and needs a second process.
A cache of successful verifications inside the server (as many HTTP servers do for Basic auth) is the best general fix, but must be implemented in static-web-server itself.
Option 1 is the smallest change and is enough to make the runtime stop being the bottleneck.
👀 Have you spent some time to check if this issue has been raised before?
Activity