👟 Reproduction steps

1. Self-hosted Appwrite with the official docker-compose.yml (we measured 2.4.0, the code is unchanged in 2.5.0), API on two nodes, Redis on a third.
2. Put steady load on the API: 400 req/s GET of a row plus 100 req/s of a session-authenticated read, split over both API nodes, 180 s, client timeout 15 s.
3. After 30 s make Redis stop answering for 60 s, without closing the connection: docker pause appwrite-redis (this is what a network blackhole, a frozen VM or a very long blocking command looks like to the API). Then docker unpause appwrite-redis.
4. For comparison, the same with docker stop appwrite-redis / docker start appwrite-redis.
5. While Redis is paused, call GET /v1/health/version on an API node.

👍 Expected behavior

A Redis that does not answer produces a quick error after a bounded time (1–2 s), written to the log, and requests that do not need Redis keep working. The node recovers as soon as Redis does.

👎 Actual Behavior

With docker pause, requests hang for as long as Redis is silent. After ≈ 4 s each API node has 1 000 requests in flight, and from then on Swoole rejects every new request immediately with 503 and an empty body, including /v1/health/version. Nothing is written to the API log for these rejections. This lasts for the whole 60 s; the node recovers ≈ 2 s after the unpause.

Two runs per variant, outage window s 30–94, g_row (400 req/s, both nodes together):

The session-authenticated read behaves the same way. With stop, Redis refuses the connection, every request retries ("Redis at redis:6379 after 5 attempts: Connection refused", 42 020 log lines in 6 minutes) and still about 60 % succeed, but slowly. After the start Redis loads its RDB file and answers LOADING for 2–7 s, which is also 503.

Cause

• Every Redis connection created in app/init/registers.php L224–L228 is opened with pconnect($host, $port) and no connect timeout argument, and then gets setOption(\Redis::OPT_READ_TIMEOUT, -1), which means: wait forever for a reply.
• The server caps in-flight requests at max_concurrency = 1_000 (packages/http/src/Adapter/Swoole/Mode.php L106). Requests that wait for Redis never finish, so the cap is reached within seconds, and Swoole answers everything above it with 503 before Appwrite code runs. That is why even /v1/health/version fails and why there is no log line.

Same 503 path without Redis: one client with many connections

The 503-without-log behaviour is also reached by load alone. In a 60-minute soak (≈ 390 req/s of normal traffic on two nodes) we opened 1 500 permanently open connections from one client against one node, twice for 60 s. That node answered ≈ 90 % of all requests with 503 (minute 20: 32 054 × 200 / 294 344 × 503; minute 40: 30 747 / 277 041), at ≈ 5 300 responses/s. In the same time ≈ 85 % of the normal requests of all other clients on that node were rejected too. This is the same max_concurrency limit. It is intended protection, but the rejection is silent and not attributable to a client; together with the missing read timeout it turns any slow dependency into a full outage of the node.

🎲 Appwrite version

2.5.0 (measured on 2.4.0, code unchanged in 2.5.0)

💻 Operating system

Linux (Debian 12), Docker

🧱 Your Environment

Self-hosted Appwrite 2.4.0, Postgres adapter, Redis 7.4 from the official compose file, two API nodes (Swoole) and one data node, docker pause/unpause and stop/start for the outage.

💡 Suggested fix

• Set a read timeout and a connect timeout on the Redis connections (for example 1–2 s) instead of OPT_READ_TIMEOUT = -1, so a silent Redis becomes a fast error. Long blocking calls (BRPOP, pub/sub subscribe) would need their own larger timeout, or the same value plus their own wait time.
• Log the resulting Redis error with the request path, so that the outage is visible.
• Log (rate limited) and count when max_concurrency rejects requests, for example one line per second with the current in-flight number, and expose the counter in the health endpoints or telemetry. Today these 503s leave no trace.
• Optionally let operators set max_concurrency via an environment variable, and document that a load balancer should limit connections per client.

👀 Have you spent some time to check if this issue has been raised before?

• I checked and didn't find a similar issue

🏢 Have you read the Code of Conduct?

• I have read the Code of Conduct