Upgrade queue to 4 by loks0n · Pull Request #13809 · appwrite/appwrite · GitHub
Skip to content

Upgrade queue to 4 - #13809

Merged
loks0n merged 2 commits into
mainfrom
codex/queue-4
Sep 21, 2026
Merged

loks0n merged 2 commits into
mainfrom
codex/queue-4

Conversation

@loks0n

@loks0n loks0n commented Sep 21, 2026 •

Copy link
Copy Markdown
Member

Upgrade utopia-php/queue to 4.0.0 for the unified batch receive API and Redis claim recovery, and Platform to 1.0.0-rc22 to allow queue 4. The lockfile changes only these two packages.

Workers now create a dedicated receive connection and a separate locked command connection for each consumer. With the old shared publisher pool, an idle blocking receive could occupy the only resource and prevent maintenance from recovering stranded jobs. The new connections use the existing Redis host, port, username, and password settings. Update the two queue test doubles to the new interfaces.

Validation

With the released dependencies installed:

  • Composer validation passed.
  • Worker initialization: 2 tests, 5 assertions.
  • Swoole worker concurrency: 4 tests, 12 assertions, including combined queues and serialized database jobs.
  • Pint and PHPStan passed for all changed PHP files.

Real Redis 7 with ACL username/password: exercised the production consumer factory and unmodified Swoole maintenance scheduler. An expired claim older than the default 90,000-second recovery guard was recovered during an otherwise idle blocking receive after 30.07 seconds; exactly one message handled, no errors, processing list empty.

The recovery age remains the library default (25 hours since publication). Each active queue consumer uses two dedicated Redis connections in addition to publisher pool usage.

Platform prerequisite: utopia-php/monorepo#314, released in https://github.com/utopia-php/platform/releases/tag/1.0.0-rc22.

@loks0n
loks0n marked this pull request as ready for review September 21, 2026 15:08
@greptile-apps

greptile-apps Bot commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

Comment thread app/worker.php
Comment on lines +94 to +96
return new Redis(
receive: new Connection(...$connection),
commands: new Locking(new Connection(...$connection)),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Recovery Behavior Is Untested

This changes the production connection lifecycle so blocking receives use a separate connection from heartbeat and recovery commands, but no observable regression test covers that behavior. The changed unit-test fake only mirrors Queue v4's Consumer::receive() signature and returns an empty batch, while the existing concurrency tests reuse one in-memory connection for both broker roles and bypass this factory. This violates the repository directive to test observable behavior instead of mirroring source interfaces. Before merging, add an integration or E2E test that blocks receive and verifies heartbeat or expired-claim recovery through the separate command connection; otherwise the stranded-job failure targeted by this change can return undetected.

Context Used: Call out and harshly judge implementation-coupled tests. We don't mirror source code, configuration, or version pins in assertions. We test observable behavior; use linters for syntax and schema checks. (source)

Prompt To Fix With AI
This is a comment left during a code review.
Path: app/worker.php
Line: 94-96

Comment:
**Recovery Behavior Is Untested**

This changes the production connection lifecycle so blocking receives use a separate connection from heartbeat and recovery commands, but no observable regression test covers that behavior. The changed unit-test fake only mirrors Queue v4's `Consumer::receive()` signature and returns an empty batch, while the existing concurrency tests reuse one in-memory connection for both broker roles and bypass this factory. This violates the repository directive to test observable behavior instead of mirroring source interfaces. Before merging, add an integration or E2E test that blocks receive and verifies heartbeat or expired-claim recovery through the separate command connection; otherwise the stranded-job failure targeted by this change can return undetected.

**Context Used:** Call out and harshly judge implementation-coupled tests. We don't mirror source code, configuration, or version pins in assertions. We test observable behavior; use linters for syntax and schema checks. ([source](https://app.greptile.com/review/custom-context?memory=instruction-0))

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Fix in Claude Code Fix in Codex

@github-actions

Copy link
Copy Markdown

✨ Benchmark results

Comparing main (before) → codex/queue-4 (after).

Metric Before After Change
🚀 Requests/sec 189.36 195.22 ⚪ +3.1%
⏱️ Latency P50 90.35 ms 89.36 ms ⚪ -1.1%
⏱️ Latency P95 217.09 ms 205.17 ms 🟢 -5.5%
Per-scenario breakdown & investigation details

Metrics below reflect the current branch (after). Δ P95 compares against the base.

Scenario P50 (ms) P95 (ms) Requests RPS Δ P95 (ms)
API total 89.36 205.17 12,312 195.22 -11.93
Account 165.34 322.17 648 10.78 -5.2
TablesDB 87.12 166.42 6,696 108.63 -11.55
Storage 81.3 174.63 3,240 53.97 -5.71
Functions 122.98 243.2 1,728 29.53 -3.08

Top API waits (after)

API request Max wait (ms)
account.name.update 644.75
account.prefs.update 422.95
tablesdb.rows.delete 412.66
functions.variables.update 405.08
functions.create 401.84

@loks0n
loks0n merged commit f2f494f into main Sep 21, 2026
56 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant