feat(stats-resources): spread project counts across the interval with the scheduler by loks0n · Pull Request #13669 · appwrite/appwrite · GitHub
Skip to content

feat(stats-resources): spread project counts across the interval with the scheduler - #13669

Merged
loks0n merged 5 commits into
mainfrom
claude/stats-resources-utopia-scheduler-1d1c1e
Sep 15, 2026
Merged

loks0n merged 5 commits into
mainfrom
claude/stats-resources-utopia-scheduler-1d1c1e

Conversation

@loks0n

@loks0n loks0n commented Sep 14, 2026 •

Copy link
Copy Markdown
Member

What does this PR do?

The stats-resources task enqueued every active project in the region at the top of every interval, so the worker (maxCoroutines: 1) got an hourly burst and sat idle the rest of the hour.

It now runs on Utopia\Schedule\Scheduler, like ScheduleFunctions, ScheduleExecutions and ScheduleMessages:

  • New Appwrite\Schedule\Source\Stats lists a concurrency row plus every project accessed in the last day in this region. Each project gets a Shifted(Interval) trigger with an offset derived from its id, the same spread Source\Functions uses, so its slot is stable across restarts and a region's counts spread evenly across the interval.
  • Incremental syncs use the (region, accessedAt) index every minute (or every interval when it is shorter, for test stacks); a full snapshot runs once per interval. Platform DB read cost stays at today's level.
  • The payload carries only the project id, since the worker reloads the document. The project listing skips the project subqueries with skipFilters inside the source, as Source\Database does, instead of disableSubqueries(), which mutated the global filter set in the combined scheduler process. A protected subqueries() hook on the task lets Cloud add its own filter names, the same shape as the spreadWindow hook on ScheduleFunctions.
  • Dispatch mirrors the sibling tasks: schedule.stats.{reconcile,enqueue,stopped} spans with occurrence.* attributes, and every failure is caught so one bad occurrence never ends the loop.
  • Concurrency sampling keeps its cadence (once per interval).

Test Plan

  • Unit tests for tests/unit/Schedule/Source pass; the E2E usage suites exercise the task end to end.
  • Pint, PHPStan and Rector pass on the changed files.

Related PRs and Issues

Cloud follow-up: appwrite-labs/cloud#5829 overrides the new subqueries() hook and re-points its composer.lock once this merges.

Checklist

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Sep 14, 2026 •

Copy link
Copy Markdown

@greptile-apps

greptile-apps Bot commented Sep 14, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

The PR appears safe to merge; no outstanding correctness, security, or repository-rule failures remain.

Summary

Refactors stats-resource collection to use the shared scheduler, spreading active-project counts across each configured interval while preserving regional and activity-window filtering.

  • Adds a scheduler source for concurrency and recently accessed projects.
  • Dispatches each due occurrence independently with telemetry and failure containment.
  • Avoids mutating global database filters in the combined scheduler process.
  • Documents the revised interval behavior.

Reviews (6) · Last reviewed commit: "test(stats-resources): drop the source u..."

Comment thread tests/unit/Schedule/Source/StatsTest.php Outdated
@loks0n loks0n self-assigned this Sep 14, 2026
@loks0n

loks0n commented Sep 14, 2026

Copy link
Copy Markdown
Member Author

@loks0n
loks0n force-pushed the claude/stats-resources-utopia-scheduler-1d1c1e branch from 6ad1d9a to 16a8a45 Compare September 14, 2026 19:56
@loks0n
loks0n enabled auto-merge September 14, 2026 22:05
loks0n and others added 5 commits September 15, 2026 10:24
… the scheduler

Replace the hourly Console::loop that enqueued every active project at once
with a Utopia\Schedule\Scheduler over a Stats source. Each project takes a
slot derived from its id inside the interval, as scheduled functions do, so
a region's resource counts spread across the hour instead of bursting at
the top of it. Concurrency sampling keeps its cadence as a row of the same
source.

Incremental syncs read the accessedAt index every minute and a full
snapshot runs once per interval, so platform DB load stays where it was.
Project sub-queries are skipped in the source instead of disabled globally.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Only the id is read, so no filter needs to run. Skipping them all rather
than APP_PROJECTS_SUBQUERIES also covers subqueries Cloud registers on
projects, which is the one thing Cloud overrode on this task.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ouble

The double now answers region, accessedAt, order, limit and cursor queries,
so the tests prove stale and out-of-region projects are excluded, since()
honours its cutoff, and paging covers every project.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…or more

Disabling every filter also skipped the JSON decode of collection metadata,
and the shared collection cache then handed HTTP a string where it expected
attributes. Skip the named project subqueries as Source\Database does, and
let an edition extend the list through a protected hook, as ScheduleFunctions
does for spreadWindow.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@loks0n
loks0n force-pushed the claude/stats-resources-utopia-scheduler-1d1c1e branch from 16a8a45 to 24530f0 Compare September 15, 2026 09:24
@loks0n
loks0n disabled auto-merge September 15, 2026 10:02
@loks0n
loks0n merged commit 16a8456 into main Sep 15, 2026
71 of 89 checks passed
abnegate added a commit that referenced this pull request Sep 22, 2026
Re-apply the stats-calculations/stats-events split on top of #13669's
scheduler-based task and #13814's queue-5 worker configuration: the renamed
Tasks\StatsCalculations keeps main's Utopia\Schedule\Scheduler body and
subqueries hook, Schedule.php passes telemetry through, and workers.php uses
the new coroutines key for both queues.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant