Reviewer cache monitoring: in-run emit + scheduled collector by kacper-mikolajczak · Pull Request #57 · callstack-internal/Expensify-App · GitHub
Skip to content

Reviewer cache monitoring: in-run emit + scheduled collector#57

Draft
kacper-mikolajczak wants to merge 2 commits into
reviewer-cache-prefixfrom
reviewer-cache-monitoring
Draft

Reviewer cache monitoring: in-run emit + scheduled collector#57
kacper-mikolajczak wants to merge 2 commits into
reviewer-cache-prefixfrom
reviewer-cache-monitoring

Conversation

@kacper-mikolajczak

@kacper-mikolajczak kacper-mikolajczak commented Jun 22, 2026

Copy link
Copy Markdown

Stacked on the cache-prefix change so this diff is just the monitoring. Split into two parts so analysis/alerting lives out of the review hot path.

1. In-run (minimal) — emit-cache-usage.sh

A tiny step in the reviewer workflow that parses the action's execution_file (always written, no show_full_output) and prints one counts-only line to the log:

CACHE_USAGE creation=<n> read=<n> cost=<n>

No message content → safe in public logs. No analysis, no threshold, no failure.

2. Out-of-band collector — reviewer-cache-monitor.yml + collect-cache-health.sh

A scheduled workflow (hourly + manual dispatch) that:

  • lists recent claude-review.yml runs, reads their logs (only — no artifacts/content), greps the CACHE_USAGE line,
  • builds a per-run warm-ratio table in the step summary,
  • computes the cold fraction over the sample and fails the run (red cron = built-in alert) if it exceeds a threshold.

Tunables: LOOKBACK (30), CACHE_CREATION_WARN_TOKENS (15K), COLD_ALERT_PCT (40), MIN_SAMPLE (10).

Why this shape

  • Hot path stays minimal — one echo per review, no per-run analysis.
  • Trend-aware — distinguishes a one-off cold start (normal, first run after an idle gap) from a sustained regression (rules edit, action/CLI bump, model change, lost 1h TTL). A per-run check can't.
  • Logs-only — the collector needs no artifacts and no message content; it just scans the emitted line.
  • Non-intrusive alert — a regression turns the scheduled run red (and surfaces in GitHub's failed-workflow notifications) instead of blocking PRs.

Validated locally: emit produces the line for warm (creation 1.4K / read 333K) and cold (creation 64K) execution files; collector parse + threshold logic checked; both workflows parse as valid YAML.

@coderabbitai

coderabbitai Bot commented Jun 22, 2026

Copy link
Copy Markdown

@kacper-mikolajczak kacper-mikolajczak changed the title Reviewer prompt-cache health monitoring + cold-cache regression warning Reviewer cache monitoring: in-run emit + scheduled collector Jun 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant