feat(alerting): per-rule/per-host alert routing (plan 30) by duyet · Pull Request #2269 · chmonitor/chmonitor · GitHub
Skip to content

feat(alerting): per-rule/per-host alert routing (plan 30) - #2269

Merged
duyet merged 2 commits into
mainfrom
advisor/30-per-rule-alert-routing
Jul 3, 2026
Merged

feat(alerting): per-rule/per-host alert routing (plan 30)#2269
duyet merged 2 commits into
mainfrom
advisor/30-per-rule-alert-routing

Conversation

@duyet

@duyet duyet commented Jul 3, 2026

Copy link
Copy Markdown
Member

Plan 30 — per-rule / per-host alert routing (reconciled onto latest main)

Implements per-rule/per-host alert routing (plans/30-per-rule-alert-routing.md) and merges it cleanly onto the current main, which had already absorbed the sibling alerting work (compound rules, Opsgenie adapter, custom rules, remediation links, etc.).

What routing adds

Operators can define routes (D1 table alert_routes) that match a rule id/type and/or host (glob or *) to one or more channel webhook URLs. Each notifying finding fans out to every matched channel; when nothing matches it falls back to the legacy global HEALTH_ALERT_WEBHOOK_URL — so deployments that never configure a route behave exactly as before. Routing is best-effort/fail-open (D1 unavailable → [] → legacy fallback).

How the merge was reconciled (server-sweep.ts)

The only real conflict in this file was plan 30 (routing fan-out) × plan 31 (compound rules). Plan 31 had refactored the inline dispatch into a shared dispatchFinding helper; plan 30 (which predated that extraction) added fan-out inline at the old call site.

Resolution:

  • Fold the fan-out into dispatchFinding so base rules and compound rules route identically. The helper gained a ruleType param and, inside the decision.notify branch, calls resolveTargets(routes, {ruleId, ruleType, hostId, hostName}, settings.webhookUrl) and posts to every returned channel.
  • Dispatch ordering / interaction: dedup (evaluateAlert) runs exactly once per finding, above the fan-out — so cooldown state is never multiplied, only the destination set is. The decision.notify gate wraps the entire fan-out, so a finding that is deduped/suppressed reaches zero channels (routed or not). Per-channel audit rows use detectAdapter(url).id, so a fan-out across mixed Slack/Discord/Opsgenie destinations is labelled per its own adapter (composes with plan 26). commit() fires when there was nothing to deliver or at least one channel succeeded; a total-failure delivery leaves no record so the next sweep retries.
  • Call sites pass ruleType: rule.type (base rules) and ruleType: 'compound' (compound rules).
  • Migration collision: renamed 0014_alert_routes.sql0015_alert_routes.sql to clear the collision with main's 0014_weekly_reports.sql (wrangler applies migrations in filename order).

Note on the other sibling plans

Plans 26/28/29/32/33 did not touch server-sweep.ts on main (verified via git log); they live in adapters / rule-registry / UI / API and merged cleanly. The suppression concepts (maintenance windows, ACKs) do not gate this session-less cron dispatch path in the current tree, so nothing there needed reconciling here.

Verification

  • apps/dashboard build + tsc --noEmit: clean
  • bun test src/lib/health/ --isolate: 196/196 pass — fan-out (single route replaces legacy, two routes fan out with dedup-once, no-match fallback) and compound dispatch (throwing rule, dedup under own key, compound-on-compound) both exercised
  • biome lint: clean
  • pre-push hook (896 tests) passed without bypass

Closes plan 30.

duyet and others added 2 commits July 4, 2026 00:07
Lets an operator route health-sweep findings to different channel webhook
URLs by rule id/type and/or host (glob or *), fanning out to every matched
route and falling back to the legacy global HEALTH_ALERT_WEBHOOK_URL when
nothing matches (today's exact behavior for deployments with no routes
configured). Dedup still runs once per finding, not once per channel.

- alert_routes D1 migration (0014) + lib/health/alert-routing.ts: pure
  matchRoutes/resolveTargets core plus best-effort D1 CRUD, mirroring
  alert-history-store.ts's fail-open pattern.
- server-sweep.ts: loads routes once per sweep, fans out postWebhook +
  per-channel audit rows over resolveTargets, commits the dedup decision
  exactly once per finding.
- /api/v1/health/routes (GET/POST/DELETE): owner-scoped, fails open to the
  OSS single-tenant owner ('') when Clerk isn't configured, gates writes
  behind sign-in only in cloud mode (lib/health/alert-routing-auth.ts).
- Routing tab in HealthSettingsDialog (alert-routing-dialog.tsx) to
  list/create/delete routes and send a test alert per route.

Co-Authored-By: duyetbot <bot@duyet.net>
Reconcile plan 30 (per-rule/per-host alert routing) with the sibling
alerting work already merged to main. In server-sweep.ts the only real
conflict was plan 30 (routing fan-out) x plan 31 (compound rules, which
had refactored the inline dispatch into a shared `dispatchFinding`
helper).

Resolution:
- Fold plan 30's route fan-out INTO main's `dispatchFinding` helper so
  both base rules and compound rules route identically. The helper now
  takes a `ruleType` param and, inside the `decision.notify` branch,
  calls `resolveTargets(routes, {ruleId, ruleType, hostId, hostName})`
  and posts to every matched channel, falling back to the legacy global
  webhook when nothing matches. Per-channel audit rows use
  `detectAdapter(url).id` (composes with plan 26's Opsgenie adapter).
- Dedup (`evaluateAlert`) still runs exactly ONCE per finding, above the
  fan-out, so cooldown state is never multiplied; the notify decision
  gates the whole fan-out (a suppressed finding reaches no channel).
- Deleted the leftover inline routing block that git could not place
  (plan 30 predated the helper extraction).
- Pass `ruleType: rule.type` at the base-rule call site and
  `ruleType: 'compound'` at the compound-rule call site.
- Renamed migration 0014_alert_routes.sql -> 0015_alert_routes.sql to
  clear the collision with main's 0014_weekly_reports.sql (wrangler
  applies migrations in filename order).

Verified: apps/dashboard build + tsc clean; `bun test src/lib/health/`
196/196 pass (fan-out single/double-route/dedup-once, no-match fallback,
compound throwing/dedup/compound-on-compound); biome lint clean.

Co-Authored-By: duyetbot <bot@duyet.net>
@github-actions github-actions Bot added the app: dashboard Changes to the apps/dashboard TanStack Start app label Jul 3, 2026
@github-actions

github-actions Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

app: dashboard Changes to the apps/dashboard TanStack Start app

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant