Why

The AI review has no measure of its quality. A prompt change, a new provider, or a new model ships on a hunch. This PR adds a fixed set of review cases with expected findings, and a runner that scores a live model against them, so a change to the review can be compared by numbers before a rollout.

What

• backend/plugin/aireview/testdata/cases/: ten cases, five for PostgreSQL and five for MySQL, the launch engines. Each case is one YAML file: the statement, the synced schema behind the read and search tools, the target facts, the policy text, and the expectation.
• Each case turns on a fact only the schema, the target, or the policy holds, never the statement:
  • postgres/index-large-table and postgres/index-small-table: the same CREATE INDEX, 48M rows against 2,400. Only the large one must be flagged.
  • postgres/drop-column-read-by-view and mysql/drop-column-read-by-view: the dropped column feeds a view.
  • postgres/delete-cascades and mysql/delete-cascades: a delete cascades through two tables.
  • postgres/policy-foreign-key-index: the project policy wants an index on every foreign key.
  • mysql/modify-column-large-table and mysql/add-column-instant: a column change that copies 61M rows, and one that is instant on the same table.
  • mysql/policy-table-charset: the workspace policy wants utf8mb4, and the new table inherits latin1.
• model_ai_cases_test.go:
  • TestReviewCases runs every case through the real Reviewer, the real catalog tools, and a live model, then logs one row per case and a total. It runs only when AIREVIEW_CASES_PROVIDER and AIREVIEW_CASES_API_KEY are set, since it needs a vendor key.
  • TestReviewCasesLoad checks every case file without a model and runs in CI. The schema is decoded strictly, so a typo in a fixture fails the load instead of vanishing.
  • A required finding matches on its line and a keyword at the start of a word in its title or evidence. Each one needs a finding of its own. A case is pass, noisy (a forbidden finding was reported), miss, or error, and only miss and error fail the test.
  • The provider's error body passes through a wrapper that removes the key before anything logs it.
• testdata/cases/README.md: the case format, the scoring, how to add a case, and how to run it per provider.

Live results

All three providers found every required finding (8 of 8):

• Gemini gemini-3.5-flash: 9 pass, 1 noisy (index-small-table, suggested CONCURRENTLY on 2,400 rows), 105k tokens.
• OpenAI gpt-5.6-terra: 10 pass, 55k tokens.
• Claude claude-sonnet-5: 9 pass, 1 noisy (index-small-table), 93k tokens.

The noisy case moves between runs. An earlier run had Claude's false flag on add-column-instant instead.

Not in this PR

• Azure OpenAI. The runner covers gemini, openai, and claude.
• Reasoning quality. Scoring checks that the deciding fact was named, not that the explanation is right.
• More engines. Cases for an engine come with its rollout.

Tests

• go test -count=1 -race ./backend/plugin/aireview/: passes, and TestReviewCases skips without the env.
• TestScoreFindings, TestScoreFindingsCountsAFindingForOneEntry, and TestRedactingModelRemovesTheKeyFromAnErrorBody cover the scorer and the key redaction.
• golangci-lint at baseline, server build ok.

🤖 Generated with Claude Code