add ai gateway phase headers by alex-m-brown · Pull Request #73491 · code-dot-org/code-dot-org · GitHub
Skip to content

add ai gateway phase headers#73491

Open
alex-m-brown wants to merge 2 commits into
stagingfrom
ai-gateway-observability/add-phase-header
Open

add ai gateway phase headers#73491
alex-m-brown wants to merge 2 commits into
stagingfrom
ai-gateway-observability/add-phase-header

Conversation

@alex-m-brown

@alex-m-brown alex-m-brown commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Sets an x-ai-gateway-phase header on each AI Gateway request from the client so the worker can tag Sentry transactions with which step of the AI Chat pipeline they belong to. Companion to the gateway-side change code-dot-org/ai-gateway#41 which reads the header and attaches it as a context.phase attribute on the root span.

Why

A single AI Chat user message produces three sequential gateway calls from the client:

  1. Input safety check (isTextSafe(userInput))
  2. Main model call (generateText({...}))
  3. Output safety check (isTextSafe(modelOutput))

Plus a separate /transcribe endpoint for voice-to-text. All four currently look identical in Sentry — same hostname, same span shape, no way to slice metrics by step. Adding a per-call phase header lets dashboards and alerts distinguish them:

  • input_filter — user-input moderation
  • generation — main model call
  • output_filter — model-output moderation
  • transcription — speech-to-text

What changed

  • apps/src/aiGateway/generateText.ts — adds an optional phase field to the gateway client API. When set, the value is sent as the x-ai-gateway-phase header.
  • apps/src/aichat/api/client/helpers/safetyHelpers.tsisTextSafe now takes a required phase argument ('input_filter' or 'output_filter') and passes it through to generateText.
  • apps/src/aichat/api/client/generateChatResponse.ts — passes the appropriate phase at all three gateway call sites in the AI Chat flow.
  • apps/src/aiGateway/transcribe.ts — hardcodes 'transcription' as the phase on the single transcribe call site (no API change since the value is invariant).

Links

Testing story

  • Sent a chat message in the AI Chat lab. Confirmed three distinct gateway transactions in Sentry with context.phase tagged as input_filter, generation, and output_filter respectively.
Screenshot 2026-06-25 at 2 50 52 PM Screenshot 2026-06-25 at 2 52 31 PM Screenshot 2026-06-25 at 2 52 53 PM
  • Triggered the microphone (speech-to-text). Confirmed the resulting transcribe transaction shows context.phase: transcription.
Screenshot 2026-06-25 at 3 00 00 PM
  • Pre-existing behavior unchanged: profanity filtering still triggers USER_PROFANITY / MODEL_PROFANITY statuses; transcribe still returns the correct text.
Screenshot 2026-06-25 at 2 58 15 PM

Deployment notes

Must deploy after ai-gateway#41 ships to production. That PR allows x-ai-gateway-phase through the worker's CORS preflight. If this PR ships first, every AI request from the browser fails at the preflight stage because the worker's Access-Control-Allow-Headers doesn't include the new header.

Safe order:

  1. ai-gateway#41 merges and deploys → worker accepts the header (no behavior change yet, since nothing sends it).
  2. This PR merges → client starts sending the header → Sentry transactions get context.phase.

@alex-m-brown alex-m-brown changed the title add phase headers add ai gateway phase headers Jun 25, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, up until now we've been pretty intentional about making the signature of the AI Gateway APIs consistent with the vercel AI SDK APIs they wrap. I'm not sure if we should start breaking away just yet, especially if we're considering moving some of the safety calls over to the gateway side anyway. Let's discuss on Slack?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants