fix: harden chat lifecycle hook dispatch and admission - #27585
fix: harden chat lifecycle hook dispatch and admission#27585ibetitsmike wants to merge 10 commits into
Conversation
|
CI note: the two failing Storybook checks are a |
b02787d to
1a3e838
Compare
0ea6c99 to
83deb68
Compare
Documentation CheckRe-checked at Updates Needed
No emdash or endash in the changed prose, tables are internally aligned, and the Automated review via Coder Agents |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 83deb68130
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9bf20a4e66
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d6323543a8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 47d87ffbfb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
|
Codex Review: Didn't find any major issues. 🎉 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7a97da7d67
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 87e19b4061
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
|
Codex Review: Didn't find any major issues. Chef's kiss. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b1d926d8cb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
|
Codex Review: Didn't find any major issues. More of your lovely PRs please. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review |
|
Codex Review: Didn't find any major issues. Swish! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
jdomeracki-coder
left a comment
There was a problem hiding this comment.
Approving, clearly a major hardening improvement
nickvigilante
left a comment
There was a problem hiding this comment.
Docs LGTM with a few changes
| - `system_prompt` is the per-chat system prompt, present on the admission for a chat create or subagent spawn and immutable afterward. Subagent types that embed the submitted prompt at system priority (computer-use) carry the built value here, and a prompt override rebuilds it from the approved prompt. | ||
| - `custom_prompt` is the chat owner's stored custom prompt as it will be injected into the turn's system prompt. It's stored through its own API without a lifecycle event, so this field is where a policy sees it, on every prompt admission. The admitted value is snapshotted with the turn and injected verbatim at generation, so rewriting the stored prompt after admission can't put unreviewed instructions in front of an already-admitted turn. | ||
| - `dynamic_tools` carries the caller-declared tool definitions admitted with a chat create. Their names and descriptions reach the model as instructions. |
There was a problem hiding this comment.
| - `system_prompt` is the per-chat system prompt, present on the admission for a chat create or subagent spawn and immutable afterward. Subagent types that embed the submitted prompt at system priority (computer-use) carry the built value here, and a prompt override rebuilds it from the approved prompt. | |
| - `custom_prompt` is the chat owner's stored custom prompt as it will be injected into the turn's system prompt. It's stored through its own API without a lifecycle event, so this field is where a policy sees it, on every prompt admission. The admitted value is snapshotted with the turn and injected verbatim at generation, so rewriting the stored prompt after admission can't put unreviewed instructions in front of an already-admitted turn. | |
| - `dynamic_tools` carries the caller-declared tool definitions admitted with a chat create. Their names and descriptions reach the model as instructions. | |
| - `system_prompt` is the per-chat system prompt, present on the admission for a chat create or subagent spawn and immutable afterward. | |
| Subagent types that embed the submitted prompt at system priority (computer-use) carry the built value here, and a prompt override rebuilds it from the approved prompt. | |
| - `custom_prompt` is the chat owner's stored custom prompt as it will be injected into the turn's system prompt. | |
| It's stored through its own API without a lifecycle event, so this field is where a policy sees it, on every prompt admission. | |
| The admitted value is snapshotted with the turn and injected verbatim at generation, so rewriting the stored prompt after admission can't put unreviewed instructions in front of an already-admitted turn. | |
| - `dynamic_tools` carries the caller-declared tool definitions admitted with a chat create. | |
| Their names and descriptions reach the model as instructions. |
Nit: one sentence per line
|
|
||
| Every `2xx` response must also carry a `Coder-Hook-Signature` header: an `HS256` JWT signed with the shared secret that echoes the request's claims (`jti`, `type`, `sub`, `aud`) with `body_sha256` recomputed over the exact response body bytes and a fresh validity window. | ||
| Responses steer chats, so Coder rejects an unsigned or mismatched response as a protocol error and fails the dispatch closed; without this requirement, anyone able to inject traffic on the return path could allow, deny, or rewrite tool input without knowing the secret. | ||
| `agenthooks.NewHTTPHandler` signs responses automatically, and `agenthooks.SignResponses` wraps any other `http.Handler` to do the same; both take the consumer's own URL and fully authenticate each request (bearer token, audience, and the claims' binding to the exact body bytes) before any handler logic runs, so neither can be used as a signing oracle for a replayed token with a different body. |
There was a problem hiding this comment.
| `agenthooks.NewHTTPHandler` signs responses automatically, and `agenthooks.SignResponses` wraps any other `http.Handler` to do the same; both take the consumer's own URL and fully authenticate each request (bearer token, audience, and the claims' binding to the exact body bytes) before any handler logic runs, so neither can be used as a signing oracle for a replayed token with a different body. | |
| `agenthooks.NewHTTPHandler` signs responses automatically, and `agenthooks.SignResponses` wraps any other `http.Handler` to do the same. | |
| Both take the consumer's own URL and fully authenticate each request (bearer token, audience, and the claims' binding to the exact body bytes) before any handler logic runs, so neither can be used as a signing oracle for a replayed token with a different body. |
| Responses steer chats, so Coder rejects an unsigned or mismatched response as a protocol error and fails the dispatch closed; without this requirement, anyone able to inject traffic on the return path could allow, deny, or rewrite tool input without knowing the secret. | ||
| `agenthooks.NewHTTPHandler` signs responses automatically, and `agenthooks.SignResponses` wraps any other `http.Handler` to do the same; both take the consumer's own URL and fully authenticate each request (bearer token, audience, and the claims' binding to the exact body bytes) before any handler logic runs, so neither can be used as a signing oracle for a replayed token with a different body. | ||
| Consumers not using the Go SDK must apply the same discipline: verify the request token against the exact body before acting on it, and mint the response token with the same claim set they verified. | ||
| Coder requests the `identity` content encoding; respond with the exact bytes the signature was computed over, without applying a content encoding, or the body hash will not match. |
There was a problem hiding this comment.
| Coder requests the `identity` content encoding; respond with the exact bytes the signature was computed over, without applying a content encoding, or the body hash will not match. | |
| Coder requests the `identity` content encoding. | |
| You must respond with the exact bytes the signature was computed over, without applying a content encoding, or the body hash will not match. |
Address findings from a security review of the chat lifecycle hooks stack (#27401, #27428, #27429, #27430): - toolschema: fail closed on tokenizer errors so a poisoned value (e.g. 1e999) cannot stop the ambiguity walk before a case-variant key smuggles a different tool input past pre_tool_use policy. - toolinput: extend the ambiguity guard to provider tools executed by a local runner; give computer-use an explicit property set covering both providers' decoders, including batched OpenAI actions. - agenthooks: authenticate hook responses. Every 2xx response must carry a Coder-Hook-Signature JWT echoing the request claims with body_sha256 over the exact response bytes; the dispatcher rejects unsigned or mismatched responses. NewHTTPHandler signs automatically and SignResponses wraps raw handlers. - dispatch: cap user_message (16 KiB), permission.reason (4 KiB), and input_override (256 KiB) instead of letting them ride the 1 MiB envelope into the prompt, DB, and UI. - chatd: include every caller-controlled instruction channel in user_prompt_submit admission events: the per-chat system prompt and dynamic tool definitions at create, and the owner's stored custom prompt on every admission, closing a policy bypass where only the prompt text was reviewed. - chathooks: report canonical tool names in pre/post_tool_use events so a name-keyed policy cannot be evaded via a deprecated alias. - subagent: fail message_agent closed on hook dispatch failure like spawn_agent, and redact dispatch errors (event, class, dispatch ID) before they persist into client-visible tool results; raw errors embed the operator's hook URL. - chatstate: reject hidden (visibility=model) rows as message edit targets so chat owners cannot overwrite hook-injected policy context; not-found keeps hidden IDs unprobeable. - exp_chats: cap postChatMessages and patchChatMessage request bodies like postChats; both feed hook dispatch and showed a measured 12x heap amplification per oversized message. - docs: document response signing, new admission fields, response caps, and input-hash deduplication for tool events.
Snapshot the admitted custom prompt on the chat row in the same transaction as every user_prompt_submit admission (create, send, edit, subagent spawn) and inject the snapshot at generation while hooks are enabled, so the model can never receive a custom prompt a policy was not shown. Give the SignResponses buffer net/http first-write-wins status semantics so a handler error cannot be flipped into a signed 2xx. Generate dynamic_tools as readonly DynamicTool[] instead of Record<string, string>.
Queued sends now store the admitted custom prompt on the queued row and every promotion path copies it to the chat row in the transaction that moves the message into history, so a later admission cannot replace an earlier queued turn's admitted value. Subagent spawn admission now carries the child's built system prompt (computer-use embeds the task prompt at system priority) and rebuilds it from a policy-approved prompt override so replaced text cannot survive as a system message.
net/http suppresses response bodies for 1xx, 204, and 304, so the signing buffer must refuse those writes too; otherwise the signature is computed over buffered bytes the real writer drops and the dispatcher fails a valid dispatch on a body-hash mismatch. Write now returns http.ErrBodyNotAllowed for those statuses, matching net/http.
SignResponses verified only the bearer token, so anyone who observed a legitimate hook request could replay its token with a modified body and use the wrapper as a signing oracle: the handler's decision for the attacker's body was signed under the original dispatch ID. The wrapper now authenticates exactly like NewHTTPHandler via a shared helper (POST-only, bearer token, audience, and the claims' body-hash binding) before invoking the handler, takes the consumer URL as the audience, and hands the handler the verified bytes.
A direct send or edit made while hooks are disabled now stamps a NULL admitted-custom-prompt snapshot instead of preserving an earlier admitted turn's value, so re-enabling hooks before that turn generates cannot inject stale admitted instructions. Also type DynamicTool.input_schema as unknown: it is an arbitrary JSON Schema document, not a flat string map.
net/http sends 1xx responses (except the final 101) without committing, so a handler emitting 103 Early Hints before its real status must still get a signed final response. The signing buffer now drops interim 1xx statuses and commits the first non-1xx status instead of locking on the informational one.
Dispatch requests now ask for the identity content encoding so conforming middleware cannot compress the signed response and the Go transport cannot transparently gunzip it; either transformation made VerifyResponse hash a different representation than was signed and fail valid dispatches closed. The signing buffer also freezes its header snapshot at the first commit like net/http, so header mutations after a handler's first write cannot desynchronize the forwarded headers from the signed body.
A dynamic tool named after a deprecated builtin alias (close_agent) survived the builtin-collision filter because the alias is never advertised as an active tool. The call then executed client-side while hook events reported the canonical builtin name (interrupt_agent) and the input guard validated the dynamic call against the builtin schema, rejecting schema-correct input. appendDynamicTools now drops dynamic tools whose name is a deprecated alias, the same way it drops builtin-name collisions, so alias names are reserved and both the event name and the schema choice stay consistent with what actually executes.
UAT confirmed six documented behaviors were stale or missing: - The ambiguous-input guard also covers provider tools Coder executes locally (computer-use), including batched OpenAI action keys. - Over-cap response values fail the dispatch closed instead of being truncated. - message_agent fails its turn closed like spawn admission, and client-visible dispatch errors are redacted to event, class, and dispatch ID. - Non-Go consumers get the full response-token claim contract, including the validity window and the signature requirement for empty-body 2xx responses. - Hook-injected model-only rows are absent from the messages API and unprobeable through the edit API; a user_prompt_submit model_context is folded into the prompt row instead. - Custom-prompt snapshots interact with break-glass windows: turns admitted while hooks are off, or never admitted through hooks, generate without the owner's custom prompt.
fec68d6 to
65bdc3e
Compare
mtojek
left a comment
There was a problem hiding this comment.
@jdomeracki-coder @ibetitsmike just being cautious here
I noticed this came from a security scan. If you're planning to backport it to a release branch, I don't think that's going to work because of the database migration. We generally don't backport changes like this due to the risk involved.
If you're not planning to backport it, feel free to ignore this comment - I just wanted to flag it 👍

Summary
Hardens the chat lifecycle hooks stack based on a deep security review of #27401, #27428, #27429, and #27430 (merged-stack review at
a31e5f3b, re-verified against6aef61a0). Four of the findings below were validated with executable PoCs during the review; the rest were confirmed by static trace.The stack has since merged to
main(byte-identical in its own paths), so this PR is rebased ontomain. Every fix was re-validated against the merged code: none were incorporated during the merge, and the only new chat endpoint since the review (getChatCost, read-only) does not feed hook dispatch.Problem / Fix
1.
pre_tool_usepolicy bypass via poisoned tool input (validated PoC).toolschema.ValidateUnambiguoustreated any JSON tokenizer error as acceptance. Prefixing tool input with"_x":1e999(a float64 overflow thatjson.Unmarshalskips inside ignored fields) aborted the walk before a case-variant key such asCOMMANDwas inspected, so the hook consumer approved one command while coderd executed another. The walk now usesjson.Numberand fails closed on any tokenizer error, with regression tests.2. Ambiguity guard did not cover computer-use. The guard only checked tools in
BuiltinToolNames; computer-use is a provider tool with a local runner and an empty advertised schema, so case-variant keys (including inside batched OpenAIactions) were never validated while both providers' decoders fold keys case-insensitively. The guard now covers locally-run provider tools, with an explicit property set for both computer-use decoders.3. Hook responses were unauthenticated. Requests carry a body-bound JWT, but any traffic injected on the return path (the documented plain-HTTP loopback hop, or any hop behind a TLS terminator) could forge
allow/deny/input_overridedecisions without knowing the secret. Every 2xx response must now carry aCoder-Hook-SignatureJWT echoing the request claims (jti,type,sub,aud) withbody_sha256recomputed over the exact response bytes.NewHTTPHandlersigns automatically; newSignResponseswraps arbitrary handlers; the dispatcher rejects unsigned/mismatched responses as protocol errors (fail closed).4. Admission events hid instruction channels (validated PoCs).
user_prompt_submitcarried only the prompt, so operator policy could be bypassed by putting instructions in the per-chatsystem_promptorunsafe_dynamic_tools[].descriptionat create, or persistently viaPUT /chats/config/user-prompt, none of which produced any hook event. The event now carriessystem_prompt,custom_prompt, anddynamic_tools, so a policy sees every channel admitted with the turn.5. Unbounded request bodies with 12x heap amplification (validated PoC).
postChatMessagesandpatchChatMessagehad nohttp.MaxBytesReader(unlikepostChats), and each oversized message multiplied into ~12x heap allocations per hook dispatch plus 2x egress to the operator endpoint. Both handlers now cap bodies at2*maxSystemPromptLenBytes(256 KiB), matchingpostChats.6. Uncapped hook response fields. Only
model_contextwas capped;user_message,permission.reason, andinput_overridecould ride the 1 MiB envelope into the LLM prompt, DB, and UI. Now capped at 16 KiB / 4 KiB / 256 KiB respectively.7. Tool-name alias evasion. Events reported the deprecated alias (
close_agent) while execution resolved the canonical tool (interrupt_agent), evading name-keyed policies. Events now report the canonical name. Deprecated alias names are also reserved fromunsafe_dynamic_tools: a dynamic tool named after one previously executed client-side while events reported the canonical builtin and the input guard validated it against the builtin schema, rejecting schema-correct calls.8.
message_agentfailed open on dispatch failure. Unlikespawn_agent, a hook dispatch failure became a tool error string the model could ignore. It now fails the turn closed, and dispatch errors are redacted (event, class, dispatch ID) before persisting into client-visible tool results; previously the raw error embedded the operator's hook URL/internal host.9. Hidden-row edit overwrote hook-injected policy context. The message edit path accepted
visibility=modeluser-role rows (exactly what hooks inject asmodel_context) as edit targets, letting a chat owner rewrite policy context the model trusts. Hidden rows are now rejected as not-found, keeping their IDs unprobeable.Docs (
chat-lifecycle-hooks.md, chatdARCHITECTURE.md) document response signing, the new admission fields, response caps, and input-hash deduplication for tool events (a replayedtool_use_idwith different input previously inherited a cached allow). A UAT pass closed six more contract gaps inchat-lifecycle-hooks.md: guard coverage for locally-run provider tools, fail-closed over-cap responses,message_agentfail-closed plus redacted dispatch errors, the full response-token claim contract for non-Go consumers, unprobeable hook-injected rows, and custom-prompt snapshot behavior across break-glass windows.Deliberately not addressed here (need a design call)
post_tool_use/stopdispatch for already-persisted results (consumer state divergence).Validation
go build ./...; full test suites forcodersdk/x/agenthooks,coderd/x/agenthooks/dispatch,coderd/x/chatd(+chatstate,chathooks,toolschema), andcoderd -run Chatall pass against a real Postgres.golangci-lintclean on touched packages;typesGenerated.tsregenerated.main: build, the same test suites, lint, and atypesGenerated.tsregeneration check all pass.🤖 Generated with Mux