feat: audit chat system instructions changes - #27668
Conversation
Docs previewCheck off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here. |
Status, 30 July 2026: not ready to mergeHanding this over. CI is green on the current head, but there is a known defect that must be fixed first. Defect: both handlers re-read the stored value after writing it, to record the new value. If that read fails while the transaction is still healthy, the handler returns 500 and rolls back a write that already succeeded. Client cancellation is enough to trigger it: transactions here are started with a background context, so cancelling a request cancels the individual query and not the transaction. On an unmodified build the write survives and only the response is lost, so this is a behaviour regression that can silently discard an administrator's change. Fix: delete the post-write read and compute the new value from what was written. The upserts store the request value verbatim. For the system prompt, derive the effective state in Go and pin that derivation with a test comparing it against Also outstanding:
What is sound and reviewed: the resource type name covers everything it carries, each setting has a stable id and a human-readable target so a row says which instruction changed, denied requests produce a 403 entry with no request-body content, and the include-default toggle records whether an explicit override row exists rather than only its effective value. Context and the wider plan: see the "Agents audit logging: handover" document on CODAGT-66.
|
Update, same day: the blocking defect is fixedNew head Also in this head:
Measured against an unmodified main build: write failures match byte for byte, including the transaction wrapper on the system prompt endpoint (transactional on main) and the raw error on plan-mode (not transactional on main). Plan-mode's begin, commit and rollback positions now return 204 with the write performed, matching main, rather than 500. Old-read failures are 204 with the write performed and no entry on both endpoints. There is no post-write read position left. Concurrency was strengthened to a barrier proof: with the lock, two concurrent identical writes produce one entry; with the lock removed and both transactions parked at the old-value read, two entries with a stale previous value, deterministically. One residual behaviour worth weighing, listed among the decisions needing human sign-off in the handover document on CODAGT-66: a lock wait exceeding the bound returns 500 where an unmodified build would have completed the write. That is the cost of recording an exact before-and-after, and dropping the lock is a legitimate alternative that several existing audited endpoints here already take. This head has not been reviewed. A review was commissioned; if its verdict is not appended below, treat
|
Review verdict on this head, and the recommended directionReviewed and came back needs-changes. Both findings are about the same thing: the per-setting advisory lock added for audit exactness.
Direction, from the ticket ownerDo not patch the lock's failure path. Delete the lock. It exists only to make the recorded before-and-after exact when two administrators edit the same setting in the same instant, which is not a problem worth a new failure mode on the write path, and the ticket owner has said so explicitly. Removing it also removes the fallback path, the bound, and the nondeterministic test above. If exactness is ever genuinely wanted, the lever is the transaction isolation level, not application-level locking: What remains after deletion is the shape that was worth building: the audited old value read inside the same transaction as the write, the new value derived from what was written with no read after the write, and every failure position matching an unmodified build.
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7ab4cd16fd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Pushed c91a6d4 to fix the failing The flake detector caught Fix (test-only): read before parking, so every waiter deterministically holds the same stale Old regardless of scheduling. Validation: reproduced the exact CI failure by amplifying the waiter wakeup delay on the old ordering (red), confirmed the same delay cannot fail with the new ordering (green), then ran both concurrency subtests with
|
|
Pushed two more commits to fix the remaining CI failure: main gained migration
Validated locally on the merged tree:
|
|
@codex review |
|
Codex Review: Didn't find any major issues. Already looking forward to the next diff. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3213004bb6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
|
Codex Review: Didn't find any major issues. Keep it up! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Register a chat_system_prompt_settings audit resource covering the deployment-wide chat system prompt, include-default toggle, and plan-mode instructions, and wire both PUT endpoints to it. The system-prompt handler reads the config pair before and after its existing conditional upserts inside the same transaction; the re-read is load-bearing because the effective include-default flag is computed from the toggle row and the current prompt, so a prompt-only write can flip it. The plan-mode handler wraps its previously transactionless upsert in a transaction to capture the old value. Write paths stay byte-identical: value-identical PUTs still upsert, and only the audit entry is suppressed by leaving both sides without a resource ID.
Review round 2 found three defects where audit observation changed endpoint behavior. Audit capture reads in both handlers now degrade best-effort: a failed Old-capture or New re-read logs a warning and skips the entry instead of aborting the request or rolling back the completed upsert, restoring byte-identical behavior with main under read failure. The write path stays the failure surface: the upserts and a new advisory lock (LockIDChatSettingsWrites) that serializes change-detection with the write still fail the request on error. The lock also closes a race where two concurrent identical PUTs each captured a stale Old and both emitted a change entry; the second transaction now sees the first's committed state and suppresses its duplicate. Handler tests pin all of it with mutation-proven regression cases; a test-only DiffOverride on InitRequest (wired through the API, nil in production) proves the fallback include-default flip reaches the audit diff with the re-read value.
Round 2 review ruled the DiffOverride seam a production-exported audit bypass: RequestParams.DiffOverride and the API setter were exported in normal builds and could replace or fabricate audit diffs for any audited resource. The seam is removed entirely from production code (RequestParams field, the InitRequest commit branch, the API field and setter), and the assertion support moves into the test double: MockAuditor gains a construction-time NewMockWithDiffFn whose supplied function computes the entry's diff, so the fallback-flip test pins the captured Old/New pair without any production-code hook. The flip test still dies under the stale-New mutation (re-proven).
Wrapping the previously non-transactional plan-mode upsert in InTx changed the error response body: main's detail was the raw write error, the wrapped version's was "execute transaction: <error>". The handler now records the callback's write error and responds from it exactly as main did, keeping the InTx wrapper only for lock, begin, commit and rollback failures so those stay distinguishable. The existing rollback regression test pins the raw detail, mutation-proven by restoring the wrapper.
The plan-mode handler built its audit New from request-derived text, contradicting the rule the system-prompt endpoint already follows: the write may normalize the value, so request-derived text can misreport the change. It now re-reads the stored value after the upsert inside the same transaction and uses that read for both New and the change comparison, with the same best-effort degradation on a failed re-read. A normalizing test store proves the audited New is the stored value, mutation-proven by restoring the request-derived assignment. Also renumber the enum migration off the 000561 collision main took.
Round 3 review found the audit machinery itself could change what a
member sees: the advisory lock and the plan-mode transaction exist
only for auditing, and their failures replaced main's successful
response. The write path is now authoritative on both endpoints: on
any audit-machinery failure (lock, begin, commit, rollback) the
handler runs main's idempotent write path directly and derives the
response from it, while write failures keep the exact response the
endpoint produced before the audit wiring existed (transaction error
for the system prompt, which was always transactional; the raw write
error for plan mode, which was not). The full InTx error is logged
before the response detail is chosen so rollback failures cannot
vanish. Accepted consequence: when the lock cannot be taken, two
concurrent identical writes can both record, which is audit
degradation, not a behavior change.
Operator decision D5 lands in the same change. The resource type is
renamed chat_instruction_settings (Go type ChatInstructionSettings,
friendly string, frontend filter label) because it carries the
plan-mode instructions too. Each endpoint now has a fixed resource ID
and a human-readable target ("System prompt", "Plan mode
instructions"), so history-by-setting works; identity is assigned
before authorization, so denied and failed PUTs record the attempt
with the real status and an empty diff, and capture-degraded writes
record instead of going silent, distinguishing "nothing changed" from
"something changed and capture degraded". No-op suppression moves off
the nil-ID skip to InitRequestWithCancel, which also removes the trap
where setting Old.ID would have emitted a row claiming every field was
cleared. Migration renumbered to 000562 (main took 000561).
The fallback rule was wrong on the post-commit position: when the audited transaction's machinery fails after Old was captured and the write succeeded, the fallback direct write lands the same value and the client sees 204, so the request is a success in every respect and gets the ordinary entry (Old to the stored New, real diff, 204). An empty diff would under-report a change that happened, and a failed entry would contradict the success the client saw. When the machinery fails before any baseline exists (lock or begin), no truthful diff is possible, so the attempt row keeps an empty diff and the real status, matching S2's documented limit with S1's D5 identity. A commit-failure test proves the ordinary entry, mutation-proven by dropping the post-fallback New population (the entry's New went empty and the test went red); a fallback-then-ordinary test proves the degraded write does not desync the next baseline.
Two fixes from review round 2. A failed post-write New read left the audit New at its identity-only zero payload while the request succeeded, so the production enterprise differ recorded Old-to-empty and the row claimed the setting was cleared when it was not. Every degraded capture branch on both endpoints now sets New = Old before returning, so a failed audit read yields an unknown value and an empty diff, never a fabricated deletion; the rule is stated in the code comment and pinned with the production enterprise differ (the empty-diff mock is exactly why it escaped earlier rounds), mutation-proven by restoring the zero New. The include-default audited state now also carries whether the override row EXISTS, not only its effective value: writing explicit false over a legacy absent row does not move the effective boolean but inserts the persistent row and changes future behavior, so the presence field (include_default_system_prompt_set) is captured, stored in the audited struct, enumerated in the audit table, and included in the no-op comparison. The nil-to-explicit transition audits even when the effective value does not move, and explicit-false to explicit-false stays silent, both pinned with the production differ.
Final structure for the audit write path, settled against a real PostgreSQL 13 instance. The best-effort premise collapses for these queries: siteconfig reads are single non-null text scans with no data-dependent conversion, and the reviewer could not make one fail while leaving the write able to succeed, so there is no reachable degraded path to defend. Audit reads (Old and New) are now fatal like usersecrets.go: the error surfaces through the endpoint's existing error path exactly as main's does. The whole fallback structure is deleted: no retried write, no direct-write path, no writeErr classification, no post-fallback re-read, no degraded rows, and the sentinel store kit and recording-sink degradation assertions that only existed to prove degradation go with it. One transaction per request, shaped like main's. The global write lock is replaced by a per-setting advisory lock taken first inside the transaction. Lock IDs derive from the exact site_configs key with GenLockID (FNV-1a 64) instead of the sequential LockID* block, so writers of different settings never contend and the IDs cannot collide with the sequential block or another subsystem's GenLockID output (the key strings are unique to these settings). LockIDChatSettingsWrites is deleted; nothing else used it. FOR UPDATE is not added: the per-key lock subsumes it and covers the absent-row case a row lock cannot. Mutation-proved by removing the plan-mode lock: two concurrent identical PUTs each captured a stale Old and both emitted, and removing it turned the suppression test red. Kept unchanged: the rename to chat_instruction_settings, the fixed per-setting identity and targets, denied-attempt rows with empty diffs, the no-fabricated-deletion guarantee (now unreachable rather than defended against), include-default presence auditing, and no-op suppression through InitRequestWithCancel taken inside the transaction. Accepted costs documented in the PR body: concurrent first writes to a not-yet-existing setting row can each record an empty Old and two identical first writes can each emit; and the added lock introduces one theoretical failure position, a lock wait exceeding the request deadline, whose only holders are sibling requests holding it for a single upsert.
GenLockID derives the per-setting advisory lock IDs from a string, so collision-freedom is a property to verify, not argue. The test computes both generated IDs, asserts they are pairwise distinct, and asserts each is distinct from every sequential LockID* constant, listed explicitly because iota constants cannot be enumerated programmatically. A future constant that collides now fails in review rather than in a production deadlock.
The previous premise, that no read failure can leave the write able to succeed, was falsified on real PostgreSQL: a database.Store wrapper can error before any SQL reaches Postgres, and query-local context cancellation fails the individual query without marking the server-side transaction failed (InTx begins it with context.Background). The post-write New read was the reachable case: on main the write lands and the client just never sees the response, while the audited handler rolled it back, silently discarding a change an administrator made. That is worse than any audit imprecision. The post-write read is deleted rather than protected. New is derived from what was written: the upserts store $1 verbatim, so the stored value is the request value, and the effective include-default flag follows GetChatSystemPromptConfig's rule from the written toggle row and the written prompt. A parity test compares the derivation against the getter's result on the happy path for every combination of written prompt and include-default input, so drift is caught. The Old capture moves back inside the per-setting lock but stays best effort: it runs before the write, so abandoning the transaction on its error costs nothing, and the handler runs the direct write path with no entry and a warning. The no-op decision is staged into a local and applied only after the transaction commits, so a commit failure cannot suppress an attempt row. The per-setting lock stays, now the only audit-added statement in the transaction, with a service-owned 5s bound on the wait so a waiter cannot hang past the client deadline. Plan-mode's audit-added transaction machinery (lock, begin, commit, rollback, and the Old read) runs the direct write instead, matching main's non-transactional behavior, while the system-prompt endpoint, already transactional on main, surfaces those errors exactly as main's. The full InTx error is logged before the response detail is chosen, so a rollback failure layered on a write failure is never absent from response, audit row and logs simultaneously. The concurrency tests are strengthened to the barrier shape: with the lock, two concurrent identical PUTs produce one entry; with the lock removed and both transactions parked at the Old capture, both read the same stale Old and both emit, the deterministic mutation proof.
The without-lock concurrency subtest parked goroutines before the Old-capture read. The goroutine that released the barrier could write and commit before a slowly waking waiter executed its read, which then saw the committed value under read committed, detected no change, and suppressed its audit entry, failing the expected-duplicate assertion. Read before parking so every waiter deterministically holds the same stale value.
Main gained 000562_oauth2_public_client_tokens, so the PR merge checkout contained two migrations with version 000562 and every database-touching CI job failed with a duplicate migration file error. No fixture or test references the number.
…overage - Fall back to the unaudited write path when the chat system prompt advisory lock times out, instead of failing the update with 500. The lock exists only for audit change detection, so contention must not block a write that succeeded before the audit wiring existed. - Add Storybook interaction coverage that opens the audit resource-type filter and selects the Chat Instruction Settings option, verifying the generated resource type and its friendly label together.
Main gained 000571_pool_aware_chat_acquisition_index and 000572_chat_files_token_crypto_key_feature while this PR aged, so the branch migration moves past both to keep versions unique and ordered.
…aseline A lock-wait timeout in putChatPlanModeInstructions fell back to the direct write but never canceled the deferred audit, emitting a successful 204 entry with an empty diff despite the instructions changing. Gate the suppression on the captured baseline instead of the Old-read error so every fallback without a baseline emits no entry, matching the system-prompt handler. A commit failure keeps its baseline and still emits the attempt row with its real diff.
0b08a10 to
c150c76
Compare
|
Rebased onto current main (per Mike's request) to linearize history and order migrations properly: the branch's two merge commits are gone, all 16 commits now sit on top of main, and the audit migration moves to 000573 (main gained 000571_pool_aware_chat_acquisition_index and 000572_chat_files_token_crypto_key_feature while the PR aged). No functional changes: the rebased tree was verified to differ from the pre-rebase tree by exactly main's new delta plus the migration rename (byte-identical migration content, R100). Validated post-rebase: migration suite, TestChatSystemPrompt, TestChatPlanModeInstructions, and a forced make gen with zero drift.
|
|
@codex review |

Adds an audit record for administrative events on the deployment-wide chat instruction settings (system prompt, the include-default toggle, and the plan-mode instructions), per CODAGT-719 and operator decision D5. Each endpoint records under a stable identity: resource type
chat_instruction_settings, a fixed resource ID and a human-readable target ("System prompt", "Plan mode instructions"), so two changes to one setting share an ID and history-by-setting works. A real change exports a Write entry with the old-to-new text visible; a value-identical PUT still upserts and still returns 204 but records nothing.Attempts are recorded, not only transitions. Identity is assigned before the authorization check, so a denied PUT exports a 403 row with an empty diff (no request content reaches it), a validation failure exports a 400 row, and a write failure exports a 500 row, each with an empty diff; an operator can tell "nothing changed" from "something changed and capture degraded" by the status code.
The write path stays authoritative. The advisory lock and, on plan-mode, the transaction exist only to serve change-detection; if any of that machinery fails (lock, begin, commit, rollback), the handler runs main's idempotent write path directly and derives the response from it, so a member-visible failure of audit-only infrastructure can never replace main's successful response. Accepted consequence: when the lock cannot be taken, two concurrent identical writes can produce two rows instead of one. That is audit degradation, which is allowed; changing a member's response is not. Write failures keep the exact response the endpoint produced before this wiring (transaction error for the system prompt, which was always transactional; the raw write error for plan mode, which was not), and the full transaction error is logged so rollback failures cannot vanish.
CODAGT-66 plan entry: S1 (verbatim)
S1
feat: audit chat system instructions changes(CODAGT-719; base: main)database.ChatSystemPromptSettings{ID uuid.UUID; SystemPrompt string; IncludeDefaultSystemPrompt bool; PlanModeInstructions string}incoderd/database/types.go(ticket-sketched shape; one struct, both endpoints).idActionIgnore, other three ActionTrack),AuditActionMapWrite-only; four request.go cases (ResourceTarget"",ResourceIDfrom struct,ResourceTypenew enum valuechat_system_prompt_settings,ResourceRequiresOrgIDfalse with the "Artificial ID / deployment singleton" comment convention).ALTER TYPE resource_type ADD VALUE IF NOT EXISTS 'chat_system_prompt_settings';comment-only no-op down (000558 shape); number picked at push per the numbering constraint.FriendlyString("chat system prompt settings");TestAuditDBEnumsCoveredforces both.coderd/audit.gopresentation switches: rely on safe defaults (no link, generic description); no FE changes (filter label falls back to capitalized value; acceptable per precedent).putChatSystemPromptandputChatPlanModeInstructions: InitRequest with Action Write; artificialID: uuid.New()onNewonly when a change is detected; no-op suppression by leaving both aReq sides unset (nil resource IDs skip the log, request.go skip rule); the write path itself stays byte-identical (upserts still run unconditionally).putChatSystemPrompt(writes two keys conditionally in one existing tx): inside that tx, read the pair viaGetChatSystemPromptConfigforOld, perform the conditional writes exactly as today, then RE-READ the pair forNew. The re-read is load-bearing:include_default_system_promptis computed from the toggle row AND the prompt, so a prompt-only write can flip the effective value without the request carrying the pointer.PlanModeInstructionsstays zero on both sides.putChatPlanModeInstructions(no tx exists today): wrap its read-upsert inInTx(behavior-preserving: same single write);Old/Newpopulate onlyPlanModeInstructions; the two system-prompt fields stay zero on both sides; no cross-key reads.audit.NewMock()asserting Write entry on change and NO entry on a value-identical PUT, for both endpoints (this also exercisesResourceRequiresOrgIDend to end); the fallback-flip case (no explicit include-default row, nonempty prompt set to empty, effective boolean flips: entry emitted with the boolean diff); diff assertions (old->new prompt text tracked, not secret) inenterprise/audit/diff_internal_test.go;TestAuditableResourcespasses by construction.Note: the plan excerpt above predates operator decision D5 (2026-07-30), which this PR implements: the resource type is
chat_instruction_settings(notchat_system_prompt_settings), each setting carries a stable ID and a display-name target (not a per-write artificial ID and an empty target), no-op suppression runs throughInitRequestWithCancel(not the nil-ID skip), and attempts (denied, failed, capture-degraded) record rows with real statuses and empty diffs. Ticket bookkeeping for CODAGT-719 was corrected on Linear at kickoff: the ticket's "matches the existing 204-on-unchanged behavior" premise does not exist on main; suppression is new, and the write path is unchanged.