{{ message }}
Repository navigation
fix(core): keep the prompt cache when compaction retains sent messages - #54107
Draft
AnnoyingTechnology wants to merge 1 commit into
Draft
AnnoyingTechnology wants to merge 1 commit into
AnnoyingTechnology wants to merge 1 commit into
Conversation
Summary compaction sent only the older part. When the retained recent part held exchanges the provider had already seen, that request was a truncated copy of the session's last request, and breakpoint or recurrent-state caches missed on it. Summarize through the latest assistant reply when it fits, and fall back to the older part otherwise. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Contributor
|
The following comment was made by an LLM, it may be inaccurate: |
3 of 6 tasks
|
Wow, it does work! I've hit multiple auto-compactions running this branch, and cache hit rate stays at 1 until the compacted (new) prompt is sent to the model. |
1 task done
3 of 6 tasks
3 of 6 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue for this PR
Closes #52761
Fixes #52761
Type of change
What does this PR do?
Summary compaction sends only
split.older.recentstarts at a user message and usually holds exchanges the provider has already seen, so the summary request is a truncated copy of the last request instead of an extension of it. Caches that only match at their own breakpoints (Anthropic) or at saved recurrent states (hybrid models on SGLang) miss, and the older part is billed or prefilled again.When the compaction model is the session model and there is room, the summary request now runs through the latest assistant reply, so it extends the last request. Unanswered trailing input stays out and the stored
recentis unchanged. If that request is rejected as too long or too large, or the summary hits the output limit, it falls back to the existing older-only path. Both attempts share one retry allowance, and the autoStartedevent is published once.Known limitation: after a cut-off first reply, the mini TUI shows the discarded draft before the final summary, as the template-reminder retry already does. The stored summary is correct. This does not explain the OpenAI rows in #52761.
How did you verify your code works?
v2and pass here: wire-level prefix for manual and auto on OpenAI Chat, Responses and Anthropic Messages; fallback after a cut-off reply, without room, over the input limit; one retry allowance across the fallback; configured session model vs another model.bun run checkpasses.packages/corepasses except "isolates global home and XDG roots", which fails the same way onv2on this machine.Screenshots / recordings
Not applicable.
Checklist
Patch, tests and live runs by Claude Opus 5.5 (via Claude Code), for @AnnoyingTechnology. Reviewed by GPT 6.1 Sol (xhigh, via Codex).
🤖 Generated with Claude Code