feat(coderd/x/chatd/chattool): improve find_tools relevance and model guidance by ibetitsmike · Pull Request #28476 · coder/coder · GitHub
Skip to content

feat(coderd/x/chatd/chattool): improve find_tools relevance and model guidance - #28476

Merged
ibetitsmike merged 7 commits into
mainfrom
mike/find-tools-limit
Aug 25, 2026
Merged

feat(coderd/x/chatd/chattool): improve find_tools relevance and model guidance#28476
ibetitsmike merged 7 commits into
mainfrom
mike/find-tools-limit

Conversation

@ibetitsmike

@ibetitsmike ibetitsmike commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator

Broad find_tools queries such as "linear issues" saturated the hard 20-match cap on every call: a large server's name token-matches every one of its tools (+8 name, +1 server), so ranking was decided by a term carrying no discriminating information and the model got 20 activations per search regardless of intent.

Three changes, mirroring what makes xum's tool_catalog_search behave well:

  • Lower default with a limit argument. Keyword matches default to 10 per call; the new optional limit raises that up to the existing hard cap of 20 (non-positive values fall back to the default). Exact names are explicit activation requests and bypass the limit up to the hard cap, so naming 15 tools still activates all 15.
  • Coverage-first ranking. Keyword matches sort by distinct query terms matched before raw score, so a query like "linear issues" ranks tools matching both terms above the dozens matching only the server name.
  • Model guidance. queries and names now carry schema descriptions (capability keywords and what they are matched against; exact-name activation), and the tool description opens by explaining what deferral means, that matches become callable on the next step, and that a "server: terms" prefix scopes a query to one server.

An earlier revision also inferred a server scope from unprefixed query words ("linear issues" behaving like "linear: issues"). Review kept surfacing edge cases in that inference, and coverage-first ranking already resolves the original saturation complaint, so it was dropped in favor of the explicit prefix.

The hard cap stays at 20 so the persisted result keeps fitting under the generic tool-result truncation budget that protects activation-recovery JSON. Budget and reservation accounting are unchanged; the frontend renderer ignores unknown argument fields, so no site/ change is needed.

Note

Xum acted on @ibetitsmike's behalf in this pull request.

@ibetitsmike ibetitsmike changed the title feat(coderd/x/chatd/chattool): add find_tools limit argument with lower default feat(coderd/x/chatd/chattool): improve find_tools relevance and model guidance Aug 24, 2026
@ibetitsmike
ibetitsmike marked this pull request as ready for review August 24, 2026 14:53
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5047e29ff6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/chattool/findtools.go Outdated
Comment thread coderd/x/chatd/chattool/findtools.go Outdated
Comment thread coderd/x/chatd/chattool/findtools.go Outdated
Comment thread coderd/x/chatd/chattool/findtools.go Outdated
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6a0cf448a5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/chattool/findtools.go Outdated
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 27c94dc381

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/chattool/findtools.go Outdated
Comment thread coderd/x/chatd/chattool/findtools.go Outdated
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 435c90d387

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/chattool/findtools.go Outdated
Comment thread coderd/x/chatd/chattool/findtools.go Outdated
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4e719f93d2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/chattool/findtools.go Outdated
…coping

Inferring scope from unprefixed query words kept accumulating edge
cases (word-inspection caps, repeat merging, fold-family spans,
per-query fallback) across review rounds. Coverage-first ranking
already keeps a server name from deciding rank, and explicit
"server:" prefixes remain for deliberate scoping, so the inference
no longer carries its weight. Queries without a recognized prefix
now always search unscoped.
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

@ibetitsmike
ibetitsmike merged commit e244cac into main Aug 25, 2026
32 checks passed
@ibetitsmike
ibetitsmike deleted the mike/find-tools-limit branch August 25, 2026 09:39
@github-actions github-actions Bot locked and limited conversation to collaborators Aug 25, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants