Print full help on command misuse for invoking agents - #14198
Conversation
When an agent misuses a command, the terse usage string does not carry the examples, JSON fields or environment variables it needs to correct itself, forcing a second `--help` invocation. Extract the help renderer out of rootHelpFunc so it can target any writer, then use it from printError when an agent is detected. Rendering directly to stderr avoids cmd.Help(), which writes to stdout and would otherwise split a single failure across two streams. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
0bfd4ba to
3f2ae26
Compare
There was a problem hiding this comment.
Pull request overview
Prints full command help on misuse when an AI agent invokes gh, while preserving terse usage for humans.
Changes:
- Extracts help rendering into a writer-aware function.
- Sends agent-facing error help entirely to stderr.
- Adds unit coverage for full-help and unchanged-error paths.
Show a summary per file
Review details
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- Files reviewed: 3/3 changed files
- Comments generated: 1
- Review effort level: Balanced
babakks
left a comment
There was a problem hiding this comment.
Thanks for tackling this, @niik! 🙏 Nice write-up in the description, and the WriteHelp extraction reads cleanly.
One thing worth another look before this lands: when an agent is detected, WriteHelp fully replaces the terse usage on root-level misuse, so the flat Available commands: block that humans see gets dropped rather than augmented. I left an inline note with a repro and a suggested tweak to append the terse usage in the fullHelp branch (plus a heads-up that cmd.UsageString() returns empty for gh commands, so it can't be captured the obvious way). I dug into this with Copilot, so please sanity check it on your end.
Everything else looks good to me. 🎉
There was a problem hiding this comment.
internal/ghcmd/cmd.go — printError, fullHelp branch (~L306-L313)
Heads-up: I noticed this discrepancy while reviewing and asked Copilot to dig into the case, so @niik please sanity check this one more time before acting on it.
Repro (compare the two):
go run ./cmd/gh xyz
AI_AGENT=foo go run ./cmd/gh xyz
The difference: the first (human) prints the terse Usage: ... Available commands: ..., while the second (agent) prints WriteHelp's full grouped help instead, so the flat Available commands: block is gone rather than just having extra help added to it.
idea: In fullHelp mode we should append the terse usage so agents keep everything a human sees. Right now WriteHelp replaces cmd.UsageString(), so the flat Available commands: block (and the flag list on leaf commands) is dropped for agents on root-level misuse.
Verified behavior:
gh zzqqxx(root unknown command) reachesprintError. Human mode shows the terseUsage: ... Available commands: ...; agent mode showsWriteHelp's grouped rendering instead, so the literalAvailable commands:block is gone.gh pr zzqqxx(nested unknown subcommand) never reachesprintErrorat all - it's handled earlier bynestedSuggestFunc, so it stays terse in both modes.
Heads-up on the "capture cmd.UsageString() first" idea: for real gh commands cmd.UsageString() returns an empty string and writes the usage to IOStreams.ErrOut as a side effect, because gh's SetUsageFunc (root.go:115) writes straight to the captured ErrOut and ignores the buffer Cobra swaps in. Empirically it returns len 0 and leaks ~946 bytes to ErrOut. So capturing it won't capture anything. Instead expose the terse renderer and call it directly.
I also checked the "WriteHelp mutates Cobra state that UsageString reads" theory - it does not; UsageString returns the same thing before and after WriteHelp. So no capture/reorder is needed; we can just append.
Suggested approach (spans two files, so not a single suggestion block):
In pkg/cmd/root/help.go, expose the terse renderer:
// WriteUsage renders the terse usage string for command to w. cmd.UsageString()
// cannot be used for this because gh sets a usage function that writes straight
// to IOStreams.ErrOut, so UsageString returns an empty string and leaks the
// usage onto ErrOut as a side effect.
func WriteUsage(w io.Writer, command *cobra.Command) error {
return rootUsageFunc(w, command)
}In internal/ghcmd/cmd.go, append it in the fullHelp branch:
if fullHelp {
// Render into out rather than calling cmd.Help(), which would send
// the help text to stdout and split a single failure across two streams.
root.WriteHelp(out, cs, cmd)
// Also append the terse usage so agents keep everything a human sees,
// including the "Available commands" list that WriteHelp replaces with
// its grouped rendering.
fmt.Fprintln(out)
_ = root.WriteUsage(out, cmd)
fmt.Fprintln(out)
return
}Verified output for AI_AGENT=foo gh zzqqxx: full help, then Usage: gh <command> <subcommand> [flags] + Available commands: list appended at the end.
Note: this will change the expected output in the newly added Test_printError full-help cases, so those wantOut values need updating to include the appended usage.
There was a problem hiding this comment.
Thanks for flagging this, and for the nudge to sanity check — I dug in properly and had the findings independently verified before acting.
Your mechanism is right on both counts. cmd.UsageString() does return "" for real gh commands: SetUsageFunc (root.go:115) writes straight to the captured ErrOut, so the usage text you see is a side effect and the fmt.Fprintln(out, cmd.UsageString()) contributes only a blank line. Confirmed on gh pr list --badFlag: 972 bytes to stderr, exactly one Usage: line and one --web line, terminated by a lone \n — if the string were actually being returned we'd see it twice. Worth noting the existing human path only works because out and ErrOut are the same stream.
And gh pr zzqqxx does bypass printError entirely, byte-identical in both modes, as you said.
Where I land differently is on the consequence, so let me show the numbers rather than just assert it. Comparing information content rather than the literal string, for gh zzqqxx:
- human: 36 bare command names (429 bytes)
- agent: the same 36 names, each with a description, plus group structure and the 8 help topics (2,837 bytes)
The set difference human-minus-agent is empty. The same holds on leaf commands, which I think is the part worth double-checking against the "(and the flag list on leaf commands) is dropped" note: for gh pr list --badFlag, agent mode contains all 14 flags the human sees, plus --help and --repo, plus the JSON fields and examples. Nothing is dropped in either case — the grouped rendering is a strict superset, not a replacement that loses something.
So appending the terse usage would reprint the same command names and flags in the less informative form: roughly +14% at root, and about +38% on gh pr create --badFlag where it would duplicate the entire FLAGS block. Given the token-cost pressure this epic is tracking, I'd rather not pay that for content already present.
One further reason to avoid WriteUsage specifically: calling rootUsageFunc directly would bypass the alias resolution added in 3a73af2, re-introducing the alias-stub problem for gh <alias> --badFlag that the other review thread caught.
That said, I think you're onto something real with the nested case. Because nestedSuggestFunc handles gh pr zzqqxx before printError, agents get the terse list and no full help there — which is arguably the more valuable gap, since that's a much more common agent mistake than root-level misuse. It's outside this PR's changed path, but I'd be happy to open a follow-up issue for it if you agree.
Happy to be overruled on the append if you still see a case I'm missing.
There was a problem hiding this comment.
Thanks for looking deeper, @niik. I agree that this is so hacky already and touching it is a scope creep. Let's leave it as is for now.
A user-defined alias is a stub command: its own help carries no flags or examples, only "Alias for ...". NewCmdRoot works around this by pointing the alias's usage and help funcs at the command it expands to, but printError calls root.WriteHelp directly and so bypassed that redirection. The result was that agents got strictly worse output than humans for aliases: `gh prs --badFlag` rendered 384 bytes of alias stub with none of `pr list`'s flags, leaving the agent no way to correct itself. Resolve the alias inside WriteHelp so the behaviour cannot be missed by a future call site. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d6afb36c-b3b5-472f-b540-025e38395b79

Closes https://github.com/github/gh-cli-and-desktop/issues/56
Follows on from #14191, now merged. This branch has been rebased onto
trunkand is a single commit.Description
When someone mistypes a
ghcommand, we print the error followed by a short usage string: the usage line and a list of flags. A person reads that, spots their mistake, and moves on.An AI agent cannot. The usage string omits the description, the examples, the JSON fields, and the environment variables, so an agent that guessed a flag wrong has no way to work out what the right one was. It has to spend a second invocation on
gh <command> --helpto find out. That extra round trip is exactly the kind of thing #14191 started addressing by detecting which agent is driving the CLI.This change reuses that detection: when an agent is invoking
ghand the failure is command misuse, we print the full help text instead of the terse usage string. Behaviour for humans is unchanged.Getting there needed a small refactor. The code that renders help lived inside
rootHelpFuncand wrote straight to stdout. It is now extracted intoroot.WriteHelp, which takes a writer.rootHelpFunccalls it with stdout exactly as before; error reporting calls it with stderr.How did you test this change?
Given no agent environment variables are set
When I run
gh pr list --badFlagThen I see
unknown flag: --badFlagfollowed byUsage: gh pr list [flags]and the flag list, exactly as beforeGiven
AI_AGENT=copilot-cliis setWhen I run
gh pr list --badFlagThen I see
unknown flag: --badFlagfollowed by the full help: the description,USAGE,ALIASES,FLAGS,INHERITED FLAGS,JSON FIELDS,EXAMPLESandLEARN MOREGiven
AI_AGENT=copilot-cliis setWhen I run
gh pr list --badFlagand redirect stderr awayThen stdout is empty, confirming the help text did not leak onto the wrong stream
I also checked that an error which is not command misuse, such as a network failure, is unaffected in either mode.
Key points
The obvious implementation is to call
cmd.Help(), and that is what the issue suggested. It is wrong here:rootHelpFuncwrites toIOStreams.Out, so the error would go to stderr and the help to stdout, splitting one failure across two streams. Anything consuming stderr, an agent included, would see a truncated message. HenceWriteHelptaking a writer, so everything lands on stderr together.The extraction is a pure move. The rendering logic is unchanged; only the destination is now a parameter.
WriteHelpbypasses per-command help funcs. In practice they all delegate to the same root renderer, exceptgh reference, whose help func pages its output. Paging into stderr during an error would be worse than not paging, so losing it there is the right outcome.The gate is a plain
fullHelp boolrather than the agent name.printErrorhas no reason to care which agent it is, and the boolean is far easier to test.I did not add an acceptance test. #14191 put one under
acceptance/testdata/telemetry/, but there is no natural home for error and help behaviour, and adding a directory plus aTestfunction felt disproportionate for a change this size. The behaviour is unit tested directly. Happy to add one if you disagree.Notes for reviewers
Start with
printErrorininternal/ghcmd/cmd.go, which is where the behaviour changes. Then readpkg/cmd/root/help.goto confirm the extraction is a faithful move.internal/ghcmd/cmd_test.gocovers both modes for flag errors and unknown commands, and asserts that other errors are untouched.One thing worth a reviewer's judgement: the full help is substantially larger than the usage string, roughly 1.7KB to 4.3KB depending on the command. That is the intended trade, one larger response instead of two round trips, but it is a real cost. I measured where those bytes go and opened a follow-up spike to look at trimming the parts an agent cannot act on: https://github.com/github/gh-cli-and-desktop/issues/319. Deliberately out of scope here.
Authorship and follow-up
Who wrote this:
Who answers review comments: