Print full help on command misuse for invoking agents by niik · Pull Request #14198 · cli/cli · GitHub
Skip to content

Print full help on command misuse for invoking agents - #14198

Merged
niik merged 2 commits into
trunkfrom
niik-agent-full-help-on-error
Sep 2, 2026
Merged

Print full help on command misuse for invoking agents#14198
niik merged 2 commits into
trunkfrom
niik-agent-full-help-on-error

Conversation

@niik

@niik niik commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Closes https://github.com/github/gh-cli-and-desktop/issues/56
Follows on from #14191, now merged. This branch has been rebased onto trunk and is a single commit.

Description

When someone mistypes a gh command, we print the error followed by a short usage string: the usage line and a list of flags. A person reads that, spots their mistake, and moves on.

An AI agent cannot. The usage string omits the description, the examples, the JSON fields, and the environment variables, so an agent that guessed a flag wrong has no way to work out what the right one was. It has to spend a second invocation on gh <command> --help to find out. That extra round trip is exactly the kind of thing #14191 started addressing by detecting which agent is driving the CLI.

This change reuses that detection: when an agent is invoking gh and the failure is command misuse, we print the full help text instead of the terse usage string. Behaviour for humans is unchanged.

Getting there needed a small refactor. The code that renders help lived inside rootHelpFunc and wrote straight to stdout. It is now extracted into root.WriteHelp, which takes a writer. rootHelpFunc calls it with stdout exactly as before; error reporting calls it with stderr.

How did you test this change?

Given no agent environment variables are set
When I run gh pr list --badFlag
Then I see unknown flag: --badFlag followed by Usage: gh pr list [flags] and the flag list, exactly as before

Given AI_AGENT=copilot-cli is set
When I run gh pr list --badFlag
Then I see unknown flag: --badFlag followed by the full help: the description, USAGE, ALIASES, FLAGS, INHERITED FLAGS, JSON FIELDS, EXAMPLES and LEARN MORE

Given AI_AGENT=copilot-cli is set
When I run gh pr list --badFlag and redirect stderr away
Then stdout is empty, confirming the help text did not leak onto the wrong stream

I also checked that an error which is not command misuse, such as a network failure, is unaffected in either mode.

Key points

The obvious implementation is to call cmd.Help(), and that is what the issue suggested. It is wrong here: rootHelpFunc writes to IOStreams.Out, so the error would go to stderr and the help to stdout, splitting one failure across two streams. Anything consuming stderr, an agent included, would see a truncated message. Hence WriteHelp taking a writer, so everything lands on stderr together.

The extraction is a pure move. The rendering logic is unchanged; only the destination is now a parameter.

WriteHelp bypasses per-command help funcs. In practice they all delegate to the same root renderer, except gh reference, whose help func pages its output. Paging into stderr during an error would be worse than not paging, so losing it there is the right outcome.

The gate is a plain fullHelp bool rather than the agent name. printError has no reason to care which agent it is, and the boolean is far easier to test.

I did not add an acceptance test. #14191 put one under acceptance/testdata/telemetry/, but there is no natural home for error and help behaviour, and adding a directory plus a Test function felt disproportionate for a change this size. The behaviour is unit tested directly. Happy to add one if you disagree.

Notes for reviewers

Start with printError in internal/ghcmd/cmd.go, which is where the behaviour changes. Then read pkg/cmd/root/help.go to confirm the extraction is a faithful move.

internal/ghcmd/cmd_test.go covers both modes for flag errors and unknown commands, and asserts that other errors are untouched.

One thing worth a reviewer's judgement: the full help is substantially larger than the usage string, roughly 1.7KB to 4.3KB depending on the command. That is the intended trade, one larger response instead of two round trips, but it is a real cost. I measured where those bytes go and opened a follow-up spike to look at trimming the parts an agent cannot act on: https://github.com/github/gh-cli-and-desktop/issues/319. Deliberately out of scope here.

Authorship and follow-up

Who wrote this:

  • A human wrote it.
  • An agent wrote it under close human direction.
  • An agent wrote it independently, and no human has guided the implementation beyond the initial prompt.

Who answers review comments:

  • @username will read and reply directly. Name the account.
  • An agent will draft replies and @niik will read them before they are posted.
  • Nobody has explicitly committed to replying.

Base automatically changed from niik-agent-spinner-output to trunk August 21, 2026 11:59
When an agent misuses a command, the terse usage string does not carry
the examples, JSON fields or environment variables it needs to correct
itself, forcing a second `--help` invocation.

Extract the help renderer out of rootHelpFunc so it can target any
writer, then use it from printError when an agent is detected. Rendering
directly to stderr avoids cmd.Help(), which writes to stdout and would
otherwise split a single failure across two streams.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@niik
niik force-pushed the niik-agent-full-help-on-error branch from 0bfd4ba to 3f2ae26 Compare August 21, 2026 12:02
@niik
niik marked this pull request as ready for review August 21, 2026 12:02
@niik
niik requested a review from a team as a code owner August 21, 2026 12:02
@niik
niik requested review from BagToad and a balanced review from Copilot August 21, 2026 12:02

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Prints full command help on misuse when an AI agent invokes gh, while preserving terse usage for humans.

Changes:

  • Extracts help rendering into a writer-aware function.
  • Sends agent-facing error help entirely to stderr.
  • Adds unit coverage for full-help and unchanged-error paths.
Show a summary per file
File Description
pkg/cmd/root/help.go Extracts reusable help rendering.
internal/ghcmd/cmd.go Enables full help for detected agents.
internal/ghcmd/cmd_test.go Tests terse and full error output.

Review details

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

  • Files reviewed: 3/3 changed files
  • Comments generated: 1
  • Review effort level: Balanced

Comment thread internal/ghcmd/cmd.go

@babakks babakks left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tackling this, @niik! 🙏 Nice write-up in the description, and the WriteHelp extraction reads cleanly.

One thing worth another look before this lands: when an agent is detected, WriteHelp fully replaces the terse usage on root-level misuse, so the flat Available commands: block that humans see gets dropped rather than augmented. I left an inline note with a repro and a suggested tweak to append the terse usage in the fullHelp branch (plus a heads-up that cmd.UsageString() returns empty for gh commands, so it can't be captured the obvious way). I dug into this with Copilot, so please sanity check it on your end.

Everything else looks good to me. 🎉

Comment thread internal/ghcmd/cmd.go

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

internal/ghcmd/cmd.goprintError, fullHelp branch (~L306-L313)

Heads-up: I noticed this discrepancy while reviewing and asked Copilot to dig into the case, so @niik please sanity check this one more time before acting on it.

Repro (compare the two):

go run ./cmd/gh xyz
AI_AGENT=foo go run ./cmd/gh xyz

The difference: the first (human) prints the terse Usage: ... Available commands: ..., while the second (agent) prints WriteHelp's full grouped help instead, so the flat Available commands: block is gone rather than just having extra help added to it.


idea: In fullHelp mode we should append the terse usage so agents keep everything a human sees. Right now WriteHelp replaces cmd.UsageString(), so the flat Available commands: block (and the flag list on leaf commands) is dropped for agents on root-level misuse.

Verified behavior:

  • gh zzqqxx (root unknown command) reaches printError. Human mode shows the terse Usage: ... Available commands: ...; agent mode shows WriteHelp's grouped rendering instead, so the literal Available commands: block is gone.
  • gh pr zzqqxx (nested unknown subcommand) never reaches printError at all - it's handled earlier by nestedSuggestFunc, so it stays terse in both modes.

Heads-up on the "capture cmd.UsageString() first" idea: for real gh commands cmd.UsageString() returns an empty string and writes the usage to IOStreams.ErrOut as a side effect, because gh's SetUsageFunc (root.go:115) writes straight to the captured ErrOut and ignores the buffer Cobra swaps in. Empirically it returns len 0 and leaks ~946 bytes to ErrOut. So capturing it won't capture anything. Instead expose the terse renderer and call it directly.

I also checked the "WriteHelp mutates Cobra state that UsageString reads" theory - it does not; UsageString returns the same thing before and after WriteHelp. So no capture/reorder is needed; we can just append.

Suggested approach (spans two files, so not a single suggestion block):

In pkg/cmd/root/help.go, expose the terse renderer:

// WriteUsage renders the terse usage string for command to w. cmd.UsageString()
// cannot be used for this because gh sets a usage function that writes straight
// to IOStreams.ErrOut, so UsageString returns an empty string and leaks the
// usage onto ErrOut as a side effect.
func WriteUsage(w io.Writer, command *cobra.Command) error {
	return rootUsageFunc(w, command)
}

In internal/ghcmd/cmd.go, append it in the fullHelp branch:

if fullHelp {
	// Render into out rather than calling cmd.Help(), which would send
	// the help text to stdout and split a single failure across two streams.
	root.WriteHelp(out, cs, cmd)
	// Also append the terse usage so agents keep everything a human sees,
	// including the "Available commands" list that WriteHelp replaces with
	// its grouped rendering.
	fmt.Fprintln(out)
	_ = root.WriteUsage(out, cmd)
	fmt.Fprintln(out)
	return
}

Verified output for AI_AGENT=foo gh zzqqxx: full help, then Usage: gh <command> <subcommand> [flags] + Available commands: list appended at the end.

Note: this will change the expected output in the newly added Test_printError full-help cases, so those wantOut values need updating to include the appended usage.


Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for flagging this, and for the nudge to sanity check — I dug in properly and had the findings independently verified before acting.

Your mechanism is right on both counts. cmd.UsageString() does return "" for real gh commands: SetUsageFunc (root.go:115) writes straight to the captured ErrOut, so the usage text you see is a side effect and the fmt.Fprintln(out, cmd.UsageString()) contributes only a blank line. Confirmed on gh pr list --badFlag: 972 bytes to stderr, exactly one Usage: line and one --web line, terminated by a lone \n — if the string were actually being returned we'd see it twice. Worth noting the existing human path only works because out and ErrOut are the same stream.

And gh pr zzqqxx does bypass printError entirely, byte-identical in both modes, as you said.

Where I land differently is on the consequence, so let me show the numbers rather than just assert it. Comparing information content rather than the literal string, for gh zzqqxx:

  • human: 36 bare command names (429 bytes)
  • agent: the same 36 names, each with a description, plus group structure and the 8 help topics (2,837 bytes)

The set difference human-minus-agent is empty. The same holds on leaf commands, which I think is the part worth double-checking against the "(and the flag list on leaf commands) is dropped" note: for gh pr list --badFlag, agent mode contains all 14 flags the human sees, plus --help and --repo, plus the JSON fields and examples. Nothing is dropped in either case — the grouped rendering is a strict superset, not a replacement that loses something.

So appending the terse usage would reprint the same command names and flags in the less informative form: roughly +14% at root, and about +38% on gh pr create --badFlag where it would duplicate the entire FLAGS block. Given the token-cost pressure this epic is tracking, I'd rather not pay that for content already present.

One further reason to avoid WriteUsage specifically: calling rootUsageFunc directly would bypass the alias resolution added in 3a73af2, re-introducing the alias-stub problem for gh <alias> --badFlag that the other review thread caught.

That said, I think you're onto something real with the nested case. Because nestedSuggestFunc handles gh pr zzqqxx before printError, agents get the terse list and no full help there — which is arguably the more valuable gap, since that's a much more common agent mistake than root-level misuse. It's outside this PR's changed path, but I'd be happy to open a follow-up issue for it if you agree.

Happy to be overruled on the append if you still see a case I'm missing.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for looking deeper, @niik. I agree that this is so hacky already and touching it is a scope creep. Let's leave it as is for now.

A user-defined alias is a stub command: its own help carries no flags or
examples, only "Alias for ...". NewCmdRoot works around this by pointing the
alias's usage and help funcs at the command it expands to, but printError
calls root.WriteHelp directly and so bypassed that redirection.

The result was that agents got strictly worse output than humans for aliases:
`gh prs --badFlag` rendered 384 bytes of alias stub with none of `pr list`'s
flags, leaving the agent no way to correct itself. Resolve the alias inside
WriteHelp so the behaviour cannot be missed by a future call site.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d6afb36c-b3b5-472f-b540-025e38395b79
@niik
niik requested a review from babakks September 1, 2026 09:53
@niik
niik merged commit 71eac35 into trunk Sep 2, 2026
11 checks passed
@niik
niik deleted the niik-agent-full-help-on-error branch September 2, 2026 09:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants