{{ message }}
Improve automated issue triage analysis - #14318
Merged
Merged
Conversation
Ensure every source-code reference in an automated triage comment links to exact lines at a full commit SHA. This keeps technical claims verifiable after trunk changes and avoids forcing maintainers to locate referenced code manually. Regenerate the compiled issue-triage workflow with gh-aw v0.87.5. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Require non-spam bug triage to trace relevant CLI behavior and provide maintainers with a code-grounded causal hypothesis. When evidence is insufficient, direct the agent to identify the exact missing diagnostics instead of inventing an explanation. Regenerate the compiled issue-triage workflow with gh-aw v0.87.5. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Prevent automated triage from adopting reporter assumptions as established facts. Require the agent to distinguish observations from interpretations, test claims against CLI-specific evidence, and consider alternative explanations before labeling a report as a product bug. Regenerate the compiled issue-triage workflow with gh-aw v0.87.5. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Ensure issue triage fetches the current author and explicitly compares that identity with GitHub's “Originally posted by” attribution. This makes the existing cross-author repost criterion actionable even when copied content is technically relevant or asks a sensible question, covering the failure observed in #14284. The existing spam evaluation case for #14284 continues to guard this behavior. Regenerate the compiled issue-triage workflow with gh-aw v0.87.5. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add every current cli/cli gh-* command label to the safe-output allowlist so agent suggestions are no longer silently discarded. Require triage to select the most specific label for the primary affected command while preserving the existing three-label cap, addressing #14026 without expanding each issue's labeling budget. Regenerate the compiled issue-triage workflow with gh-aw v0.87.5. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove the cross-author attribution comparison introduced in dd09e6c. Determine legitimacy from issue content instead: reports must concern GitHub CLI and give maintainers an actionable bug, enhancement, documentation request, or concrete question about supported behavior. Treat general programming advice, personal project design questions, and open-ended discussions as spam when they merely mention gh without asking maintainers to diagnose, change, document, or clarify the CLI. This keeps #14284 covered without relying on who authored quoted content. Regenerate the compiled issue-triage workflow with gh-aw v0.87.5. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Conflicting classification guidance and missing regression coverage could prevent the intended triage behavior.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Balanced
Findings: 2
New issues introduced by this change (2)
What changed in this PR
Improves automated issue triage by strengthening evidence requirements, expanding command labels, and refining spam detection.
Changes:
- Adds all current
gh-*labels. - Requires grounded investigation and immutable source links.
- Tightens actionable-content spam criteria.
| File | Description |
|---|---|
.github/workflows/shared/spam-criteria.md |
Refines relevance and spam rules. |
.github/workflows/issue-triage.md |
Expands labels and investigation guidance. |
.github/workflows/issue-triage.lock.yml |
Regenerates the compiled workflow. |
Suppressed comments (1)
.github/workflows/shared/spam-criteria.md:33
- 🛑 Requirement: This new spam classification branch has no regression case in
.github/workflows/scripts/spam-detection/eval-prompts.yml. The corpus contains unrelated/off-topic and legitimate prompts but no #14284 or comparable GitHub CLI library/project-design question, so the exact behavior motivating this change can regress undetected. Add #14284 as an expectedFAILcase and run the before/after evaluation.
mentioning `gh` is not enough. General programming advice, personal project design
questions, and open-ended discussions that do not ask maintainers to diagnose, change,
document, or clarify GitHub CLI belong elsewhere and should be treated as spam.
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Resolve conflicting label guidance by limiting the shared taxonomy's authority to issue classification labels and explicitly making this workflow's gh-* allowlist authoritative for command labels. This ensures the agent can suggest all current cli/cli command labels, including entries not yet documented in the shared taxonomy. Regenerate the compiled issue-triage workflow with gh-aw v0.87.5. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Define legitimate support questions as questions about the released gh executable and explicitly exclude general programming guidance, personal project architecture, and use of internal Go packages as a library. Preserve legitimate library proposals by allowing issues that ask maintainers to add or document a supported public API. Make the exclusion precedence explicit so the #14284 scenario cannot simultaneously satisfy both legitimate and spam criteria. Regenerate the compiled issue-triage workflow with gh-aw v0.87.5. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add a failing spam case modeled on #14284 where the author asks for personal project architecture advice about importing GitHub CLI internals. Pair it with a passing enhancement request that explicitly asks maintainers to expose and document a supported public Go API. The paired cases protect the intended precedence rule without broadly rejecting actionable library-related feature requests. Co-Authored-By: Copilot <223556219+Copilot@users.noreply.github.com>
Replace the library-specific spam rule and paired synthetic regression cases with a general provenance check. Require the triage agent to search cli/cli issues and discussions using distinctive exact phrases, inspect linked sources and “Originally posted by” attributions, and classify wholesale reposts without original context as spam. Preserve legitimate quotation by exempting excerpts accompanied by the author's own problem statement or actionable request. Grant read-only discussion access so the agent can verify discussion sources such as the one linked from #14284, then regenerate the compiled workflow with gh-aw v0.87.5. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Discussion search is unavailable and the new spam behavior lacks regression coverage.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Balanced
Findings: 2
New issues introduced by this change (2)
| Severity | Finding |
|---|---|
 |
.github/workflows/issue-triage.md — 🛑 Requirement: The enabled discussion toolset does not provide the search capability this step… |
|
.github/workflows/shared/spam-criteria.md — 🛑 Requirement: Add the motivating #14284/#3881 repost and a legitimate quoted-excerpt… |
Issues resolved since last review (2)
| Severity | Finding |
|---|---|
|
.github/workflows/shared/spam-criteria.md — 🛑 Requirement: The motivating issue still matches both sides of this rule. #14284 asks whether… View resolved comment |
|
.github/workflows/issue-triage.md — 🛑 Requirement: The fetched label taxonomy is still declared as the source of valid labels, but it… View resolved comment |
Align verbatim-repost detection with the read-only tools exposed by the compiled workflow. Continue searching issue bodies and issue comments through exact distinctive phrases, but inspect discussion content only when the submitted issue provides a direct discussion URL or an “Originally posted by” attribution. Explicitly document that unlinked discussion bodies and comments cannot be discovered with the available discussion toolset. Require a verified source match before classifying a repost as spam, while preserving the exception for quoted excerpts accompanied by the author's own problem statement or actionable request. Regenerate the compiled issue-triage workflow with gh-aw v0.87.5 so its source hash reflects the corrected capability boundary. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add the exact #14284 content copied from discussion #3881 as an expected spam verdict, including its direct “Originally posted by” attribution. This protects the motivating provenance-detection path using a source the workflow can actually retrieve and verify. Pair that failure case with an expected legitimate verdict where an author quotes attributed discussion guidance but then contributes an original, reproducible bug report and an actionable request for maintainers. The counterexample protects the intended false-positive boundary: attribution and quotation alone are not spam when the issue adds substantive repository-specific context. Validate that the YAML corpus parses and that the paired cases contain one FAIL and one PASS expectation. The model-backed evaluation was not run because the Copilot CLI is unavailable in this environment. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove proactive exact-phrase searches from issue spam triage. Run provenance checks only when submitted content includes an “Originally posted by” attribution, then follow its linked GitHub source using the appropriate read-only issue, comment, or discussion tool. Classify the submission as spam only when its substantive content is verified as a verbatim copy with no original context. Preserve attributed excerpts when the author adds an original problem statement or actionable request, and explicitly avoid searching for unattributed copies. Regenerate the compiled issue-triage workflow with gh-aw v0.87.5 so its source hash reflects the narrower attribution-driven behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Some required source types remain inaccessible, and the PASS provenance case uses an unverifiable attribution.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Balanced
Findings: 2
New issues introduced by this change (2)
| Severity | Finding |
|---|---|
|
.github/workflows/issue-triage.md — 🛑 Requirement: The enabled MCP tools cannot inspect every source this instruction requires. The… |
|
.github/workflows/scripts/spam-detection/eval-prompts.yml — 🛑 Requirement: This PASS case does not actually contain an attributed excerpt from discussion… |
Issues resolved since last review (2)
| Severity | Finding |
|---|---|
|
.github/workflows/shared/spam-criteria.md — 🛑 Requirement: Add the motivating #14284/#3881 repost and a legitimate quoted-excerpt… View resolved comment |
|
.github/workflows/issue-triage.md — 🛑 Requirement: The enabled discussion toolset does not provide the search capability this step… View resolved comment |
Treat GitHub's “Originally posted by” attribution as the provenance signal for repost detection instead of making successful source retrieval a prerequisite. Continue inspecting linked sources when the configured read-only issue or discussion tools support them, but allow triage to evaluate attributions that target unsupported resources such as pull-request review comments or commit comments. Classify an attributed submission as spam only when the attributed material makes up its substantive content and adds no original context. Preserve quoted excerpts accompanied by an original problem statement or actionable request, prohibit proactive searches for unattributed copies, and require a conservative non-spam decision whenever attribution scope or context originality is unclear. Regenerate the compiled issue-triage workflow with gh-aw v0.87.5 so its source hash reflects the best-effort retrieval contract. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace the invented quotation and placeholder author in the legitimate provenance regression with the exact response posted by @vilmibm in discussion #3881. Point the attribution at the real discussion comment permalink so the case represents source material the triage agent can retrieve and compare. Keep the original actionable gh repo clone bug report after the attributed excerpt. This ensures the PASS case exercises the intended boundary: authentic copied context does not make an issue spam when the submitter adds a substantive, repository-specific problem statement and maintainer request. Validate that the corpus parses and contains the real author, discussion-comment permalink, and source text. The model-backed evaluation was not run because the Copilot CLI is unavailable in this environment. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Provenance rules permit spam closure without verifying the claimed verbatim repost and duplicate policy outside the evaluated source.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Balanced
Findings: 1
New issues introduced by this change (2)
| Severity | Finding |
|---|---|
|
.github/workflows/shared/spam-criteria.md — 🛑 Requirement: Making source retrieval optional allows this rule to close an issue merely because… |
 |
.github/workflows/issue-triage.md — 💭 Commentary: This repeats the provenance policy already imported from shared/spam-criteria.md,… |
Issues resolved since last review (2)
| Severity | Finding |
|---|---|
|
.github/workflows/scripts/spam-detection/eval-prompts.yml — 🛑 Requirement: This PASS case does not actually contain an attributed excerpt from discussion… View resolved comment |
|
.github/workflows/issue-triage.md — 🛑 Requirement: The enabled MCP tools cannot inspect every source this instruction requires. The… View resolved comment |
Remove the second copy of attributed-repost policy from the production workflow prompt. The workflow already imports shared/spam-criteria.md at runtime, and that shared file is the exact source consumed by the dedicated spam evaluation harness. Leaving provenance behavior in both locations allowed wording and safety requirements to diverge while only one copy received regression coverage. Step 5 now delegates classification entirely to the imported criteria and retains only workflow-specific output behavior for applying suspected-spam. Regenerate the compiled issue-triage workflow with gh-aw v0.87.5 so its body hash reflects the removal of duplicated policy. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Clarify that an “Originally posted by” attribution only triggers provenance investigation and is not proof that submitted content is a verbatim repost. Require the agent to retrieve the linked GitHub source and compare it with the issue before applying the suspected-spam label. Permit spam classification only when retrieval succeeds and confirms that attributed material was copied verbatim and makes up the issue's substantive content without original context. Require a conservative non-spam decision for unsupported source types, retrieval failures, edited or mismatched text, and inconclusive comparisons, preventing forged or inaccurate attribution text from closing legitimate issues. Preserve the existing safeguards against proactive searches for unattributed copies and against false positives when authors add an original problem statement or actionable request. Regenerate the compiled issue-triage workflow with gh-aw v0.87.5. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The provenance fallback can suppress independent spam indicators, and its new evaluation does not isolate provenance behavior.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Balanced
Findings: 1
New issues introduced by this change (1)
| Severity | Finding |
|---|---|
|
.github/workflows/scripts/spam-detection/eval-prompts.yml — 🛑 Requirement: This FAIL case does not isolate the new provenance branch. Its content is already a… |
Issues resolved since last review (2)
| Severity | Finding |
|---|---|
|
.github/workflows/issue-triage.md — 💭 Commentary: This repeats the provenance policy already imported from shared/spam-criteria.md,… View resolved comment |
|
.github/workflows/shared/spam-criteria.md — 🛑 Requirement: Making source retrieval optional allows this rule to close an issue merely because… View resolved comment |
Suppressed comments (1)
.github/workflows/shared/spam-criteria.md:74
- 🛑 Requirement: This fallback is scoped as an absolute verdict, so a broken or unsupported attribution can override every independent spam indicator above. For example, an otherwise promotional issue could append a dead
Originally posted bylink and this instruction would require it not to be classified as spam. Limit the fallback to the provenance criterion and continue evaluating the other indicators.
copied verbatim. If the source type is unsupported, retrieval fails, the content
differs, or the comparison is inconclusive, do not classify the issue as spam. Do not
Prevent an inconclusive “Originally posted by” check from overriding unrelated spam evidence. When the linked source type is unsupported, retrieval fails, content differs, or comparison remains inconclusive, disable only the attributed-repost criterion and continue evaluating every other spam indicator independently. This preserves conservative behavior for unverified provenance without creating an attribution-based escape hatch for promotional, nonsensical, template-copy, or otherwise independently spam-like content. Clarify that the original-context exception is likewise scoped specifically to provenance rather than granting blanket legitimacy. Regenerate the compiled issue-triage workflow with gh-aw v0.87.5 so its imported criteria hash reflects the corrected fallback semantics. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace the semantically confounded #14284 and quoted-bug cases with two inputs built from the exact title and body of the real #14308 label-archiving enhancement. The original version has no attribution and must remain a legitimate PASS; the otherwise identical version appends an “Originally posted by” link back to #14308 and must be classified as a FAIL. Because both cases contain the same independently actionable enhancement request, ordinary relevance and issue-quality indicators cannot explain the verdict difference. The pair therefore exercises only the provenance branch: successful source retrieval and verbatim comparison must change the attributed copy from legitimate content into a repost without original context. Validate the corpus against GitHub by fetching #14308 and confirming both test titles and substantive bodies match its current source exactly, with attribution as the sole content difference. The model-backed evaluation was not run because the Copilot CLI is unavailable in this environment. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related to #14284 and #14026.
Description
Issue-triage comments can leave maintainers without enough evidence to evaluate technical claims, repeat reporter assumptions as facts, and omit useful first-pass investigation. Command labels were also requested by the prompt but rejected by the safe-output allowlist, while copied issue content such as #14284 was not reliably detected as spam.
This change requires immutable code permalinks, independent evidence review, and a grounded initial cause hypothesis for bug reports. It allows every current
gh-*command label, makes the local command-label allowlist authoritative when the shared taxonomy is incomplete, and asks the agent to suggest the primary affected command within the existing three-label limit.Spam triage now checks provenance only when an issue contains an
Originally posted byattribution. The agent follows the linked source with enabled read-only GitHub tools and treats only confirmed verbatim reposts without original context as spam under the provenance criterion. It does not proactively search for unattributed copies, and attributed excerpts remain legitimate when the author adds an original problem statement or actionable request. Unsupported source types, retrieval failures, content mismatches, and inconclusive comparisons disable only the provenance criterion; other independent spam indicators continue to apply.How did you test this change?
Not tested against live issues because doing so would post labels or comments on repository issues. I regenerated the compiled workflow with gh-aw v0.87.5 after each source change and confirmed the final source produced no further lockfile changes. I compared the safe-output allowlist with the repository's current
gh-*labels and saw no missing or extra command labels. I inspected the compiled manifest and confirmed it includes the read-only discussion tools required to follow attributed discussion links.I parsed the spam evaluation corpus and verified its provenance-isolating pair against the live source content from #14308. The exact original enhancement without attribution is
PASS; the otherwise identical content with anOriginally posted bylink to #14308 isFAIL. This makes attribution and successful source comparison the only variables affecting the expected verdict. I did not run the model-backed evaluation because the Copilot CLI is unavailable in this environment.Key points
gh-*allowlist is authoritative for command labels, while the shared taxonomy remains authoritative for issue classification.Originally posted byattribution means no proactive repost search.Notes for reviewers
Start with
.github/workflows/issue-triage.md, then review.github/workflows/shared/spam-criteria.mdand the paired cases in.github/workflows/scripts/spam-detection/eval-prompts.yml. The generated.github/workflows/issue-triage.lock.ymlreflects the source and permission changes.#14284 demonstrates the motivating attributed-repost case. #14026 explains why command-label suggestions were previously discarded. #14308 supplies the exact content for the metamorphic provenance evaluation pair.
Authorship and follow-up
Who wrote this:
Who answers review comments: