Introduce testscript acceptance tests generally, and for the PR command specifically by williammartin · Pull Request #9745 · cli/cli · GitHub
Skip to content

Introduce testscript acceptance tests generally, and for the PR command specifically#9745

Merged
williammartin merged 29 commits into
trunkfrom
wm/testscript-pr
Oct 14, 2024
Merged

Introduce testscript acceptance tests generally, and for the PR command specifically#9745
williammartin merged 29 commits into
trunkfrom
wm/testscript-pr

Conversation

@williammartin

@williammartin williammartin commented Oct 11, 2024

Copy link
Copy Markdown
Member

Description

The goal of this work is to have a set of automated tests that we can use to point gh at a real GitHub host in order to have some sense of blackbox behavioural validation. This PR introduces the use of testscript using the gh pr command as a motivating example.

It is a non-goal right now to have these running as part of our CI suite.

Reviewer Notes

The README contains most information that I won't repeat in this description.

One big change here is that in order to avoid building gh and to enable code coverage, I had to move the RealMain that used to be in the cmd/gh/main.go into its own package called ghcmd (main packages can't be imported), and to export the function. I can't really see much of an issue with this, it's actually pretty idiomatic.

Accuweaty24 added a commit to Accuweaty24/cli that referenced this pull request Oct 11, 2024
Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.
@probablycorey @matschaffer @digitalfu cli#9745 cli#9721 cli#9728 cli#9746 №accuweaty24 #

@jtmcg jtmcg left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Mostly just some questions 🙂

Comment thread acceptance/README.md Outdated
Comment thread acceptance/README.md
Comment thread acceptance/README.md
Comment thread acceptance/README.md Outdated
Comment thread acceptance/README.md

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️

Comment thread acceptance/testdata/pr/pr-merge-merge-strategy.txtar
Comment thread acceptance/acceptance_test.go
Comment thread acceptance/README.md
> [!WARNING]
> Verbose mode dumps the `testscript` environment variables, including the `GH_TOKEN`, so be careful.

By default `testscript` removes the directory in which it was running the script, and if you've been a conscientious engineer, you should be cleaning up resources using the `defer` statement. However, this can be an impediment to debugging. As such you can set `GH_ACCEPTANCE_PRESERVE_WORK_DIR=true` and `GH_ACCEPTANCE_SKIP_DEFER=true` to skip these cleanup steps.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went back and forth a bit on having a single env var that could have different values but I decided until we get more usage and understand what the enumeration of values might be, it was better to be explicit.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I recognise that this is almost identical to pr-create-basic.txtar. I'm being quite intentional about keeping them separate. Just because they look the same now doesn't mean they will always look the same.

@andyfeller andyfeller left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partial review while reading through PR acceptance tests.

Comment thread acceptance/README.md Outdated
Comment thread acceptance/README.md Outdated
Comment thread acceptance/README.md Outdated
Comment thread acceptance/acceptance_test.go

@andyfeller andyfeller left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After finishing the 2nd half, I only have questions about nuance rather than anything blocking.

Comment thread acceptance/acceptance_test.go
Comment thread acceptance/testdata/pr/pr-create-with-metadata.txtar Outdated
@williammartin williammartin merged commit c657cfa into trunk Oct 14, 2024
@williammartin williammartin deleted the wm/testscript-pr branch October 14, 2024 18:23
tmeijn pushed a commit to tmeijn/dotfiles that referenced this pull request Oct 17, 2024
This MR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [cli/cli](https://github.com/cli/cli) | minor | `v2.58.0` -> `v2.59.0` |

MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot).

**Proposed changes to behavior should be submitted there as MRs.**

---

### Release Notes

<details>
<summary>cli/cli (cli/cli)</summary>

### [`v2.59.0`](https://github.com/cli/cli/releases/tag/v2.59.0): GitHub CLI 2.59.0

[Compare Source](cli/cli@v2.58.0...v2.59.0)

#### What's Changed

-   Allow community submitted design work by [@&#8203;BagToad](https://github.com/BagToad) in cli/cli#9683
-   Improve `SECURITY.md` with expectations for privately reported vulnerabilities by [@&#8203;BagToad](https://github.com/BagToad) in cli/cli#9687
-   Emit a log message when extension installation falls back to a `darwin-amd64` binary on an Apple Silicon macOS device by [@&#8203;timrogers](https://github.com/timrogers) in cli/cli#9650
-   Print the login URL even when opening a browser by [@&#8203;ulfjack](https://github.com/ulfjack) in cli/cli#7091
-   configurable maxwidth for markdown WithWrap() by [@&#8203;smemsh](https://github.com/smemsh) in cli/cli#9626
-   Handle errors when parsing hostname in auth flow by [@&#8203;BagToad](https://github.com/BagToad) in cli/cli#9729
-   Add `repo license list/view` and `repo gitignore list/view` by [@&#8203;BagToad](https://github.com/BagToad) in cli/cli#9721
-   Introduce testscript acceptance tests generally, and for the MR command specifically by [@&#8203;williammartin](https://github.com/williammartin) in cli/cli#9745
-   Support `GH_ACCEPTANCE_SCRIPT` env var to target a single script by [@&#8203;williammartin](https://github.com/williammartin) in cli/cli#9756
-   Ensure Acceptance defer failures are debuggable by [@&#8203;williammartin](https://github.com/williammartin) in cli/cli#9754
-   Add acceptance task to makefile by [@&#8203;williammartin](https://github.com/williammartin) in cli/cli#9748
-   Add Acceptance tests for `issue` command by [@&#8203;williammartin](https://github.com/williammartin) in cli/cli#9757
-   Update IsEnterprise and IsTenancy for orthogonality using go-gh by [@&#8203;jtmcg](https://github.com/jtmcg) in cli/cli#9755
-   Supporting filtering on `gist list` by [@&#8203;heaths](https://github.com/heaths) in cli/cli#9728

#### New Contributors

-   [@&#8203;ulfjack](https://github.com/ulfjack) made their first contribution in cli/cli#7091
-   [@&#8203;smemsh](https://github.com/smemsh) made their first contribution in cli/cli#9626

**Full Changelog**: cli/cli@v2.58.0...v2.59.0

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this MR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this MR, check this box

---

This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy40NDAuNyIsInVwZGF0ZWRJblZlciI6IjM3LjQ0MC43IiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJSZW5vdmF0ZSBCb3QiXX0=-->
proggarapsody added a commit to proggarapsody/bitbottle that referenced this pull request Jun 18, 2026
Five loop-sized PRD scopes implementing the #658 follow-ups, off-the-shelf
tooling (go-vcr, kin-openapi, gh-style acceptance per cli/cli#9745):
- #661 VCR-CASSETTES, #662 OPENAPI-VALIDATE (foundations)
- #663 ACCEPTANCE-LIVE-WIRE (makes documented Tier 6 real)
- #664 E2E-QUEUE-FEEDBACK, #665 SMOKE-METRIC (consume the live signal)

Each PRD dogfoods the Assumptions & Evidence gate. backend-quirks.md gains
a 'Planned hardening' pointer so the ledger links to the enforcement work.

refs #658
proggarapsody added a commit to proggarapsody/bitbottle that referenced this pull request Jun 18, 2026
…post-mortem)

* docs(workflow): backend-quirks ledger + spec-time evidence gate (#655 post-mortem)

The iteration loop's verification chain (TDD fakes, design-judge, CI) all
validate the implementer's assumptions against each other, never against a
real backend. A wrong assumption about Server behavior is reproduced
identically in code, fake, and test — all three agree and ship the bug.
That is how #655 shipped: pr edit wiped reviewers and pr request-review
400'd, while the full-object-PUT + version rule was already known and
correctly applied three functions away in the same file.

- docs/backend-quirks.md: new append-only ledger of real Server/Cloud
  behaviors no linter or hand-written fake can infer (BQ-1..BQ-5), seeded
  from #655 and the existing httpx.Transport policies, each with a code
  citation of the correct AND broken pattern. Every future backend bug
  appends a row in its fix PR.
- README §2: write/new-API PRDs now require an Assumptions & Evidence
  section; every backend-behavior claim is CITED, LEDGER, or a blocking
  ASSUMED-UNVERIFIED that a reality probe must settle before TDD. These
  PRDs are reclassified mechanical -> judgment.
- README §3: TDD subagent brief must consult the ledger and assert on the
  captured request, not just stdout.
- pre-merge-check §6a: stdout-only write-op test, or a violated quirk, is
  a design-judge BLOCKER.
- quickref + agent-primer: model-tier reclassification, anti-patterns,
  per-write consult pointer.

refs #655

* docs(backlog): queue VERIFICATION-LOOP-HARDENING initiative (#661-#665)

Five loop-sized PRD scopes implementing the #658 follow-ups, off-the-shelf
tooling (go-vcr, kin-openapi, gh-style acceptance per cli/cli#9745):
- #661 VCR-CASSETTES, #662 OPENAPI-VALIDATE (foundations)
- #663 ACCEPTANCE-LIVE-WIRE (makes documented Tier 6 real)
- #664 E2E-QUEUE-FEEDBACK, #665 SMOKE-METRIC (consume the live signal)

Each PRD dogfoods the Assumptions & Evidence gate. backend-quirks.md gains
a 'Planned hardening' pointer so the ledger links to the enforcement work.

refs #658

* docs(reports): archive stream 178-187 + cycles 158-187 analysis
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants