Introduce testscript acceptance tests generally, and for the PR command specifically#9745
Conversation
0e1e364 to
f9b2499
Compare
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. @probablycorey @matschaffer @digitalfu cli#9745 cli#9721 cli#9728 cli#9746 №accuweaty24 #
jtmcg
left a comment
There was a problem hiding this comment.
Looks good. Mostly just some questions 🙂
| > [!WARNING] | ||
| > Verbose mode dumps the `testscript` environment variables, including the `GH_TOKEN`, so be careful. | ||
|
|
||
| By default `testscript` removes the directory in which it was running the script, and if you've been a conscientious engineer, you should be cleaning up resources using the `defer` statement. However, this can be an impediment to debugging. As such you can set `GH_ACCEPTANCE_PRESERVE_WORK_DIR=true` and `GH_ACCEPTANCE_SKIP_DEFER=true` to skip these cleanup steps. |
There was a problem hiding this comment.
I went back and forth a bit on having a single env var that could have different values but I decided until we get more usage and understand what the enumeration of values might be, it was better to be explicit.
There was a problem hiding this comment.
I recognise that this is almost identical to pr-create-basic.txtar. I'm being quite intentional about keeping them separate. Just because they look the same now doesn't mean they will always look the same.
andyfeller
left a comment
There was a problem hiding this comment.
Partial review while reading through PR acceptance tests.
Co-authored-by: Andy Feller <andyfeller@github.com>
Co-authored-by: Andy Feller <andyfeller@github.com>
Co-authored-by: Andy Feller <andyfeller@github.com>
andyfeller
left a comment
There was a problem hiding this comment.
After finishing the 2nd half, I only have questions about nuance rather than anything blocking.
This MR contains the following updates: | Package | Update | Change | |---|---|---| | [cli/cli](https://github.com/cli/cli) | minor | `v2.58.0` -> `v2.59.0` | MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot). **Proposed changes to behavior should be submitted there as MRs.** --- ### Release Notes <details> <summary>cli/cli (cli/cli)</summary> ### [`v2.59.0`](https://github.com/cli/cli/releases/tag/v2.59.0): GitHub CLI 2.59.0 [Compare Source](cli/cli@v2.58.0...v2.59.0) #### What's Changed - Allow community submitted design work by [@​BagToad](https://github.com/BagToad) in cli/cli#9683 - Improve `SECURITY.md` with expectations for privately reported vulnerabilities by [@​BagToad](https://github.com/BagToad) in cli/cli#9687 - Emit a log message when extension installation falls back to a `darwin-amd64` binary on an Apple Silicon macOS device by [@​timrogers](https://github.com/timrogers) in cli/cli#9650 - Print the login URL even when opening a browser by [@​ulfjack](https://github.com/ulfjack) in cli/cli#7091 - configurable maxwidth for markdown WithWrap() by [@​smemsh](https://github.com/smemsh) in cli/cli#9626 - Handle errors when parsing hostname in auth flow by [@​BagToad](https://github.com/BagToad) in cli/cli#9729 - Add `repo license list/view` and `repo gitignore list/view` by [@​BagToad](https://github.com/BagToad) in cli/cli#9721 - Introduce testscript acceptance tests generally, and for the MR command specifically by [@​williammartin](https://github.com/williammartin) in cli/cli#9745 - Support `GH_ACCEPTANCE_SCRIPT` env var to target a single script by [@​williammartin](https://github.com/williammartin) in cli/cli#9756 - Ensure Acceptance defer failures are debuggable by [@​williammartin](https://github.com/williammartin) in cli/cli#9754 - Add acceptance task to makefile by [@​williammartin](https://github.com/williammartin) in cli/cli#9748 - Add Acceptance tests for `issue` command by [@​williammartin](https://github.com/williammartin) in cli/cli#9757 - Update IsEnterprise and IsTenancy for orthogonality using go-gh by [@​jtmcg](https://github.com/jtmcg) in cli/cli#9755 - Supporting filtering on `gist list` by [@​heaths](https://github.com/heaths) in cli/cli#9728 #### New Contributors - [@​ulfjack](https://github.com/ulfjack) made their first contribution in cli/cli#7091 - [@​smemsh](https://github.com/smemsh) made their first contribution in cli/cli#9626 **Full Changelog**: cli/cli@v2.58.0...v2.59.0 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this MR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this MR, check this box --- This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy40NDAuNyIsInVwZGF0ZWRJblZlciI6IjM3LjQ0MC43IiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJSZW5vdmF0ZSBCb3QiXX0=-->
Five loop-sized PRD scopes implementing the #658 follow-ups, off-the-shelf tooling (go-vcr, kin-openapi, gh-style acceptance per cli/cli#9745): - #661 VCR-CASSETTES, #662 OPENAPI-VALIDATE (foundations) - #663 ACCEPTANCE-LIVE-WIRE (makes documented Tier 6 real) - #664 E2E-QUEUE-FEEDBACK, #665 SMOKE-METRIC (consume the live signal) Each PRD dogfoods the Assumptions & Evidence gate. backend-quirks.md gains a 'Planned hardening' pointer so the ledger links to the enforcement work. refs #658
…post-mortem) * docs(workflow): backend-quirks ledger + spec-time evidence gate (#655 post-mortem) The iteration loop's verification chain (TDD fakes, design-judge, CI) all validate the implementer's assumptions against each other, never against a real backend. A wrong assumption about Server behavior is reproduced identically in code, fake, and test — all three agree and ship the bug. That is how #655 shipped: pr edit wiped reviewers and pr request-review 400'd, while the full-object-PUT + version rule was already known and correctly applied three functions away in the same file. - docs/backend-quirks.md: new append-only ledger of real Server/Cloud behaviors no linter or hand-written fake can infer (BQ-1..BQ-5), seeded from #655 and the existing httpx.Transport policies, each with a code citation of the correct AND broken pattern. Every future backend bug appends a row in its fix PR. - README §2: write/new-API PRDs now require an Assumptions & Evidence section; every backend-behavior claim is CITED, LEDGER, or a blocking ASSUMED-UNVERIFIED that a reality probe must settle before TDD. These PRDs are reclassified mechanical -> judgment. - README §3: TDD subagent brief must consult the ledger and assert on the captured request, not just stdout. - pre-merge-check §6a: stdout-only write-op test, or a violated quirk, is a design-judge BLOCKER. - quickref + agent-primer: model-tier reclassification, anti-patterns, per-write consult pointer. refs #655 * docs(backlog): queue VERIFICATION-LOOP-HARDENING initiative (#661-#665) Five loop-sized PRD scopes implementing the #658 follow-ups, off-the-shelf tooling (go-vcr, kin-openapi, gh-style acceptance per cli/cli#9745): - #661 VCR-CASSETTES, #662 OPENAPI-VALIDATE (foundations) - #663 ACCEPTANCE-LIVE-WIRE (makes documented Tier 6 real) - #664 E2E-QUEUE-FEEDBACK, #665 SMOKE-METRIC (consume the live signal) Each PRD dogfoods the Assumptions & Evidence gate. backend-quirks.md gains a 'Planned hardening' pointer so the ledger links to the enforcement work. refs #658 * docs(reports): archive stream 178-187 + cycles 158-187 analysis

Description
The goal of this work is to have a set of automated tests that we can use to point
ghat a real GitHub host in order to have some sense of blackbox behavioural validation. This PR introduces the use oftestscriptusing thegh prcommand as a motivating example.It is a non-goal right now to have these running as part of our CI suite.
Reviewer Notes
The README contains most information that I won't repeat in this description.
One big change here is that in order to avoid building
ghand to enable code coverage, I had to move theRealMainthat used to be in thecmd/gh/main.gointo its own package calledghcmd(mainpackages can't be imported), and to export the function. I can't really see much of an issue with this, it's actually pretty idiomatic.