feat(playwright): initial Playwright package with a remote Docker browser by FRSgit · Pull Request #424 · pixsame/pixsame · GitHub
Skip to content

feat(playwright): initial Playwright package with a remote Docker browser - #424

Open
FRSgit wants to merge 9 commits into
feat/manifest-emissionfrom
feat/playwright
Open

FRSgit wants to merge 9 commits into
feat/manifest-emissionfrom
feat/playwright

Conversation

@FRSgit

@FRSgit FRSgit commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

This is the Playwright adapter from RENDERER-DESIGN.md (section 4 and phase 3 of section 16), as a first cut. It lives in packages/playwright-visual-regression-diff and is meant to be published as @frsource/playwright-visual-regression-diff.

What's in it:

  • A matchImage fixture (import { test, expect } from '@frsource/playwright-visual-regression-diff', or base.extend(visualRegressionFixtures) if you extend test yourself). Same options as the Cypress command where they make sense: imagesPath with {spec_path} / {platform} / {os} / {browser}, maxDiffThreshold, diffConfig, createMissingImages, updateImages incl. 'failures-only', title, matchAgainstPath, screenshotConfig, deterministicRendering (animations off, caret hidden). Globals go through use: { visualRegression: {...} }.
  • The same comparison as the plugin: sharp decode/pad, pixelmatch with includeAA: false, same _#n.png / .actual.png / .diff.png layout, same FRSOURCE_CPVRD_V marker, so baselines are interchangeable between the two tools. On failure the three images are attached to the Playwright report.
  • The run manifest, one file per worker in Playwright's outputDir (cp-visual-regression-diff-manifest.playwright.w<n>.json), so the GitHub App's default glob already picks them up and merges them like manifests from several machines. Same shape as feat: run manifest for CI consumers, shipped as the @frsource/visual-regression-manifest standard #414 incl. renderer and hashes; only runner is Playwright-specific.
  • remoteBrowser() (@frsource/playwright-visual-regression-diff/remote): starts or reuses a container from mcr.microsoft.com/playwright:v<your version>-noble running playwright run-server, and hands you connectOptions plus globalSetup / globalTeardown for the config. The runner, the app and the test code stay on the host, only the browser is in Docker, which is why this came before DOM snapshots for Playwright. exposeNetwork: '<loopback>' is on by default, so localhost dev servers just work without host.docker.internal. Containers are labelled (frsource.visual-renderer=1, version, port) and reused; Docker is detected but never launched; every failure has a code (DAEMON_UNREACHABLE, CONTAINER_START_FAILED, CONTAINER_UNHEALTHY, PLAYWRIGHT_VERSION_UNKNOWN) with a one-line fix. When the remote browser is in use the manifest says renderer: { backend: 'docker', rendererVersion: <playwright version>, imageDigest: <sha256 of the image> }.

Things I checked:

  • Unit tests (vitest, 35 tests, ~96% lines): comparison statuses and file effects, path tokens, manifest writer incl. retry purge and CI detection, remote lifecycle against a scripted fake docker (start, reuse, unreachable daemon, failed start, exited container, timeout).
  • pnpm test:smoke (real Chromium, page.setContent): first run creates baselines, second passes, a colour change fails with .actual.png + .diff.png kept and attached, manifest written. Ran on this Mac.
  • pnpm test:smoke:remote against Docker Desktop: fresh start pulls nothing (image pre-pulled), container comes up in ~3 s, all 4 pass; the second run reuses the container. Same page rendered by the same Chromium 153 build differs by 0.7–1.0% of pixels between macOS and the Linux container (fonts), and by 0% between two container runs, which is the whole point.
  • One gotcha worth knowing: Docker's port proxy accepts TCP before run-server listens, so readiness is an HTTP probe (any response = up), not a TCP connect.

Deliberately not in this PR:

  • No shared core package yet. compare, the path tokens, the manifest types and ci.utils are mirrored from the plugin (marked as such in the files). Extracting @frsource/visual-regression-core is the obvious next step once the API settles; I didn't want to refactor the Cypress plugin in the same PR.
  • No toMatchImage() matcher, no DOM-snapshot capture, no CLI (doctor / pull). The renderer client package from section 11 of the design is a separate piece of work; remoteBrowser() is intentionally a thin docker CLI wrapper that fits in one file.
  • No CI job: the smoke tests need a browser or Docker, and I'd rather keep the billed minutes down until the package earns it. The unit tests run in the existing test-integration-coverage job via pnpm -r.
  • version is 0.0.0 and it's registered in release-please, so the first feat on main cuts 0.1.0.

Retargeted onto the 4.x stack

The branch was rebased from feat/manifest (the v5 line) onto feat/github-app (#423), the top of the 4.x stack (#425 → #426 → #427 → #428 → #429 → #430 → #423), so the package ships independently of the plugin major. Two follow-up commits update what the description above says about the manifest and the release:

  • Manifest through @frsource/visual-regression-manifest (feat(manifest): publish @pixsame/manifest 1.0.0-beta.1 on beta #427). The mirrored manifest.ts / manifest.types.ts / ci.utils.ts are gone; src/manifest.utils.ts maps a worker's config onto the package's ManifestWriter (runner: { name: 'playwright', ... }, host platform, ci via detectCi, per-entry renderer from remoteBrowser() as before, entries with the same statuses and fields the 4.3 plugin records). The file is now named with getManifestFileName('playwright.w<n>'), i.e. <outputDir>/visual-regression-manifest.playwright.w<n>.json, which the GitHub App's default MANIFEST_FILE_GLOB matches and parseManifestJson accepts unchanged. The unit tests and the smoke spec validate the written file against the package's JSON Schema (validateManifest returns no issues on a real Chromium run). The sharp comparison and the path tokens stay in this package; a shared core is still a later step.
  • Release: 0.1.0-beta.1 on the next dist-tag through release-please's prerelease mode, registered exactly like the manifest package (versioning: prerelease, prerelease: true, release-as: 0.1.0-beta.1, 0.0.0 placeholder in .release-please-manifest.json, which scripts/publish.mjs skips). Release order per CONTRIBUTING: the manifest package's release PR merges and publishes first, this package's after it, since workspace:^ is rewritten to the manifest version in the repo at publish time. README says the package is 0.x and options may still change.

Verified on the retargeted branch: Playwright package build + 31 unit tests, smoke spec 4/4 twice (create, then compare), plugin 176 tests, GitHub App 86 tests, manifest package 60 tests, root eslint --max-warnings 0, prettier.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Fa3bWmHWmzRkXcueAfwotL

Review follow-ups (2026-09-26)

  • Retargeted onto feat/manifest-emission (feat: write a run manifest for CI consumers (4.x) #429) instead of the App PR, so the two merge independently.
  • Renamed to @pixsame/playwright before the first publish; depends on @pixsame/manifest; the remote-browser env vars are PIXSAME_PLAYWRIGHT_REMOTE*. The README now says that the 4.x Cypress plugin compares with includeAA: true while this package uses false, until 5.0 aligns them.
  • The CONTRIBUTING commit from this branch was folded into feat: write a run manifest for CI consumers (4.x) #429's wording during the rebase (it said the same thing).

@FRSgit
FRSgit changed the base branch from feat/manifest to feat/github-app September 26, 2026 14:40
@FRSgit
FRSgit changed the base branch from feat/github-app to feat/manifest-emission September 26, 2026 18:27
@FRSgit
FRSgit force-pushed the feat/manifest-emission branch from b2be329 to b49f3fa Compare September 30, 2026 14:09
FRSgit and others added 8 commits September 30, 2026 16:41
…wser

@frsource/playwright-visual-regression-diff brings the matchImage flow to
Playwright as a fixture: same pixelmatch comparison, padding and PNG marker
as the Cypress plugin, same baseline file layout and path tokens, and the
same run manifest (one file per worker, discoverable with the existing
*cp-visual-regression-diff-manifest*.json glob). remoteBrowser() runs the
browser inside the official Playwright image while the runner, app and
test code stay on the host, which is the Playwright adapter's first render
mode in RENDERER-DESIGN.md; the manifest's renderer block then reports
backend docker with the image digest.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ression-manifest

The package mirrored the plugin's manifest writer, types and CI detection
from before the manifest package existed. Now that the standard lives in
@frsource/visual-regression-manifest, the fixture maps a worker's config
onto its ManifestWriter (src/manifest.utils.ts) and drops manifest.ts,
manifest.types.ts and ci.utils.ts together with their tests.

The file is named with getManifestFileName('playwright.w<n>'), i.e.
<outputDir>/visual-regression-manifest.playwright.w<n>.json, so the
GitHub App's default MANIFEST_FILE_GLOB picks it up unchanged. The
renderer block keeps the docker/native identity from remoteBrowser().
The unit tests and the smoke spec validate the written file against the
package's schema.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fa3bWmHWmzRkXcueAfwotL
Register the package in release-please's prerelease mode the way the
manifest package is: the first release PR cuts 0.1.0-beta.1, which
scripts/publish.mjs sends to the `next` dist-tag. Ship the LICENSE file
and note in CONTRIBUTING that this package, like the plugin, has to be
released after @frsource/visual-regression-manifest.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fa3bWmHWmzRkXcueAfwotL
…orker keeps it

Playwright restarts a worker process after a test failure. The new process
keeps the slot's `parallelIndex` but gets a fresh `workerIndex`, and the
manifest file was named after `parallelIndex`: `ManifestWriter` rewrites the
whole file from the process's own state, so the first `matchImage` in the
restarted worker replaced `w<parallelIndex>.json` and every entry the earlier
process had recorded was gone, the failed one included. Reproduced with three
tests in one worker: after the second failed, the file held only the third.

The file is `visual-regression-manifest.playwright.w<workerIndex>.json` now,
unique per process for the whole run; consumers merge the files like manifests
of several machines, which the new unit test does through `mergeManifests`
(no warnings, the retry's entry wins). A configured
`visualRegressionManifestPath` gets the same `.w<workerIndex>` label before
its extension, since one file for all workers had the same problem. The
`runner` block records both `parallelIndex` and `workerIndex`.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fa3bWmHWmzRkXcueAfwotL
The Playwright package takes its Tozsame name before its first publish
(0.1.0-beta.1 on next): package name, release-please component, README,
the internal remote-browser environment variables (TOZSAME_PLAYWRIGHT_*)
and the manifest dependency, now @tozsame/manifest. The README also says
which comparison default differs from the 4.x Cypress plugin (includeAA)
until 5.0 aligns them.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fa3bWmHWmzRkXcueAfwotL
The package publishes as @pixsame/playwright, depends on @pixsame/manifest,
reads PIXSAME_PLAYWRIGHT_REMOTE* and points its homepage at https://pixsame.com.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0151fsftRgaK9YzhSySZjakh
…t/playwright

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant