fix(docs): explain off-site iframe limits beside iframe and vision guidance · Issue #1270 · tester-army/e2e · GitHub
Skip to content

fix(docs): explain off-site iframe limits beside iframe and vision guidance #1270

Open

Description

@coygeek

Summary

The iframe and agent authoring guidance does not connect the browser engine's intentional off-site frame boundary to the ways a test can inspect that content. A visible payment form can be absent from the semantic tree used by an agent judgment. The boundary is documented in the security reference and SECURITY.md, but readers of the iframe, Vision, and bundled skill guidance have to discover that connection themselves.

Page

  • docs/browser.mdx, "Reach into an iframe".
  • docs/reference/agent.mdx, "Vision".
  • skills/e2e/references/writing-tests.md, iframe guidance; skills/e2e/references/agent.md, Vision guidance; skills/e2e/references/debugging.md, iframe and inconclusive-judgment troubleshooting.

What is wrong

The browser guide introduces the feature with "browser.frameLocator scopes queries to a frame" and examples for nested frames. It does not explain that these exact queries have different coverage from the semantic observation supplied to agent.act, agent.assert, agent.waitFor, and agent.extract. The Vision reference lists the default evidence as "Semantic tree", but does not identify off-site frame contents as a case where visible UI is missing from that tree. The debugging reference mentions iframes for LOCATOR_NOT_FOUND and pixels for ASSERTION_INCONCLUSIVE, without explaining the shared authoring scenario.

Steps to reproduce

  1. Inspect the listed pages in either source baseline below. Compare them with the security reference's "Site check" section, which states that observations omit child frames from other sites and data: frames.
  2. For a model-free demonstration, install the example repository's locked dependencies with npm ci and start its app as documented in its README. Its default host is http://localhost:3000; its card and nested field frames use http://127.0.0.1:3000.
  3. Configure an e2e web target with app.url set to the host URL. Run e2e mcp, open a session, and call its observe tool. No model is required.
  4. Close the session. Configure app.url as the frame hostname instead, open another session, and observe again.
  5. Independently check exact access from the original off-site host with the following test. Import test from @e2e-dev/web and expect from e2e:
test("reads the off-site pay button", async ({ app, browser }) => {
  await app.open("/");
  await expect(
    browser.frameLocator('[data-testid="checkout-iframe"]')
      .getByRole("button", { name: "Pay $165.00" })
  ).toBeVisible();
});

The executed probes used an unused port instead of the default, with both public origin variables updated as the example README instructs. They used a config with no model. The example's model-backed suite was not rerun.

Expected behavior

The iframe authoring path should explain the existing distinction between exact frame queries and the agent's semantic tree, and direct readers to the applicable judgment evidence mode. The Vision and bundled guidance should make the same limitation discoverable. This asks for documentation of the existing behavior, without changing the observation boundary.

Actual behavior

The off-site observation contained three nodes:

#root document "e2e iframe checkout"
 #n2 main
  #n3 iframe "Card payment" testid="checkout-iframe"

There were no child contents or truncation warning. With the host opened on the frame hostname, the observation contained 23 nodes, including the card document, nested field documents, textboxes, and a Pay button. The exact off-site frameLocator assertion passed, 1/1 test with retries disabled.

The current main authoring pages still omit the boundary/remedy connection. This is a documentation omission; the observed filtering agrees with the security contract.

Runtime or environment

  • Published packages in the isolated install: e2e 0.21.2 and @e2e-dev/web 0.15.2, both npm latest at verification.
  • Documentation inspected at release tag e2e@0.21.2, commit 393005aa6952eaef62886cd03c8d8060f8f05a4f, and default-branch main commit ee901b333ed12352418dfb07bda5eafe32e71c8d.
  • Example: tester-army/e2e-iframes, commit 0e2cbe8b6ca32ba7205216955edfd463b6ec99ce.
  • Node 26.12.0, macOS arm64, Darwin 27.0.0; default headless web engine.
  • Runtime observations concern the published packages; main was inspected as source, not built or run.

Evidence

  • Expected source: CONTRIBUTING.md, "The contract" says code is authoritative when documentation and code differ. The security reference's "Site check" establishes the actual frame boundary.
  • Failure source: The inspected iframe guide, Vision reference, and bundled paths listed above omit that connection; the model-free comparison demonstrates why it affects authoring.
  • Evidence provenance: observed
  • Local verification: reproduced
  • Reproduction completeness: complete

Documentation inspection, both semantic observations, and the exact frame assertion were performed for this report.

The implementation in packages/web/src/observation.ts enters a child frame only when isOnSiteFrame(frame.url(), deps.site) holds, and explicitly says omitted off-site frames are boundary nodes rather than truncation. The site's comparison is an approximation of the registrable domain, not a blanket same-origin rule; localhost and an IP literal count as separate sites.

Restoration check

From "Reach into an iframe", a reader should be able to determine why an off-site frame's visible contents are missing from an agent's tree, choose exact frameLocator queries or an appropriate Vision judgment, and find the same explanation in the bundled agent skill and debugging guidance. Keep the runtime control intact: off-site content stays outside the tree while exact frame access still works.

Suggested wording

"Agent semantic observations omit child-frame contents from other sites, leaving an iframe boundary node. This intentional omission is not truncation. Exact browser.frameLocator(...) queries can inspect the frame directly. For a judgment about visible frame content, use vision: true when screenshots are available, or the documented visual-only mode when appropriate. Screenshots cover the current viewport and remain subject to the Vision withholding rules."

A short explanation or cross-reference beside the existing iframe and Vision sections, mirrored into the bundled guidance, would make the existing behavior discoverable without adding a separate guide.

Triage assessment

  • Impact level: P2
  • Assessment status: supported
  • Impact basis: Test authors can ask a tree-only judgment about visible embedded content that the engine deliberately omits; the authoring and troubleshooting pages do not explain this bounded evidence gap.
  • Workaround status: available
  • Workaround basis: The exact frame assertion passed against off-site content. The existing Vision reference documents screenshot evidence modes and their availability limits; no Vision model run is claimed here.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions