iframe-proxy

factspark23-hash · 2026-04-23T07:12:26Z

What

Add a self-healing element recovery engine that automatically recovers when element lookups fail due to DOM changes.

Why

One of the most common failure modes in browser automation is elements becoming stale or disappearing after page navigation, AJAX updates, or dynamic content changes. Currently, when an element index lookup fails, the agent gets an error and must manually retry with a refreshed browser state.

This PR adds automatic recovery — the engine fingerprints elements before interaction and attempts multiple recovery strategies when lookups fail.

How

Recovery Strategies (in priority order):

Text content match — Find element with same visible text (highest confidence: 80%)
Accessibility label match — Match by aria-label, placeholder, title, or alt attributes
Role/tag fallback — Find same tag type in similar structural position (lower confidence: 60%)

Flow:

Element found → Fingerprint stored → Click/Input proceeds normally
Element NOT found → Auto-heal attempts recovery → Reports result

New action:

heal_stats — Shows self-healing statistics (attempts, successes, failures, success rate)

Changes

Testing

10/10 tests pass ✅

Tests cover:

Engine initialization
Element fingerprinting
Fingerprint eviction when at capacity
Healing with no fingerprint (graceful failure)
Successful text-based healing
All strategies exhausted (graceful failure)
Stats tracking
Clear functionality
Confidence scoring per strategy

Design Decisions

Fingerprint before, heal after: Elements are fingerprinted BEFORE click/input, so we have a reference if they disappear
Non-blocking: Healing attempts are fast JS evaluations, no network calls
Observable: Full stats tracking for debugging and monitoring
Graceful degradation: If healing fails, original error behavior is preserved

Summary by cubic

Adds a self-healing element recovery engine that auto-recovers missing elements after DOM changes and plugs into click/input flows. Uses a CDP session with Runtime.evaluate and adds a heal_stats action for visibility.

New Features
- Fingerprints elements before interaction; on failure, tries text, accessibility labels, and role/tag with structural hints.
- Integrated into click and input: auto-heals when index lookups fail and returns a retry message after state refresh.
- heal_stats action reports attempts, successes, failures, success rate, and stored fingerprints; 10 tests cover core behavior.
Bug Fixes
- Switched to CDP: use get_or_create_cdp_session() and Runtime.evaluate for reliable healing.
- Use ax_node for element name/role instead of snapshot_node to fix attribute errors and restore CI stability.

^{Written for commit e5a9c16. Summary will update on new commits.}

When element lookups fail (e.g., DOM changed after navigation or dynamic update), the auto-heal engine attempts recovery using multiple strategies: 1. Text content match — find element with same visible text 2. Accessibility label match — aria-label, placeholder, title, alt 3. Role/tag fallback — same tag type in similar structural position How it works: - Elements are fingerprinted BEFORE interaction (click/input) - When lookup fails, recovery strategies are attempted automatically - Healing stats are tracked for observability - New 'heal_stats' action shows recovery statistics Changes: - browser_use/dom/auto_heal.py — Self-healing engine (new module) - browser_use/tools/service.py — Integrated into click and input actions - tests/test_auto_heal.py — 10 tests covering all core functionality Inspired by Agent-OS auto-heal engine, adapted for browser-use's CDP-based architecture.

CLAassistant · 2026-04-23T07:12:36Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

cubic-dev-ai

4 issues found across 3 files

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="browser_use/dom/auto_heal.py">

<violation number="1" location="browser_use/dom/auto_heal.py:151">
P2: A11y recovery selectors interpolate raw attribute values without escaping, so special characters can break `querySelector` and silently disable this healing strategy.</violation>

<violation number="2" location="browser_use/dom/auto_heal.py:240">
P2: Structural fingerprint fields (parent_tag, sibling_index) are never captured, so the role/structure healing strategy cannot perform structural disambiguation and degrades to fallback selection.</violation>

<violation number="3" location="browser_use/dom/auto_heal.py:316">
P1: Healing is gated on `backendNodeId`, but recovery JS reads it from DOM nodes (`el.backendNodeId`) without any demonstrated runtime annotation path, causing likely false healing failures.</violation>

<violation number="4" location="browser_use/dom/auto_heal.py:348">
P1: `page.evaluate` is called with a single list argument while JS expects multiple positional parameters, causing heal strategies to receive wrong argument types and fail.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review, or fix all with cubic.}

cubic-dev-ai · 2026-04-23T07:18:11Z

+		for strategy_name, strategy_fn, args in strategies:
+			try:
+				result = await strategy_fn(*args)
+				if result and result.get('backendNodeId'):


P1: Healing is gated on backendNodeId, but recovery JS reads it from DOM nodes (el.backendNodeId) without any demonstrated runtime annotation path, causing likely false healing failures.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At browser_use/dom/auto_heal.py, line 316: <comment>Healing is gated on `backendNodeId`, but recovery JS reads it from DOM nodes (`el.backendNodeId`) without any demonstrated runtime annotation path, causing likely false healing failures.</comment> <file context> @@ -0,0 +1,388 @@ + for strategy_name, strategy_fn, args in strategies: + try: + result = await strategy_fn(*args) + if result and result.get('backendNodeId'): + fp.healed_count += 1 + fp.last_healed_via = strategy_name </file context>

cubic-dev-ai · 2026-04-23T07:18:11Z

+		classes = []
+		data_attrs = {}
+		tag = ''
+		parent_tag = ''


P2: Structural fingerprint fields (parent_tag, sibling_index) are never captured, so the role/structure healing strategy cannot perform structural disambiguation and degrades to fallback selection.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At browser_use/dom/auto_heal.py, line 240: <comment>Structural fingerprint fields (parent_tag, sibling_index) are never captured, so the role/structure healing strategy cannot perform structural disambiguation and degrades to fallback selection.</comment> <file context> @@ -0,0 +1,388 @@ + classes = [] + data_attrs = {} + tag = '' + parent_tag = '' + sibling_index = 0 + </file context>

cubic-dev-ai · 2026-04-23T07:18:11Z

+	_FIND_BY_A11Y_JS = """
+	(ariaLabel, placeholder, title, alt) => {
+		const selectors = [];
+		if (ariaLabel) selectors.push(`[aria-label="${ariaLabel}"]`);


P2: A11y recovery selectors interpolate raw attribute values without escaping, so special characters can break querySelector and silently disable this healing strategy.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At browser_use/dom/auto_heal.py, line 151: <comment>A11y recovery selectors interpolate raw attribute values without escaping, so special characters can break `querySelector` and silently disable this healing strategy.</comment> <file context> @@ -0,0 +1,388 @@ + _FIND_BY_A11Y_JS = """ + (ariaLabel, placeholder, title, alt) => { + const selectors = []; + if (ariaLabel) selectors.push(`[aria-label="${ariaLabel}"]`); + if (placeholder) selectors.push(`[placeholder="${placeholder}"]`); + if (title) selectors.push(`[title="${title}"]`); </file context>

- Replace browser_session.current_page with get_or_create_cdp_session() - Update auto_heal.py to use CDP Runtime.evaluate instead of page.evaluate() - Update tests to mock CDP sessions instead of Playwright pages - Fixes code-style and type-checker CI failures

…to-heal EnhancedSnapshotNode does not have 'name' or 'role' attributes. These belong to EnhancedAXNode (accessed via node.ax_node). This fixes the 'EnhancedSnapshotNode object has no attribute name' error that caused test_dom_serializer and test_ax_name_matching CI failures.

cubic-dev-ai Bot reviewed Apr 23, 2026

View reviewed changes

factspark23-hash added 2 commits April 23, 2026 16:54

File	Description
`browser_use/dom/auto_heal.py`	Self-healing engine (new module, 388 lines)
`browser_use/tools/service.py`	Integrated into `click` and `input` actions
`tests/test_auto_heal.py`	10 tests covering all core functionality

Sunbelt Computer Software

PL/B Language Development and Support

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: self-healing element recovery engine#4732

feat: self-healing element recovery engine#4732
factspark23-hash wants to merge 3 commits intobrowser-use:mainfrom
factspark23-hash:feat/self-healing-selectors

factspark23-hash commented Apr 23, 2026 •

edited by cubic-dev-ai Bot

Loading

Uh oh!

CLAassistant commented Apr 23, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Uh oh!

cubic-dev-ai Bot Apr 23, 2026 •

edited

Loading

Uh oh!

Uh oh!

cubic-dev-ai Bot Apr 23, 2026 •

edited

Loading

Uh oh!

cubic-dev-ai Bot Apr 23, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Sunbelt Computer Software

PL/B Language Development and Support

Conversation

factspark23-hash commented Apr 23, 2026 • edited by cubic-dev-ai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

How

Recovery Strategies (in priority order):

Flow:

New action:

Changes

Testing

Design Decisions

Summary by cubic

Uh oh!

CLAassistant commented Apr 23, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cubic-dev-ai Bot Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

factspark23-hash commented Apr 23, 2026 •

edited by cubic-dev-ai Bot

Loading

cubic-dev-ai Bot Apr 23, 2026 •

edited

Loading

cubic-dev-ai Bot Apr 23, 2026 •

edited

Loading

cubic-dev-ai Bot Apr 23, 2026 •

edited

Loading