fix(filesystem): return image payload for internal images and guard replace on binary files by codeXsidd · Pull Request #6116 · browser-use/browser-use · GitHub
Skip to content

fix(filesystem): return image payload for internal images and guard replace on binary files - #6116

Open
codeXsidd wants to merge 3 commits into
browser-use:mainfrom
codeXsidd:fix/issue-6115-internal-binary-files
Open

codeXsidd wants to merge 3 commits into
browser-use:mainfrom
codeXsidd:fix/issue-6115-internal-binary-files

Conversation

@codeXsidd

@codeXsidd codeXsidd commented Oct 17, 2026 •

Copy link
Copy Markdown

Summary

Enables structured image reading for internal Base64BinaryFile files in FileSystem.read_file_structured() so vision models receive image payloads when reading agent-created images, adds a clear error guard in FileSystem.replace_file_str() to prevent invalid text replacements on binary files, and extends MIME detection in read_state_images prompt generation for GIF and WebP image files.

Problem

  1. When binary/image files (PngFile, GifFile, JpgFile, JpegFile, WebpFile) are written to the agent's file system, calling read_file_structured() treated them as text. It extracted file_obj.read() (the summary string [binary <ext> file, <N> bytes]) and returned images=None. Consequently, invoking the read_file action on internal images gave the vision model no image data in ActionResult.images.
  2. Calling replace_file_str() on binary files attempted string replacement on the stub description and passed it back to file_obj.write(), resulting in a base64 decode validation failure instead of a clear error.
  3. In AgentMessagePrompt, read_state_images only explicitly identified .png and defaulted all other image files to image/jpeg, mislabeling .gif and .webp payloads.

Solution

  1. In FileSystem.read_file_structured(): check isinstance(file_obj, Base64BinaryFile). If true, populate result['images'] = [{'name': file_obj.full_name, 'data': file_obj.content}] and set result['message'] = f'{sanitize_note}Read image file {resolved}.', matching external image read behavior.
  2. In FileSystem.replace_file_str(): guard against isinstance(file_obj, Base64BinaryFile) and return Error: Cannot replace text in binary file '{full_filename}'. Overwrite it instead., matching append_file_content.
  3. In browser_use/agent/prompts.py: extend MIME detection in AgentMessagePrompt for .gif (image/gif) and .webp (image/webp) so the model receives the correct media type.

Testing

  • Added test_read_internal_binary_image_returns_images_payload verifying read_file_structured and read_file return the image data payload for internal binary files.
  • Added test_replace_file_str_on_binary_file_returns_clear_error verifying replace_file_str returns a descriptive error and does not corrupt existing binary files.
  • Added tests in test_agent_message_prompt_images_included verifying GIF and WebP MIME types and data URLs in read_state_images.
  • Ran pytest on all related test suites (tests/ci/test_file_system_images.py, tests/ci/test_file_system_llm_integration.py, and tests/ci/infrastructure/test_filesystem.py): all 124 tests passed.
  • Ran ruff check, codespell, and pyright: 0 errors.

Related Issue

Closes #6115

Copilot AI balanced review requested due to automatic review settings October 17, 2026 05:40

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 17, 2026 •

Copy link
Copy Markdown

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 2 files

Reply to a comment to ask cubic a question or push back. It learns from your replies.
Click Fix with cubic on a comment, or reply @cubic-dev-ai fix this, and cubic writes the fix.

View guided diff | Re-trigger cubic

Comment thread browser_use/filesystem/file_system.py
Comment thread browser_use/filesystem/file_system.py
@codeXsidd

Copy link
Copy Markdown
Author

Addressed in commit 0d2c4e3: extended MIME detection in AgentMessagePrompt for .gif (image/gif) and .webp (image/webp) with unit tests.

@codeXsidd
codeXsidd requested a balanced review from Copilot October 17, 2026 06:07

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@mintlify

mintlify Bot commented Oct 22, 2026

Copy link
Copy Markdown

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: FileSystem.read_file_structured returns empty images for internal image files and replace_file_str fails on binary files

3 participants