iframe-proxy

riandradiva18 · 2026-10-21T09:52:31Z

Summary

Adds a MrScraper block with three operations:

Scrape Page: fetch a URL as HTML, Markdown, and/or a screenshot. Supports browser rendering (auto/on/off), real-device routing, proxy country, wait-for-selector, resource blocking, and retry/token-cap controls.
Google Search: search by query text or a pasted Google search URL (q, gl, hl, start are picked up). Returns organic results, search metadata, and pagination, or the raw results page HTML.
Get Account: subscription status, token usage/limit/remaining, and rate limit.

Also includes the icon, docs page, registry entries, and regenerated tool metadata, integration catalog, deployment config, and block names.

Some MrScraper API behavior that shaped the scrape tool:

HTML-only requests go to the fetch endpoint as a GET with the target URL in the query. That endpoint replays the incoming method against the target site, so a POST fails on sites that reject it (books.toscrape.com answers 405). Markdown requests go to the browser scraper as a POST with a JSON body.
The fetch endpoint returns raw text/html and reports token_usage in a response header; the browser scraper returns JSON. The transform handles both and reads metrics from headers as a fallback.
Screenshots come back as PNG from the fetch endpoint and JPEG from the browser scraper, so the file type comes from the base64 signature.
maxRetries and tokenCap only apply to the browser scraper, so those fields only show when Markdown is on.
The API key is sent in headers only, never in the query string.
The SERP endpoint reports current_page: 1 for every page, so the tool reports the page that was requested instead.

Type of Change

Testing

Ran every operation against the live MrScraper API with a real key. Each call went through the tool's own request builder (via prepareToolRequest) and transformResponse, with block defaults mapped through the block's tools.config.params. All 11 live calls passed, and calls with an invalid key fail with a 401 as expected. The full results and sample responses are in the comment below.

Locally: bun run lint, bun run type-check, bun run check:audits, and check-block-registry.ts against staging.

I didn't add unit tests. Following the repo's testing guidance, I tested against the real API instead.

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

- send HTML-only and screenshot-only scrapes as GET with the target URL in the query; the fetch endpoint replays the request method, so a POST fails on sites that reject it - read raw text/html responses and take token usage from response headers - send screenshot=full and detect PNG vs JPEG from the returned image - keep the API key out of the query string - report the requested search page; the SERP API returns current_page 1 for every page - show retry and token-cap fields only for Markdown requests - use defaultValue for switch defaults - regenerate the integration catalog, tool metadata, block names, and docs manifest

riandradiva18 · 2026-10-21T09:52:38Z

@waleedlatif1 this is ready for a look when you have time.

I tested every operation against the live MrScraper API with a real key. Each call went through the tool's own request builder (prepareToolRequest) and transformResponse, with block defaults mapped through tools.config.params.

Case	Request	HTTP	Result
Get Account	`GET /api/v1/subscription-accounts`	200	pass
Search, query + region/language	`POST /api/google/serp/v2/sync`	200	pass, 9 organic results
Search, pasted Google URL (`gl`, `hl`, `start=10`)	`POST`	200	pass, page 2
Search, `page: 2`	`POST`	200	pass, `currentPage: 2`, `nextPage: 3`
Search, HTML format	`POST`	200	pass, raw results HTML
Scrape, block defaults (HTML only), books.toscrape.com	`GET` fetch endpoint	200 `text/html`	pass, 51 KB HTML, 1 token
Scrape, HTML only, kempinski.com hotel page	`GET`	200 `text/html`	pass, 208 KB HTML
Scrape, Markdown only	`POST` browser scraper	200 JSON	pass
Scrape, HTML + Markdown, quotes.toscrape.com	`POST`	200 JSON	pass, target status 200
Scrape, screenshot	`GET`	200 JSON	pass, PNG stored as a file
Scrape, Markdown + screenshot	`POST`	200 JSON	pass, JPEG stored as a file
Scrape, invalid key	`GET`	401	fails with `invalid API token`, as expected
Get Account, invalid key	`GET`	401	fails with `Unauthorized access`, as expected

The API key never appeared in a request URL in any of these calls.

Sample outputs (long strings trimmed)

Get Account (name and email removed)

{
  "subscriptionStatus": "active",
  "enterprise": false,
  "tokenUsage": 5242300,
  "tokenLimit": 10000000,
  "tokenRemaining": 4757700,
  "usagePercent": 52.42,
  "rateLimit": 0,
  "rateTtl": 0,
  "autoRenew": false,
  "endsAt": "2027-10-08T17:00:00.000Z",
  "user": { "name": "…", "email": "…", "verified": true }
}

Google Search, query: "best coffee jakarta", region: "id", language: "en", page: 2

{
  "searchResults": [
    {
      "title": "15 Cafe di Jakarta Barat yang Wajib Masuk Wishlist Kamu!",
      "link": "https://blog.cove.id/cafe-di-jakarta-barat/",
      "description": "28 Feb 2024 — 2. Satu Meja · 3. Upstairs Coffee Signature · …",
      "source": "Cove Blog",
      "displayLink": "https://blog.cove.id › cafe-di-jakart...",
      "rank": 1,
      "globalRank": 1
    }
  ],
  "html": null,
  "general": {
    "query": "best coffee jakarta",
    "resultsCount": 10,
    "searchTime": 10.7,
    "language": "en",
    "location": "id",
    "pageTitle": "best coffee jakarta - Google Search",
    "timestamp": "2026-10-21T09:33:19.013Z"
  },
  "pagination": {
    "currentPage": 2,
    "nextPage": 3,
    "nextPageStart": 20,
    "nextPageLink": "https://www.google.com/search?q=best+coffee+jakarta&…",
    "pages": [{ "page": 1, "start": 0, "link": "…" }, { "page": 3, "start": 20, "link": "…" }]
  },
  "tokenUsage": 1
}

Scrape Page, block defaults (HTML only)

{
  "html": "<!DOCTYPE html>\n<!--[if lt IE 7]>      <html lang=\"en-us\" class=\"no-js lt-ie9 …",
  "markdown": null,
  "tokenUsage": 1,
  "runtime": null,
  "retryCount": null,
  "targetStatusCode": null,
  "error": null
}

Scrape Page, HTML + Markdown

{
  "html": "<!DOCTYPE html><html lang=\"en\"><head>\n\t<meta charset=\"UTF-8\">\n\t<title>Quotes to Scrape</title> …",
  "markdown": "#  [Quotes to Scrape](/)\n\n[Login](/login)\n\n“The world as we have created it is a process of our thinking. …",
  "tokenUsage": 3,
  "runtime": 5.95,
  "retryCount": 0,
  "targetStatusCode": 200,
  "error": null
}

Scrape Page, Markdown + screenshot

{
  "html": null,
  "markdown": "[Books to Scrape](index.html) We love being scraped! …",
  "screenshot": { "name": "screenshot.jpg", "mimeType": "image/jpeg", "size": 272776 },
  "tokenUsage": 6,
  "runtime": 11.33,
  "retryCount": 0,
  "targetStatusCode": 200,
  "error": null
}

Locally: bun run lint, bun run type-check, bun run check:audits, check-block-registry.ts against staging, plus blocks.test.ts, password-masking.test.ts, registry-lookup.test.ts, blocks/utils.test.ts, and request-transport.test.ts (70 tests passing).

vercel · 2026-10-21T09:52:42Z

@riandradiva18 is attempting to deploy a commit to the Sim Team on Vercel.

A member of the Team first needs to authorize it.

greptile-apps · 2026-10-21T09:58:04Z

Confidence Score: 4/5

[Medium impact] The PR adds a new, bounded web-scraping and search workflow integration.

The PR appears safe to merge, with a non-blocking fix needed for searches containing ordinary URLs.

Findings

Literal URL searches fail ▶

Summary

Adds MrScraper tools for page retrieval, Google search, and account usage.

MrScraper retrieves web pages as HTML, Markdown, or screenshots.
MrScraper turns Google queries into results or page HTML.
MrScraper reports account usage and subscription details.
Workflow builders can set up MrScraper from a registered block.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[MrScraper block or agent] --> B{Operation}
  B -->|Scrape| C{Markdown requested?}
  C -->|No| D[GET fetch request]
  C -->|Yes| E[POST browser request]
  D --> F[Transform page response]
  E --> F
  F --> G[Store screenshot when present]
  B -->|Search| H[POST Google search]
  H --> I[Map results and pagination]
  B -->|Get Account| J[GET account summary]

_{Reviews (1) · Last reviewed commit: "fix(mrscraper): route html-only scrapes ..." · Reviewed by Greptile}

greptile-apps · 2026-10-21T09:58:07Z

+  if (/^https?:\/\//i.test(value)) {
+    const url = new URL(value)
+    query = url.searchParams.get('q')?.trim() || ''
+    if (!query) throw new Error('A Google search URL must contain a q parameter.')


Literal URL searches fail

This branch treats every HTTP URL as a Google search URL. Searching for https://example.com throws because it has no q parameter. A non-Google URL containing q instead searches for that parameter and drops the supplied URL. Only extract parameters from Google search URLs; pass other URLs through as query text.

riandradiva18 added 2 commits October 21, 2026 16:28

feat(integrations): add MrScraper tools and block

44b277c

greptile-apps Bot reviewed Oct 21, 2026

View reviewed changes

Sunbelt Computer Software

PL/B Language Development and Support

Navigation Menu

Repository navigation

feat(mrscraper): add MrScraper integration for page scraping, Google search, and account usage - #9277

feat(mrscraper): add MrScraper integration for page scraping, Google search, and account usage#9277
riandradiva18 wants to merge 2 commits into
simstudioai:stagingfrom
mrscraper-com:feat/mrscraper-integration

riandradiva18 commented Oct 21, 2026

Uh oh!

riandradiva18 commented Oct 21, 2026

Uh oh!

vercel Bot commented Oct 21, 2026

Uh oh!

greptile-apps Bot commented Oct 21, 2026 •

edited

Loading

Diagram

Uh oh!

greptile-apps Bot Oct 21, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Sunbelt Computer Software

PL/B Language Development and Support

Conversation

riandradiva18 commented Oct 21, 2026

Summary

Type of Change

Testing

Checklist

Uh oh!

riandradiva18 commented Oct 21, 2026

Uh oh!

vercel Bot commented Oct 21, 2026

Uh oh!

greptile-apps Bot commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 4/5

Findings

Summary

Diagram

Uh oh!

greptile-apps Bot Oct 21, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

greptile-apps Bot commented Oct 21, 2026 •

edited

Loading