{{ message }}
Repository navigation
feat(batch-scrape): add a batch-scrape command and JSON extraction guidance - #324
Merged
Merged
Conversation
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 9 files
Shadow auto-approve: would not auto-approve because issues were found.
View guided diff | Turn on auto-fix | Re-trigger cubic
Contributor
There was a problem hiding this comment.
0 issues found across 4 files (changes from recent commits).
Confidence score: 5/5
- Automated review surfaced no issues in the provided summaries.
- No files require special attention.
Shadow auto-approve: would auto-approve. Adds a new batch-scrape command (option merging, job status/cancel, docs, skills) with unit tests; no existing command behavior changes, and the dismissed option-merging findings are covered by the current validation and documented explicit-flag precedence.
View guided diff | Turn on auto-fix | Re-trigger cubic
erikengervall
force-pushed
the
capy/add-a-batch-scrape
branch
5 times, most recently
from
October 20, 2026 19:36
45ff959 to
fdb9592
Compare
erikengervall
force-pushed
the
capy/add-a-batch-scrape
branch
from
October 20, 2026 19:40
fdb9592 to
9806545
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
firecrawl batch-scrape, a command for the/v2/batch/scrapeAPI, and teaches the bundled skills when to use it and how to extract JSON fields from a page.Today
firecrawl scrape url1 url2 ...runs concurrent single scrapes that keep markdown only.batch-scraperuns one async job that keeps every requested format, supports JSON extraction, webhooks and cancellation, and can be checked later by job ID.Command
--urls-file(one per line; blank lines and#comments ignored). URLs are normalised and de-duplicated.--statusand--cancelwork likecrawl.--format,--only-main-content,--wait-for,--max-age, and--scrape-options/--scrape-options-fileas the base that explicit flags override.--schema,--schema-fileand--promptadd thejsonformat;--format jsonwithout either fails locally.--wait,--progress,--poll-interval,--timeout,--webhook,--max-concurrency,--ignore-invalid-urls,-o,--pretty. Without--waitit prints the job ID and a status hint, matchingcrawl.startBatchScrape,batchScrape,getBatchScrapeStatusandcancelBatchScrapewithintegration: 'cli'.Skills and docs
firecrawl-scrape: new "Structured JSON from a page" (scrape --format json --schema) and "Many known URLs" (batch-scrape) sections.firecrawl: a command-table row and a pointer for batch scraping.batch-scrapesection aftercrawl.Tests
cli-aliasescovers the job-status case against the built CLI.pnpm run type-check,pnpm run format:checkandpnpm testpass.markdown,links, a no-wait start then status by job ID,--schemaand--promptextraction,--urls-filethen--cancel, and the two local error paths. Webhook delivery and a live--timeoutexpiry were not exercised.Summary by cubic
Adds
firecrawl batch-scrape, a command for the/v2/batch/scrapeAPI, and teaches the bundled skills when to use it and how to extract JSON fields from a page. Unlike multi-URLscrape, which keeps markdown only,batch-scraperuns one async job that keeps every requested format, supports JSON extraction, webhooks, and cancellation, and can be checked later by job ID.--urls-file(one per line, blank lines and#comments ignored), normalised and de-duplicated.--statusand--cancelwork likecrawl.--schema,--schema-file, or--promptadd thejsonformat and keep existing--scrape-optionsformats; ajsonformat without any of them fails locally.startBatchScrape,batchScrape,getBatchScrapeStatus, andcancelBatchScrapewithintegration: 'cli'; version bumped to 1.26.18.Written for commit 9806545. Summary will update on new commits.