|
| 1 | +--- |
| 2 | +id: quick-start |
| 3 | +title: Quick start |
| 4 | +description: Build and run your first Crawlee crawler in a few minutes, and pick the crawler type that fits your project. |
| 5 | +--- |
| 6 | + |
| 7 | +import ApiLink from '@site/src/components/ApiLink'; |
| 8 | +import Tabs from '@theme/Tabs'; |
| 9 | +import TabItem from '@theme/TabItem'; |
| 10 | +import CodeBlock from '@theme/CodeBlock'; |
| 11 | +import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock'; |
| 12 | + |
| 13 | +import BeautifulsoupCrawlerExample from '!!raw-loader!roa-loader!./code_examples/beautifulsoup_crawler_example.py'; |
| 14 | +import ParselCrawlerExample from '!!raw-loader!roa-loader!./code_examples/parsel_crawler_example.py'; |
| 15 | +import PlaywrightCrawlerExample from '!!raw-loader!roa-loader!./code_examples/playwright_crawler_example.py'; |
| 16 | + |
| 17 | +import PlaywrightCrawlerHeadfulExample from '!!raw-loader!./code_examples/playwright_crawler_headful_example.py'; |
| 18 | + |
| 19 | +This page gets you from installation to a running crawler in a few minutes. To build a complete scraper step by step, follow the [Introduction](./introduction). |
| 20 | + |
| 21 | +## Choose your crawler |
| 22 | + |
| 23 | +Crawlee's main crawler classes are <ApiLink to="class/BeautifulSoupCrawler">`BeautifulSoupCrawler`</ApiLink>, <ApiLink to="class/ParselCrawler">`ParselCrawler`</ApiLink>, and <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink>. All crawlers share the same interface, so you can switch between them without rewriting your handlers. |
| 24 | + |
| 25 | +:::caution Minimum Python version |
| 26 | + |
| 27 | +Crawlee requires Python 3.10 or higher. |
| 28 | + |
| 29 | +::: |
| 30 | + |
| 31 | +### BeautifulSoupCrawler |
| 32 | + |
| 33 | +The <ApiLink to="class/BeautifulSoupCrawler">`BeautifulSoupCrawler`</ApiLink> is a plain HTTP crawler that parses HTML using the well-known [BeautifulSoup](https://pypi.org/project/beautifulsoup4/) library. It fetches pages with an HTTP client that mimics a browser. It's fast and light, but it can't render JavaScript. |
| 34 | + |
| 35 | +### ParselCrawler |
| 36 | + |
| 37 | +The <ApiLink to="class/ParselCrawler">`ParselCrawler`</ApiLink> is similar to the <ApiLink to="class/BeautifulSoupCrawler">`BeautifulSoupCrawler`</ApiLink> but uses the [Parsel](https://pypi.org/project/parsel/) library for HTML parsing. Parsel extracts data from HTML with CSS selectors and XPath, and it's the selector library [Scrapy](https://scrapy.org/) uses. Like the <ApiLink to="class/BeautifulSoupCrawler">`BeautifulSoupCrawler`</ApiLink>, the <ApiLink to="class/ParselCrawler">`ParselCrawler`</ApiLink> can't render JavaScript. |
| 38 | + |
| 39 | +### PlaywrightCrawler |
| 40 | + |
| 41 | +The <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> uses a headless browser controlled by the [Playwright](https://playwright.dev/) library. It can drive Chromium, Firefox, and WebKit. Choose it when the target pages render their content with JavaScript. |
| 42 | + |
| 43 | +## Installation |
| 44 | + |
| 45 | +Crawlee is available as the [`crawlee`](https://pypi.org/project/crawlee/) package on PyPI. The package includes the core functionality, and additional features are available as optional extras to keep dependencies small. |
| 46 | + |
| 47 | +To install Crawlee with all extras using [pip](https://pip.pypa.io/en/stable/), run: |
| 48 | + |
| 49 | +```sh |
| 50 | +python -m pip install 'crawlee[all]' |
| 51 | +``` |
| 52 | + |
| 53 | +Verify that Crawlee is successfully installed: |
| 54 | + |
| 55 | +```sh |
| 56 | +python -c 'import crawlee; print(crawlee.__version__)' |
| 57 | +``` |
| 58 | + |
| 59 | +If you plan to use the <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink>, also install the Playwright browser binaries: |
| 60 | + |
| 61 | +```sh |
| 62 | +playwright install |
| 63 | +``` |
| 64 | + |
| 65 | +For detailed installation instructions, see [Setting up](./introduction/setting-up). |
| 66 | + |
| 67 | +## Crawling |
| 68 | + |
| 69 | +Run the following example to perform a recursive crawl of the Crawlee website using the selected crawler. |
| 70 | + |
| 71 | +<Tabs groupId="quickStart"> |
| 72 | + <TabItem value="BeautifulSoupCrawler" label="BeautifulSoupCrawler" default> |
| 73 | + <RunnableCodeBlock className="language-python" language="python"> |
| 74 | + {BeautifulsoupCrawlerExample} |
| 75 | + </RunnableCodeBlock> |
| 76 | + </TabItem> |
| 77 | + <TabItem value="ParselCrawler" label="ParselCrawler"> |
| 78 | + <RunnableCodeBlock className="language-python" language="python"> |
| 79 | + {ParselCrawlerExample} |
| 80 | + </RunnableCodeBlock> |
| 81 | + </TabItem> |
| 82 | + <TabItem value="PlaywrightCrawler" label="PlaywrightCrawler"> |
| 83 | + <RunnableCodeBlock className="language-python" language="python"> |
| 84 | + {PlaywrightCrawlerExample} |
| 85 | + </RunnableCodeBlock> |
| 86 | + </TabItem> |
| 87 | +</Tabs> |
| 88 | + |
| 89 | +When you run the example, the crawler logs its progress to your terminal. |
| 90 | + |
| 91 | +{/*TODO: improve the logging and add here a sample*/} |
| 92 | + |
| 93 | +## Running headful browser |
| 94 | + |
| 95 | +By default, browsers controlled by Playwright run in headless mode, without a visible window. During development, it helps to run the browser in headful mode and watch what it does. You can also switch from the default Chromium browser to Firefox or WebKit. |
| 96 | + |
| 97 | +<CodeBlock language="python"> |
| 98 | + {PlaywrightCrawlerHeadfulExample} |
| 99 | +</CodeBlock> |
| 100 | + |
| 101 | +When you run the example, a browser window opens and navigates through the Crawlee website. |
| 102 | + |
| 103 | +{/*TODO: add video example*/} |
| 104 | + |
| 105 | +## Results |
| 106 | + |
| 107 | +By default, Crawlee stores data in the `./storage` directory within your current working directory. The crawl results are saved as JSON files under `./storage/datasets/default/`. |
| 108 | + |
| 109 | +To view the results, use the `cat` command: |
| 110 | + |
| 111 | +```sh |
| 112 | +cat ./storage/datasets/default/000000001.json |
| 113 | +``` |
| 114 | + |
| 115 | +The JSON file contains data similar to the following: |
| 116 | + |
| 117 | +```json |
| 118 | +{ |
| 119 | + "url": "https://crawlee.dev/", |
| 120 | + "title": "Crawlee · Build reliable crawlers. Fast. | Crawlee" |
| 121 | +} |
| 122 | +``` |
| 123 | + |
| 124 | +:::tip |
| 125 | + |
| 126 | +To change the storage directory, set the `CRAWLEE_STORAGE_DIR` environment variable to your preferred path. |
| 127 | + |
| 128 | +::: |
| 129 | + |
| 130 | +## Next steps |
| 131 | + |
| 132 | +- To build a complete scraper for a real website step by step, follow the [Introduction](./introduction). |
| 133 | +- To learn how crawlers, requests, storages, and other parts of Crawlee work, read the [Concepts](./concepts). |
| 134 | +- For solutions to specific tasks, see the [Guides](./guides). |
0 commit comments