docs: restructure documentation to align with SDK and Client docs (#2… · apify/crawlee-python@1d40908 · GitHub
Skip to content

Commit 1d40908

Browse files
authored
docs: restructure documentation to align with SDK and Client docs (#2197)
Aligns the docs structure with apify-sdk-python and apify-client-python. - New Concepts section (26 pages in 7 groups). Component pages move here from Guides, and new pages cover requests, crawling context, events, statistics, configuration, and logging. - Guides keep only task-oriented how-tos. The Examples section is dissolved into Concepts and Guides. - Other sidebar categories stay the same. - Every moved or removed URL gets a redirect. The frozen 1.11 and 0.6 versions are left untouched, so the new structure is live at `/docs/next` until the next snapshot. *✍️ Drafted by Claude Code*
1 parent 508f544 commit 1d40908

321 files changed

Lines changed: 1669 additions & 1273 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎README.md‎

Lines changed: 1 addition & 1 deletion

docs/quick-start/code_examples/beautifulsoup_crawler_example.py renamed to docs/01_quick-start/code_examples/beautifulsoup_crawler_example.py

File renamed without changes.

docs/quick-start/code_examples/parsel_crawler_example.py renamed to docs/01_quick-start/code_examples/parsel_crawler_example.py

File renamed without changes.

docs/quick-start/code_examples/playwright_crawler_example.py renamed to docs/01_quick-start/code_examples/playwright_crawler_example.py

File renamed without changes.

docs/quick-start/code_examples/playwright_crawler_headful_example.py renamed to docs/01_quick-start/code_examples/playwright_crawler_headful_example.py

File renamed without changes.

‎docs/01_quick-start/index.mdx‎

Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
---
2+
id: quick-start
3+
title: Quick start
4+
description: Build and run your first Crawlee crawler in a few minutes, and pick the crawler type that fits your project.
5+
---
6+
7+
import ApiLink from '@site/src/components/ApiLink';
8+
import Tabs from '@theme/Tabs';
9+
import TabItem from '@theme/TabItem';
10+
import CodeBlock from '@theme/CodeBlock';
11+
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
12+
13+
import BeautifulsoupCrawlerExample from '!!raw-loader!roa-loader!./code_examples/beautifulsoup_crawler_example.py';
14+
import ParselCrawlerExample from '!!raw-loader!roa-loader!./code_examples/parsel_crawler_example.py';
15+
import PlaywrightCrawlerExample from '!!raw-loader!roa-loader!./code_examples/playwright_crawler_example.py';
16+
17+
import PlaywrightCrawlerHeadfulExample from '!!raw-loader!./code_examples/playwright_crawler_headful_example.py';
18+
19+
This page gets you from installation to a running crawler in a few minutes. To build a complete scraper step by step, follow the [Introduction](./introduction).
20+
21+
## Choose your crawler
22+
23+
Crawlee's main crawler classes are <ApiLink to="class/BeautifulSoupCrawler">`BeautifulSoupCrawler`</ApiLink>, <ApiLink to="class/ParselCrawler">`ParselCrawler`</ApiLink>, and <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink>. All crawlers share the same interface, so you can switch between them without rewriting your handlers.
24+
25+
:::caution Minimum Python version
26+
27+
Crawlee requires Python 3.10 or higher.
28+
29+
:::
30+
31+
### BeautifulSoupCrawler
32+
33+
The <ApiLink to="class/BeautifulSoupCrawler">`BeautifulSoupCrawler`</ApiLink> is a plain HTTP crawler that parses HTML using the well-known [BeautifulSoup](https://pypi.org/project/beautifulsoup4/) library. It fetches pages with an HTTP client that mimics a browser. It's fast and light, but it can't render JavaScript.
34+
35+
### ParselCrawler
36+
37+
The <ApiLink to="class/ParselCrawler">`ParselCrawler`</ApiLink> is similar to the <ApiLink to="class/BeautifulSoupCrawler">`BeautifulSoupCrawler`</ApiLink> but uses the [Parsel](https://pypi.org/project/parsel/) library for HTML parsing. Parsel extracts data from HTML with CSS selectors and XPath, and it's the selector library [Scrapy](https://scrapy.org/) uses. Like the <ApiLink to="class/BeautifulSoupCrawler">`BeautifulSoupCrawler`</ApiLink>, the <ApiLink to="class/ParselCrawler">`ParselCrawler`</ApiLink> can't render JavaScript.
38+
39+
### PlaywrightCrawler
40+
41+
The <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> uses a headless browser controlled by the [Playwright](https://playwright.dev/) library. It can drive Chromium, Firefox, and WebKit. Choose it when the target pages render their content with JavaScript.
42+
43+
## Installation
44+
45+
Crawlee is available as the [`crawlee`](https://pypi.org/project/crawlee/) package on PyPI. The package includes the core functionality, and additional features are available as optional extras to keep dependencies small.
46+
47+
To install Crawlee with all extras using [pip](https://pip.pypa.io/en/stable/), run:
48+
49+
```sh
50+
python -m pip install 'crawlee[all]'
51+
```
52+
53+
Verify that Crawlee is successfully installed:
54+
55+
```sh
56+
python -c 'import crawlee; print(crawlee.__version__)'
57+
```
58+
59+
If you plan to use the <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink>, also install the Playwright browser binaries:
60+
61+
```sh
62+
playwright install
63+
```
64+
65+
For detailed installation instructions, see [Setting up](./introduction/setting-up).
66+
67+
## Crawling
68+
69+
Run the following example to perform a recursive crawl of the Crawlee website using the selected crawler.
70+
71+
<Tabs groupId="quickStart">
72+
<TabItem value="BeautifulSoupCrawler" label="BeautifulSoupCrawler" default>
73+
<RunnableCodeBlock className="language-python" language="python">
74+
{BeautifulsoupCrawlerExample}
75+
</RunnableCodeBlock>
76+
</TabItem>
77+
<TabItem value="ParselCrawler" label="ParselCrawler">
78+
<RunnableCodeBlock className="language-python" language="python">
79+
{ParselCrawlerExample}
80+
</RunnableCodeBlock>
81+
</TabItem>
82+
<TabItem value="PlaywrightCrawler" label="PlaywrightCrawler">
83+
<RunnableCodeBlock className="language-python" language="python">
84+
{PlaywrightCrawlerExample}
85+
</RunnableCodeBlock>
86+
</TabItem>
87+
</Tabs>
88+
89+
When you run the example, the crawler logs its progress to your terminal.
90+
91+
{/*TODO: improve the logging and add here a sample*/}
92+
93+
## Running headful browser
94+
95+
By default, browsers controlled by Playwright run in headless mode, without a visible window. During development, it helps to run the browser in headful mode and watch what it does. You can also switch from the default Chromium browser to Firefox or WebKit.
96+
97+
<CodeBlock language="python">
98+
{PlaywrightCrawlerHeadfulExample}
99+
</CodeBlock>
100+
101+
When you run the example, a browser window opens and navigates through the Crawlee website.
102+
103+
{/*TODO: add video example*/}
104+
105+
## Results
106+
107+
By default, Crawlee stores data in the `./storage` directory within your current working directory. The crawl results are saved as JSON files under `./storage/datasets/default/`.
108+
109+
To view the results, use the `cat` command:
110+
111+
```sh
112+
cat ./storage/datasets/default/000000001.json
113+
```
114+
115+
The JSON file contains data similar to the following:
116+
117+
```json
118+
{
119+
"url": "https://crawlee.dev/",
120+
"title": "Crawlee · Build reliable crawlers. Fast. | Crawlee"
121+
}
122+
```
123+
124+
:::tip
125+
126+
To change the storage directory, set the `CRAWLEE_STORAGE_DIR` environment variable to your preferred path.
127+
128+
:::
129+
130+
## Next steps
131+
132+
- To build a complete scraper for a real website step by step, follow the [Introduction](./introduction).
133+
- To learn how crawlers, requests, storages, and other parts of Crawlee work, read the [Concepts](./concepts).
134+
- For solutions to specific tasks, see the [Guides](./guides).
Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,15 @@
11
---
22
id: setting-up
33
title: Setting up
4+
description: How to install Crawlee, set up your environment, and bootstrap a new project.
45
---
56

67
import ApiLink from '@site/src/components/ApiLink';
78
import CodeBlock from '@theme/CodeBlock';
89
import Tabs from '@theme/Tabs';
910
import TabItem from '@theme/TabItem';
1011

11-
This guide will help you get started with Crawlee by setting it up on your computer. Follow the steps below to ensure a smooth installation process.
12+
This tutorial takes you from the simplest crawler, which prints a page title, to a scraper that crawls a real website and extracts product data. The first step is setting up Crawlee on your computer.
1213

1314
## Prerequisites
1415

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
---
22
id: first-crawler
33
title: First crawler
4+
description: Build your first Crawlee crawler - set up a request queue, write a request handler, and crawl your first page.
45
---
56

67
import ApiLink from '@site/src/components/ApiLink';

docs/introduction/03_adding_more_urls.mdx renamed to docs/02_introduction/03_adding_more_urls.mdx

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
---
22
id: adding-more-urls
33
title: Adding more URLs
4+
description: Grow the crawl by discovering and enqueuing new links, with filtering and deduplication handled for you.
45
---
56

67
import ApiLink from '@site/src/components/ApiLink';

docs/introduction/04_real_world_project.mdx renamed to docs/02_introduction/04_real_world_project.mdx

Lines changed: 1 addition & 0 deletions

0 commit comments

Comments
 (0)