You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 61083c3
Browse filesBrowse the repository at this point in the historyBrowse files
fix: error handling in CLI and templates documentation (apify#273)
### Description
- [x] Rename `local_storage_dir` to `storage_dir`.
- [x] Mention templates and CLI in readme.
- [x] Mention templates and CLI in doc
- [x] Fix CLI when project directory already exists.
- [x] Handle CTRL+C in CLI.
- ~~Poetry templates to Pip templates.~~
### Issues
- Related apify#266
- Closes: apify#268
### Testing
- All scenarios were executed locally. Handling CTRL+C is still not
perfect. However, this should be ok for the initial release.
### Checklist
- [x] Changes are described in the `CHANGELOG.md`
- [x] CI passed
Copy file name to clipboardExpand all lines: README.md
+40-28Lines changed: 40 additions & 28 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,7 +11,7 @@
11
11
12
12
Crawlee covers your crawling and scraping end-to-end and **helps you build reliable scrapers. Fast.**
13
13
14
-
Your crawlers will appear almost human-like and fly under the radar of modern bot protections even with the default configuration. Crawlee gives you the tools to crawl the web for links, scrape data, and store it to disk or cloud while staying configurable to suit your project's needs.
14
+
Your crawlers will appear almost human-like and fly under the radar of modern bot protections even with the default configuration. Crawlee gives you the tools to crawl the web for links, scrape data and persistently store it in machine-readable formats, without having to worry about the technical details. And thanks to rich configuration options, you can tweak almost any aspect of Crawlee to suit your project's needs if the default settings don't cut it.
15
15
16
16
> 👉 **View full documentation, guides and examples on the [Crawlee project website](https://crawlee.dev/python/)** 👈
17
17
@@ -23,64 +23,53 @@ We recommend visiting the [Introduction tutorial](https://crawlee.dev/python/doc
23
23
24
24
Crawlee is available as the [`crawlee`](https://pypi.org/project/crawlee/) PyPI package.
25
25
26
-
```bash
26
+
```sh
27
27
pip install crawlee
28
28
```
29
29
30
30
Additional, optional dependencies unlocking more features are shipped as package extras.
31
31
32
32
If you plan to use `BeautifulSoupCrawler`, install `crawlee` with `beautifulsoup` extra:
33
33
34
-
```bash
34
+
```sh
35
35
pip install 'crawlee[beautifulsoup]'
36
36
```
37
37
38
38
If you plan to use `PlaywrightCrawler`, install `crawlee` with the `playwright` extra:
39
39
40
-
```bash
40
+
```sh
41
41
pip install 'crawlee[playwright]'
42
42
```
43
43
44
44
Then, install the Playwright dependencies:
45
45
46
-
```bash
46
+
```sh
47
47
playwright install
48
48
```
49
49
50
50
You can install multiple extras at once by using a comma as a separator:
51
51
52
-
```bash
52
+
```sh
53
53
pip install 'crawlee[beautifulsoup,playwright]'
54
54
```
55
55
56
-
## Features
57
-
58
-
Why Crawlee is the preferred choice for web scraping and crawling?
56
+
### With Crawlee CLI
59
57
60
-
### Why use Crawlee instead of just a random HTTP library with an HTML parser?
61
-
62
-
- Unified interface for **HTTP & headless browser** crawling.
63
-
- Automatic **parallel crawling** based on available system resources.
64
-
- Written in Python with **type hints** - enhances DX (IDE autocompletion) and reduces bugs (static type checking).
65
-
- Automatic **retries** on errors or when you’re getting blocked.
66
-
- Integrated **proxy rotation** and session management.
67
-
- Configurable **request routing** - direct URLs to the appropriate handlers.
68
-
- Persistent **queue for URLs** to crawl.
69
-
- Pluggable **storage** of both tabular data and files.
70
-
- Robust **error handling**.
58
+
The quickest way to get started with Crawlee is by using the Crawlee CLI and selecting one of the prepared templates. First, ensure you have [Pipx](https://pipx.pypa.io/) installed:
71
59
72
-
### Why to use Crawlee rather than Scrapy?
60
+
```sh
61
+
pipx --help
62
+
```
73
63
74
-
- Crawlee has out-of-the-box support for **headless browser** crawling (Playwright).
75
-
- Crawlee has a **minimalistic & elegant interface** - Set up your scraper with fewer than 10 lines of code.
76
-
- Complete **type hint** coverage.
77
-
- Based on standard **Asyncio**.
64
+
Then, run the CLI and choose from the available templates:
78
65
79
-
## Introduction
66
+
```sh
67
+
pipx run crawlee create my-crawler
68
+
```
80
69
81
-
Crawlee covers your crawling and scraping end-to-end and helps you build reliable scrapers. Fast.
70
+
## Examples
82
71
83
-
Your crawlers will appear human-like and fly under the radar of modern bot protections even with the default configuration. Crawlee gives you the tools to crawl the web for links, scrape data and persistently store it in machine-readable formats, without having to worry about the technical details. And thanks to rich configuration options, you can tweak almost any aspect of Crawlee to suit your project's needs if the default settings don't cut it.
72
+
Here are some practical examples to help you get started with different types of crawlers in Crawlee. Each example demonstrates how to set up and run a crawler for specific use cases, whether you need to handle simple HTML pages or interact with JavaScript-heavy sites.
84
73
85
74
### BeautifulSoupCrawler
86
75
@@ -167,6 +156,29 @@ if __name__ == '__main__':
167
156
168
157
Explore our [Examples](https://crawlee.dev/python/docs/examples) page in the Crawlee documentation for a wide range of additional use cases and demonstrations.
169
158
159
+
## Features
160
+
161
+
Why Crawlee is the preferred choice for web scraping and crawling?
162
+
163
+
### Why use Crawlee instead of just a random HTTP library with an HTML parser?
164
+
165
+
- Unified interface for **HTTP & headless browser** crawling.
166
+
- Automatic **parallel crawling** based on available system resources.
167
+
- Written in Python with **type hints** - enhances DX (IDE autocompletion) and reduces bugs (static type checking).
168
+
- Automatic **retries** on errors or when you’re getting blocked.
169
+
- Integrated **proxy rotation** and session management.
170
+
- Configurable **request routing** - direct URLs to the appropriate handlers.
171
+
- Persistent **queue for URLs** to crawl.
172
+
- Pluggable **storage** of both tabular data and files.
173
+
- Robust **error handling**.
174
+
175
+
### Why to use Crawlee rather than Scrapy?
176
+
177
+
- Crawlee has out-of-the-box support for **headless browser** crawling (Playwright).
178
+
- Crawlee has a **minimalistic & elegant interface** - Set up your scraper with fewer than 10 lines of code.
179
+
- Complete **type hint** coverage.
180
+
- Based on standard **Asyncio**.
181
+
170
182
## Running on the Apify platform
171
183
172
184
Crawlee is open-source and runs anywhere, but since it's developed by [Apify](https://apify.com), it's easy to set up on the Apify platform and run in the cloud. Visit the [Apify SDK website](https://docs.apify.com/sdk/python/) to learn more about deploying Crawlee to the Apify platform.
<!-- TODO: Update this section once templates are ready -->
61
+
## With Crawlee CLI
62
+
63
+
The quickest way to get started with Crawlee is by using the Crawlee CLI and selecting one of the prepared templates. First, ensure you have [Pipx](https://pipx.pypa.io/) installed:
64
+
65
+
```sh
66
+
pipx --help
67
+
```
68
+
69
+
Then, run the CLI and choose from the available templates:
0 commit comments