fix: error handling in CLI and templates documentation (#273) · lexsf/crawlee-python@61083c3 · GitHub
Skip to content

Commit 61083c3

Browse files
authored
fix: error handling in CLI and templates documentation (apify#273)
### Description - [x] Rename `local_storage_dir` to `storage_dir`. - [x] Mention templates and CLI in readme. - [x] Mention templates and CLI in doc - [x] Fix CLI when project directory already exists. - [x] Handle CTRL+C in CLI. - ~~Poetry templates to Pip templates.~~ ### Issues - Related apify#266 - Closes: apify#268 ### Testing - All scenarios were executed locally. Handling CTRL+C is still not perfect. However, this should be ok for the initial release. ### Checklist - [x] Changes are described in the `CHANGELOG.md` - [x] CI passed
1 parent ed88d8d commit 61083c3

20 files changed

Lines changed: 226 additions & 168 deletions

File tree

‎.github/pull_request_template.md‎

Lines changed: 2 additions & 2 deletions

‎CHANGELOG.md‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,15 @@
11
# Changelog
22

3-
## [0.0.8](../../releases/tag/v0.0.8) - Unreleased
3+
## [0.1.0](../../releases/tag/v0.1.0) - Unreleased
44

55
### Adds
66

77
- new project bootstrapping via `pipx run crawlee create`
88

9+
### Fixes
10+
11+
- improve error handling in project bootstrapping
12+
913
## [0.0.7](../../releases/tag/v0.0.7) - 2024-06-27
1014

1115
### Fixes

‎CONTRIBUTING.md‎

Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -12,15 +12,15 @@ We use [Poetry](https://python-poetry.org/) for project management. Install it a
1212

1313
To install this package and its development dependencies, run:
1414

15-
```bash
15+
```sh
1616
make install-dev
1717
```
1818

1919
## Code checking
2020

2121
To execute all code checking tools together, run:
2222

23-
```bash
23+
```sh
2424
make check-code
2525
```
2626

@@ -30,7 +30,7 @@ We utilize [ruff](https://docs.astral.sh/ruff/) for linting, which analyzes code
3030

3131
To run linting:
3232

33-
```bash
33+
```sh
3434
make lint
3535
```
3636

@@ -40,7 +40,7 @@ Our automated code formatting also leverages [ruff](https://docs.astral.sh/ruff/
4040

4141
To run formatting:
4242

43-
```bash
43+
```sh
4444
make format
4545
```
4646

@@ -50,7 +50,7 @@ Type checking is handled by [mypy](https://mypy.readthedocs.io/), verifying code
5050

5151
To run type checking:
5252

53-
```bash
53+
```sh
5454
make type-check
5555
```
5656

@@ -62,13 +62,13 @@ We use [pytest](https://docs.pytest.org/) as a testing framework with many plugi
6262

6363
To run unit tests:
6464

65-
```bash
65+
```sh
6666
make unit-tests
6767
```
6868

6969
To run unit tests with HTML coverage report:
7070

71-
```bash
71+
```sh
7272
make unit-tests-cov
7373
```
7474

@@ -90,25 +90,25 @@ To run the documentation locally, you need to have Node.js version 20 or higher
9090

9191
Navigate to the `website/` directory:
9292

93-
```bash
93+
```sh
9494
cd website/
9595
```
9696

9797
Enable Corepack, which installs Yarn automatically:
9898

99-
```bash
99+
```sh
100100
corepack enable
101101
```
102102

103103
Install the necessary dependencies:
104104

105-
```bash
105+
```sh
106106
yarn
107107
```
108108

109109
Start the project in development mode with Hot Module Replacement (HMR):
110110

111-
```bash
111+
```sh
112112
yarn start
113113
```
114114

‎README.md‎

Lines changed: 40 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@
1111

1212
Crawlee covers your crawling and scraping end-to-end and **helps you build reliable scrapers. Fast.**
1313

14-
Your crawlers will appear almost human-like and fly under the radar of modern bot protections even with the default configuration. Crawlee gives you the tools to crawl the web for links, scrape data, and store it to disk or cloud while staying configurable to suit your project's needs.
14+
Your crawlers will appear almost human-like and fly under the radar of modern bot protections even with the default configuration. Crawlee gives you the tools to crawl the web for links, scrape data and persistently store it in machine-readable formats, without having to worry about the technical details. And thanks to rich configuration options, you can tweak almost any aspect of Crawlee to suit your project's needs if the default settings don't cut it.
1515

1616
> 👉 **View full documentation, guides and examples on the [Crawlee project website](https://crawlee.dev/python/)** 👈
1717
@@ -23,64 +23,53 @@ We recommend visiting the [Introduction tutorial](https://crawlee.dev/python/doc
2323

2424
Crawlee is available as the [`crawlee`](https://pypi.org/project/crawlee/) PyPI package.
2525

26-
```bash
26+
```sh
2727
pip install crawlee
2828
```
2929

3030
Additional, optional dependencies unlocking more features are shipped as package extras.
3131

3232
If you plan to use `BeautifulSoupCrawler`, install `crawlee` with `beautifulsoup` extra:
3333

34-
```bash
34+
```sh
3535
pip install 'crawlee[beautifulsoup]'
3636
```
3737

3838
If you plan to use `PlaywrightCrawler`, install `crawlee` with the `playwright` extra:
3939

40-
```bash
40+
```sh
4141
pip install 'crawlee[playwright]'
4242
```
4343

4444
Then, install the Playwright dependencies:
4545

46-
```bash
46+
```sh
4747
playwright install
4848
```
4949

5050
You can install multiple extras at once by using a comma as a separator:
5151

52-
```bash
52+
```sh
5353
pip install 'crawlee[beautifulsoup,playwright]'
5454
```
5555

56-
## Features
57-
58-
Why Crawlee is the preferred choice for web scraping and crawling?
56+
### With Crawlee CLI
5957

60-
### Why use Crawlee instead of just a random HTTP library with an HTML parser?
61-
62-
- Unified interface for **HTTP & headless browser** crawling.
63-
- Automatic **parallel crawling** based on available system resources.
64-
- Written in Python with **type hints** - enhances DX (IDE autocompletion) and reduces bugs (static type checking).
65-
- Automatic **retries** on errors or when you’re getting blocked.
66-
- Integrated **proxy rotation** and session management.
67-
- Configurable **request routing** - direct URLs to the appropriate handlers.
68-
- Persistent **queue for URLs** to crawl.
69-
- Pluggable **storage** of both tabular data and files.
70-
- Robust **error handling**.
58+
The quickest way to get started with Crawlee is by using the Crawlee CLI and selecting one of the prepared templates. First, ensure you have [Pipx](https://pipx.pypa.io/) installed:
7159

72-
### Why to use Crawlee rather than Scrapy?
60+
```sh
61+
pipx --help
62+
```
7363

74-
- Crawlee has out-of-the-box support for **headless browser** crawling (Playwright).
75-
- Crawlee has a **minimalistic & elegant interface** - Set up your scraper with fewer than 10 lines of code.
76-
- Complete **type hint** coverage.
77-
- Based on standard **Asyncio**.
64+
Then, run the CLI and choose from the available templates:
7865

79-
## Introduction
66+
```sh
67+
pipx run crawlee create my-crawler
68+
```
8069

81-
Crawlee covers your crawling and scraping end-to-end and helps you build reliable scrapers. Fast.
70+
## Examples
8271

83-
Your crawlers will appear human-like and fly under the radar of modern bot protections even with the default configuration. Crawlee gives you the tools to crawl the web for links, scrape data and persistently store it in machine-readable formats, without having to worry about the technical details. And thanks to rich configuration options, you can tweak almost any aspect of Crawlee to suit your project's needs if the default settings don't cut it.
72+
Here are some practical examples to help you get started with different types of crawlers in Crawlee. Each example demonstrates how to set up and run a crawler for specific use cases, whether you need to handle simple HTML pages or interact with JavaScript-heavy sites.
8473

8574
### BeautifulSoupCrawler
8675

@@ -167,6 +156,29 @@ if __name__ == '__main__':
167156

168157
Explore our [Examples](https://crawlee.dev/python/docs/examples) page in the Crawlee documentation for a wide range of additional use cases and demonstrations.
169158

159+
## Features
160+
161+
Why Crawlee is the preferred choice for web scraping and crawling?
162+
163+
### Why use Crawlee instead of just a random HTTP library with an HTML parser?
164+
165+
- Unified interface for **HTTP & headless browser** crawling.
166+
- Automatic **parallel crawling** based on available system resources.
167+
- Written in Python with **type hints** - enhances DX (IDE autocompletion) and reduces bugs (static type checking).
168+
- Automatic **retries** on errors or when you’re getting blocked.
169+
- Integrated **proxy rotation** and session management.
170+
- Configurable **request routing** - direct URLs to the appropriate handlers.
171+
- Persistent **queue for URLs** to crawl.
172+
- Pluggable **storage** of both tabular data and files.
173+
- Robust **error handling**.
174+
175+
### Why to use Crawlee rather than Scrapy?
176+
177+
- Crawlee has out-of-the-box support for **headless browser** crawling (Playwright).
178+
- Crawlee has a **minimalistic & elegant interface** - Set up your scraper with fewer than 10 lines of code.
179+
- Complete **type hint** coverage.
180+
- Based on standard **Asyncio**.
181+
170182
## Running on the Apify platform
171183

172184
Crawlee is open-source and runs anywhere, but since it's developed by [Apify](https://apify.com), it's easy to set up on the Apify platform and run in the cloud. Visit the [Apify SDK website](https://docs.apify.com/sdk/python/) to learn more about deploying Crawlee to the Apify platform.

‎docs/introduction/01-setting-up.md‎

Lines changed: 21 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -10,55 +10,67 @@ To run Crawlee on your computer, ensure you meet the following requirements:
1010

1111
You can verify these by running the following commands:
1212

13-
```bash
13+
```sh
1414
python --version
1515
```
1616

17-
```bash
17+
```sh
1818
pip --version
1919
```
2020

2121
## Installation
2222

2323
Crawlee is available as the [`crawlee`](https://pypi.org/project/crawlee/) PyPI package.
2424

25-
```bash
25+
```sh
2626
pip install crawlee
2727
```
2828

2929
Additional, optional dependencies unlocking more features are shipped as package extras.
3030

3131
If you plan to use `BeautifulSoupCrawler`, install `crawlee` with `beautifulsoup` extra:
3232

33-
```bash
33+
```sh
3434
pip install 'crawlee[beautifulsoup]'
3535
```
3636

3737
If you plan to use `PlaywrightCrawler`, install `crawlee` with the `playwright` extra:
3838

39-
```bash
39+
```sh
4040
pip install 'crawlee[playwright]'
4141
```
4242

4343
Then, install the Playwright dependencies:
4444

45-
```bash
45+
```sh
4646
playwright install
4747
```
4848

4949
You can install multiple extras at once by using a comma as a separator:
5050

51-
```bash
51+
```sh
5252
pip install 'crawlee[beautifulsoup,playwright]'
5353
```
5454

5555
Verify that Crawlee is successfully installed:
5656

57-
```bash
57+
```sh
5858
python -c 'import crawlee; print(crawlee.__version__)'
5959
```
6060

61-
<!-- TODO: Update this section once templates are ready -->
61+
## With Crawlee CLI
62+
63+
The quickest way to get started with Crawlee is by using the Crawlee CLI and selecting one of the prepared templates. First, ensure you have [Pipx](https://pipx.pypa.io/) installed:
64+
65+
```sh
66+
pipx --help
67+
```
68+
69+
Then, run the CLI and choose from the available templates:
70+
71+
```sh
72+
pipx run crawlee create my-crawler
73+
```
6274

6375
## Next steps
6476

‎docs/quick-start/index.md‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -30,33 +30,33 @@ Crawlee requires Python 3.9 or later.
3030

3131
Crawlee is available as the [`crawlee`](https://pypi.org/project/crawlee/) PyPI package.
3232

33-
```bash
33+
```sh
3434
pip install crawlee
3535
```
3636

3737
Additional, optional dependencies unlocking more features are shipped as package extras.
3838

3939
If you plan to use `BeautifulSoupCrawler`, install `crawlee` with `beautifulsoup` extra:
4040

41-
```bash
41+
```sh
4242
pip install 'crawlee[beautifulsoup]'
4343
```
4444

4545
If you plan to use `PlaywrightCrawler`, install `crawlee` with the `playwright` extra:
4646

47-
```bash
47+
```sh
4848
pip install 'crawlee[playwright]'
4949
```
5050

5151
Then, install the Playwright dependencies:
5252

53-
```bash
53+
```sh
5454
playwright install
5555
```
5656

5757
You can install multiple extras at once by using a comma as a separator:
5858

59-
```bash
59+
```sh
6060
pip install 'crawlee[beautifulsoup,playwright]'
6161
```
6262

@@ -182,7 +182,7 @@ The JSON file will contain data similar to the following:
182182

183183
:::tip
184184

185-
If you want to change the storage directory, you can set the `CRAWLEE_LOCAL_STORAGE_DIR` environment variable to your preferred path.
185+
If you want to change the storage directory, you can set the `CRAWLEE_STORAGE_DIR` environment variable to your preferred path.
186186

187187
:::
188188

‎pyproject.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ build-backend = "poetry.core.masonry.api"
44

55
[tool.poetry]
66
name = "crawlee"
7-
version = "0.0.8"
7+
version = "0.1.0"
88
description = "Crawlee for Python"
99
authors = ["Apify Technologies s.r.o. <support@apify.com>"]
1010
license = "Apache-2.0"

‎src/crawlee/__init__.py‎

Lines changed: 1 addition & 1 deletion

0 commit comments

Comments
 (0)