chore: Add templates Apify E2E tests (#1109) · apify/crawlee-python@ab5234b · GitHub
Skip to content

Commit ab5234b

Browse files
Pijukatelvdusek
andauthored
chore: Add templates Apify E2E tests (#1109)
### Description Add `crawlee` templates integration tests with `Apify` platform. (only `Parsel` and `Beautifulsoup` templates) Add workflow for manual triggering of integration tests (Will become automatic once it is running for a while to prove stability.) ### Issues - Closes: #1093 ### Testing Tested on forked repo: - Example of no issue run: https://github.com/Pijukatel/crawlee-python/actions/runs/13969844251 - Example of error in the template docker build (bad change in`uv` related part of the dockerfile): https://github.com/Pijukatel/crawlee-python/actions/runs/13970390933 - Example of error in the actor execution expectations (bad change in `parsel` actor template): https://github.com/Pijukatel/crawlee-python/actions/runs/13970267119 ### Next steps Add tests for `Playwright` based templates. [Fix current problems with `Playwright`](#1110) template that has failing run in default settings. Making the integration tests each run in separate job. Each test can take long time and can be unstable due to the network issues. This brings the necessity to have the possibility to re-trigger individual tests and not the whole test suite. Since Playwright based crawlers are slower, this will be introduced together with their testing. Example such run, where only one template test can be re-triggered: ![image](https://github.com/user-attachments/assets/0074edd3-ed14-4cc1-85fa-ae8cee5c56df) --------- Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>
1 parent 77cbcc1 commit ab5234b

14 files changed

Lines changed: 333 additions & 18 deletions

File tree

Lines changed: 53 additions & 0 deletions

‎CONTRIBUTING.md‎

Lines changed: 11 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -72,13 +72,19 @@ To run unit tests with HTML coverage report:
7272
make unit-tests-cov
7373
```
7474

75-
<!--
76-
TODO:
75+
## End-to-end tests
7776

78-
## Integration tests
77+
Pre-requisites for running end-to-end tests:
78+
- [apify-cli](https://docs.apify.com/cli/docs/installation) correctly installed
79+
- `apify-cli` available in `PATH` environment variable
80+
- Your [apify token](https://docs.apify.com/platform/integrations/api#api-token) is available in `APIFY_TEST_USER_API_TOKEN` environment variable
7981

80-
...
81-
-->
82+
83+
To run end-to-end tests:
84+
85+
```sh
86+
make e2e-templates-tests
87+
```
8288

8389
## Documentation
8490

‎Makefile‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
integration-tests format check-code build-api-reference run-docs
33

44
# This is default for local testing, but GitHub workflows override it to a higher value in CI
5-
INTEGRATION_TESTS_CONCURRENCY = 1
5+
E2E_TESTS_CONCURRENCY = 1
66

77
clean:
88
rm -rf .mypy_cache .pytest_cache .ruff_cache build dist htmlcov .coverage
@@ -32,8 +32,8 @@ unit-tests:
3232
unit-tests-cov:
3333
uv run pytest --numprocesses=auto --verbose --cov=src/crawlee --cov-report=html tests/unit
3434

35-
integration-tests:
36-
uv run pytest --numprocesses=$(INTEGRATION_TESTS_CONCURRENCY) --verbose tests/integration
35+
e2e-templates-tests:
36+
uv run pytest --numprocesses=$(E2E_TESTS_CONCURRENCY) --verbose tests/e2e/project_template
3737

3838
format:
3939
uv run ruff check --fix

‎pyproject.toml‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -92,6 +92,8 @@ crawlee = "crawlee._cli:cli"
9292

9393
[dependency-groups]
9494
dev = [
95+
"apify_client", # For e2e tests.
96+
"build~=1.2.2", # For e2e tests.
9597
"mypy~=1.15.0",
9698
"pre-commit~=4.2.0",
9799
"proxy-py~=2.4.0",
@@ -119,7 +121,7 @@ packages = ["src/crawlee"]
119121
[tool.ruff]
120122
line-length = 120
121123
include = ["src/**/*.py", "tests/**/*.py", "docs/**/*.py", "website/**/*.py"]
122-
extend-exclude = ["project_template"]
124+
extend-exclude = ["src/crawlee/project_template"]
123125

124126
[tool.ruff.lint]
125127
select = ["ALL"]
@@ -207,7 +209,7 @@ timeout = 1200
207209
[tool.mypy]
208210
python_version = "3.9"
209211
plugins = ["pydantic.mypy"]
210-
exclude = ["project_template"]
212+
exclude = ["src/crawlee/project_template"]
211213
files = ["src", "tests", "docs", "website"]
212214
check_untyped_defs = true
213215
disallow_incomplete_defs = true
@@ -229,6 +231,8 @@ module = [
229231
"jaro", # Untyped and stubs not available
230232
"loguru", # Example code shows integration of loguru and crawlee for JSON logging.
231233
"sklearn.linear_model", # Untyped and stubs not available
234+
"cookiecutter.*", # Untyped and stubs not available
235+
"inquirer.*", # Untyped and stubs not available
232236
]
233237
ignore_missing_imports = true
234238

‎src/crawlee/_cli.py‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7,10 +7,10 @@
77
from typing import Annotated, Optional, cast
88

99
try:
10-
import inquirer # type: ignore[import-untyped]
10+
import inquirer
1111
import typer
12-
from cookiecutter.main import cookiecutter # type: ignore[import-untyped]
13-
from inquirer.render.console import ConsoleRender # type: ignore[import-untyped]
12+
from cookiecutter.main import cookiecutter
13+
from inquirer.render.console import ConsoleRender
1414
from rich.progress import Progress, SpinnerColumn, TextColumn
1515
except ModuleNotFoundError as exc:
1616
raise ImportError(
@@ -21,7 +21,8 @@
2121
cli = typer.Typer(no_args_is_help=True)
2222

2323
template_directory = importlib.resources.files('crawlee') / 'project_template'
24-
cookiecutter_json = json.load((template_directory / 'cookiecutter.json').open())
24+
with open(str(template_directory / 'cookiecutter.json')) as f:
25+
cookiecutter_json = json.load(f)
2526

2627
crawler_choices = cookiecutter_json['crawler_type']
2728
http_client_choices = cookiecutter_json['http_client']

‎src/crawlee/project_template/hooks/post_gen_project.py‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,6 @@
22
import subprocess
33
from pathlib import Path
44

5-
Path('_pyproject.toml').rename('pyproject.toml')
65

76
# % if cookiecutter.package_manager in ['poetry', 'uv']
87
Path('requirements.txt').unlink()

‎src/crawlee/project_template/{{cookiecutter.project_name}}/Dockerfile‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,8 +20,7 @@ RUN pip install -U pip setuptools \
2020
# Second, copy just poetry.lock and pyproject.toml into the Actor image,
2121
# since those should be the only files that affects the dependency install in the next step,
2222
# in order to speed up the build
23-
COPY pyproject.toml ./
24-
COPY poetry.lock ./
23+
COPY pyproject.toml poetry.lock ./
2524

2625
# Install the dependencies
2726
RUN echo "Python version:" \

src/crawlee/project_template/{{cookiecutter.project_name}}/_pyproject.toml renamed to src/crawlee/project_template/{{cookiecutter.project_name}}/pyproject.toml

File renamed without changes.

‎tests/e2e/__init__.py‎

Whitespace-only changes.

‎tests/e2e/conftest.py‎

Lines changed: 44 additions & 0 deletions

0 commit comments

Comments
 (0)