chore: Add e2e tests for playwright templates and pip based templates… · realam/crawlee-python@8ca6f0d · GitHub
Skip to content

Commit 8ca6f0d

Browse files
authored
chore: Add e2e tests for playwright templates and pip based templates (apify#1136)
### Description - Add e2e tests for playwright templates - Add e2e tests for pip-based templates - Fix template issue with pip template that has additional dependencies - Separate all e2e tests in CI to run in it's own runner to allow selective re-triggers (e2e tests work on real network and it will always have some flakiness due to that) - Reduce `max_requests_per_crawl` to 10 in all templates to be aligned with doc examples and to make the tests faster. ### Issues - Closes: apify#1109
1 parent 196555b commit 8ca6f0d

11 files changed

Lines changed: 136 additions & 38 deletions

File tree

‎.github/workflows/templates_e2e_tests.yaml‎

Lines changed: 10 additions & 6 deletions

‎Makefile‎

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,17 @@
1-
.PHONY: clean install-dev build publish-to-pypi lint type-check unit-tests unit-tests-cov \
2-
integration-tests format check-code build-api-reference run-docs
1+
.PHONY: clean install-sync install-dev build publish-to-pypi lint type-check unit-tests unit-tests-cov \
2+
e2e-templates-tests format check-code build-api-reference run-docs
33

44
# This is default for local testing, but GitHub workflows override it to a higher value in CI
55
E2E_TESTS_CONCURRENCY = 1
66

77
clean:
88
rm -rf .mypy_cache .pytest_cache .ruff_cache build dist htmlcov .coverage
99

10-
install-dev:
10+
install-sync:
1111
uv sync --all-extras
12+
13+
install-dev:
14+
make install-sync
1215
uv run pre-commit install
1316
uv run playwright install
1417

@@ -32,8 +35,8 @@ unit-tests:
3235
unit-tests-cov:
3336
uv run pytest --numprocesses=auto --verbose --cov=src/crawlee --cov-report=html tests/unit
3437

35-
e2e-templates-tests:
36-
uv run pytest --numprocesses=$(E2E_TESTS_CONCURRENCY) --verbose tests/e2e/project_template
38+
e2e-templates-tests $(args):
39+
uv run pytest --numprocesses=$(E2E_TESTS_CONCURRENCY) --verbose tests/e2e/project_template "$(args)"
3740

3841
format:
3942
uv run ruff check --fix

‎src/crawlee/project_template/templates/main_beautifulsoup.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,6 @@
77
# % block instantiation
88
crawler = BeautifulSoupCrawler(
99
request_handler=router,
10-
max_requests_per_crawl=50,
10+
max_requests_per_crawl=10,
1111
{{ self.http_client_instantiation() }})
1212
# % endblock

‎src/crawlee/project_template/templates/main_parsel.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,6 @@
77
# % block instantiation
88
crawler = ParselCrawler(
99
request_handler=router,
10-
max_requests_per_crawl=50,
10+
max_requests_per_crawl=10,
1111
{{ self.http_client_instantiation() }})
1212
# % endblock

‎src/crawlee/project_template/templates/main_playwright.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,6 @@
88
crawler = PlaywrightCrawler(
99
request_handler=router,
1010
headless=True,
11-
max_requests_per_crawl=50,
11+
max_requests_per_crawl=10,
1212
{{ self.http_client_instantiation() }})
1313
# % endblock

‎src/crawlee/project_template/{{cookiecutter.project_name}}/Dockerfile‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -72,8 +72,8 @@ RUN echo "Python version:" \
7272
&& cat requirements.txt | \
7373
# Replace playwright version so that it matches whatever is pre-installed in the image (the `hash` checks if playwright is installed)
7474
sed "s/^playwright==\(.*\)/playwright==$(hash playwright 2>/dev/null && (playwright --version | cut -d ' ' -f 2) || echo '\1')/" | \
75-
# Install everything using pip (ignore dependency checks - the lockfile is correct, period)
76-
pip install -r /dev/stdin --no-dependencies \
75+
# Install everything using pip
76+
pip install -r /dev/stdin \
7777
&& echo "All installed Python packages:" \
7878
&& pip freeze
7979
# % elif cookiecutter.package_manager == 'manual'

‎src/crawlee/project_template/{{cookiecutter.project_name}}/pyproject.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ requires-python = ">=3.9,<4.0"
1919
dependencies = [
2020
"crawlee[{{ extras|join(',') }}]",
2121
# % if cookiecutter.crawler_type == 'playwright-camoufox'
22-
"camoufox[geoip]>=0.4.5",
22+
"camoufox[geoip]~=0.4.5",
2323
# % endif
2424
# % if cookiecutter.enable_apify_integration
2525
"apify",

‎src/crawlee/project_template/{{cookiecutter.project_name}}/requirements.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# % if cookiecutter.crawler_type == 'playwright-camoufox'
2-
camoufox
2+
camoufox[geoip]~=0.4.5
33
# % set extras = ['playwright']
44
# % else
55
# % set extras = [cookiecutter.crawler_type]

‎tests/e2e/conftest.py‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,11 +2,27 @@
22
from pathlib import Path
33

44
import pytest
5+
from _pytest.config import Config
56
from filelock import FileLock
67

78
_CRAWLEE_ROOT_PATH = Path(__file__).parent.parent.parent.resolve()
89

910

11+
def pytest_configure(config: Config) -> None:
12+
for marker in [
13+
'httpx',
14+
'curl_impersonate',
15+
'playwright',
16+
'playwright_camoufox',
17+
'parsel',
18+
'beautifulsoup',
19+
'uv',
20+
'poetry',
21+
'pip',
22+
]:
23+
config.addinivalue_line('markers', f'{marker}: Integration test parameter marker.')
24+
25+
1026
@pytest.fixture(scope='session')
1127
def crawlee_wheel_path(tmp_path_factory: pytest.TempPathFactory, testrun_uid: str) -> Path:
1228
"""Build the package wheel if it hasn't been built yet, and return the path to the wheel."""

‎tests/e2e/project_template/test_static_crawlers_templates.py‎

Lines changed: 41 additions & 12 deletions

0 commit comments

Comments
 (0)