chore: Add templates Apify E2E tests by Pijukatel · Pull Request #1109 · apify/crawlee-python · GitHub
Skip to content

chore: Add templates Apify E2E tests - #1109

Merged
Pijukatel merged 19 commits into
masterfrom
templates-integration-tests
Apr 3, 2025
Merged

Pijukatel merged 19 commits into
masterfrom
templates-integration-tests

Conversation

@Pijukatel

@Pijukatel Pijukatel commented Mar 20, 2025 •

Copy link
Copy Markdown
Collaborator

Description

Add crawlee templates integration tests with Apify platform. (only Parsel and Beautifulsoup templates)
Add workflow for manual triggering of integration tests (Will become automatic once it is running for a while to prove stability.)

Issues

Testing

Tested on forked repo:

Next steps

Add tests for Playwright based templates.
Fix current problems with Playwright template that has failing run in default settings.

Making the integration tests each run in separate job. Each test can take long time and can be unstable due to the network issues. This brings the necessity to have the possibility to re-trigger individual tests and not the whole test suite.

Since Playwright based crawlers are slower, this will be introduced together with their testing.
Example such run, where only one template test can be re-triggered:
image

@Pijukatel Pijukatel added t-tooling Issues with this label are in the ownership of the tooling team. debt Code quality improvement or decrease of technical debt. labels Mar 20, 2025
Comment thread pyproject.toml Outdated
@github-actions github-actions Bot added this to the 110th sprint - Tooling team milestone Mar 20, 2025
@github-actions github-actions Bot added the tested Temporary label used only programatically for some analytics. label Mar 20, 2025
@Pijukatel
Pijukatel requested a review from vdusek March 20, 2025 14:34
@Pijukatel
Pijukatel marked this pull request as ready for review March 20, 2025 14:34
@Pijukatel
Pijukatel requested a review from janbuchar March 20, 2025 14:35

@vdusek vdusek left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, just a few things...

Also, integration tests or end-to-end tests? 🙂 cc @janbuchar

Comment thread pyproject.toml
Comment on lines +38 to +39

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will it use the Python version we want and not the system one? Maybe we could use something from the GH marketplace, like for the uv?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that here it does not matter. This poetry is used only to inject the new wheel file to pyproject.toml and generate lock file. As far as I know, the lock file is generated not based on the current Python version, but based on the versions of the Python written in the pyproject.toml.

Poetry used to install the project just before the test is different one and is the one that is in the docker container.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's a justification for doing it wrong.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I understand your point. My point is, that the version here is not relevant, so in that case any attempt to enforce specific version is kind of misleading, but maybe I am wrong. Can you give me some example case where the current "version unspecific" solution causes an issue?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do I understand it right that poetry is only used for editing pyproject.toml and poetry.lock here? Because uv does not do it?

If that's the case, I'd just drop a comment above the installation step and call it a day.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Btw. Apart from no immediate benefit, another small additional reason why I am hesitant adding third party poetry github action is, that there is no "official" action like it is in case of uv. As far as I can see, poetry available actions are made by people unrelated to the poetry)

Comment thread tests/integration/conftest.py
Comment thread .github/workflows/templates_apify_integration_tests.yaml Outdated
@Pijukatel
Pijukatel requested a review from vdusek March 28, 2025 09:59
Comment thread CONTRIBUTING.md Outdated
Comment thread CONTRIBUTING.md Outdated
Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>
@janbuchar

janbuchar commented Mar 28, 2025 •

Copy link
Copy Markdown
Collaborator

Also, integration tests or end-to-end tests? 🙂 cc @janbuchar

Well, I'm slightly inclined towards calling them e2e.

Comment on lines +38 to +39
- name: Install poetry
run: pipx install poetry

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do I understand it right that poetry is only used for editing pyproject.toml and poetry.lock here? Because uv does not do it?

If that's the case, I'd just drop a comment above the installation step and call it a day.

name: Templates Apify integration tests

on:
workflow_dispatch:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we run this nightly, in addition to this?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but I would like to do it in two steps. First merge it with manual trigger and change it to automatic later when it is proven to work well in the repo.

Currently I am testing this workflow on forked repo and even though I had run it many times, it is still kind of a clumsy test setup, so it feels safer to merge it first with only manual trigger.

crawlee_extras = re.findall(r'crawlee(\[.*\])', pyproject)[0] or ''

# Inject crawlee wheel file to the docker image and update project to depend on it."""
with open(project_path / 'Dockerfile') as f:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Patching the lockfile and the Dockerfile seems pretty involved. Can't we replace apify push with make_actor from the SDK integration tests? Then you could just put the wheel inside the build context and pip-install it in the Dockerfile.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The point of this somewhat complicated way of patching is to ensure it tests both the template package managers and the way they install the project and that it is as similar to user workflow as possible.

By using the make actor similar approach I think I would do slightly different installation steps, but more importantly I would have to make sure that make actor is maintained together with any changes to apify-cli.

Imagine situation where there is some braking change to apify push. If I bypass the command by using make actor approach, then this test will be still passing, while out in the wilds it will start crashing for the people.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Huh, testing apify push sounds way out of scope even for end-to-end tests. Also, non-backward-compatible changes to the API for building stuff are super unlikely. We already have a make_actor implementation in the SDK, so it would make a lot of sense to me to use the same approach here.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make_actor is in different repo and is designed for code that is in the string format as the actors are generated on the fly by the test code. Here the actors are generated the way user generates them as a folder structure with files.

I would suggest to start with this approach and iteratively improve it later based on any potential issues we encounter. The tests are written in a way, that this discussed "patch function" is well isolated from the rest of the test, so we can just change the patch function and the rest of the test remains the same.

@Pijukatel
Pijukatel marked this pull request as draft March 28, 2025 20:15
@Pijukatel
Pijukatel marked this pull request as ready for review March 31, 2025 07:05
@Pijukatel

Copy link
Copy Markdown
Collaborator Author

@Pijukatel
Pijukatel requested a review from vdusek March 31, 2025 07:06
Comment thread Makefile Outdated
Comment thread Makefile Outdated
Comment thread pyproject.toml Outdated
Comment thread tests/e2e/project_template/test_static_crawlers_templates.py Outdated
Comment thread tests/e2e/project_template/test_static_crawlers_templates.py Outdated
Comment thread tests/integration/README.md
@Pijukatel
Pijukatel requested a review from vdusek March 31, 2025 14:46
@vdusek vdusek changed the title chore: Add templates apify integration tests chore: Add templates Apify E2E tests Apr 1, 2025

@vdusek vdusek left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Pijukatel
Pijukatel merged commit ab5234b into master Apr 3, 2025
@Pijukatel
Pijukatel deleted the templates-integration-tests branch April 3, 2025 08:31
Pijukatel added a commit that referenced this pull request Apr 7, 2025
…#1136)

### Description
- Add e2e tests for playwright templates
- Add e2e tests for pip-based templates
- Fix template issue with pip template that has additional dependencies
- Separate all e2e tests in CI to run in it's own runner to allow
selective re-triggers (e2e tests work on real network and it will always
have some flakiness due to that)
- Reduce `max_requests_per_crawl` to 10 in all templates to be aligned
with doc examples and to make the tests faster.

### Issues

- Closes: #1109
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

debt Code quality improvement or decrease of technical debt. t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add integration tests for actor templates

3 participants