Repository navigation
chore: Add templates Apify E2E tests - #1109
Conversation
TODO: Adapt it to paltform environment. Generalize.
TODO: Test in github
vdusek
left a comment
There was a problem hiding this comment.
Looks good, just a few things...
Also, integration tests or end-to-end tests? 🙂 cc @janbuchar
There was a problem hiding this comment.
Will it use the Python version we want and not the system one? Maybe we could use something from the GH marketplace, like for the uv?
There was a problem hiding this comment.
I think that here it does not matter. This poetry is used only to inject the new wheel file to pyproject.toml and generate lock file. As far as I know, the lock file is generated not based on the current Python version, but based on the versions of the Python written in the pyproject.toml.
Poetry used to install the project just before the test is different one and is the one that is in the docker container.
There was a problem hiding this comment.
I don't think it's a justification for doing it wrong.
There was a problem hiding this comment.
Not sure I understand your point. My point is, that the version here is not relevant, so in that case any attempt to enforce specific version is kind of misleading, but maybe I am wrong. Can you give me some example case where the current "version unspecific" solution causes an issue?
There was a problem hiding this comment.
Do I understand it right that poetry is only used for editing pyproject.toml and poetry.lock here? Because uv does not do it?
If that's the case, I'd just drop a comment above the installation step and call it a day.
There was a problem hiding this comment.
(Btw. Apart from no immediate benefit, another small additional reason why I am hesitant adding third party poetry github action is, that there is no "official" action like it is in case of uv. As far as I can see, poetry available actions are made by people unrelated to the poetry)
Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>
Well, I'm slightly inclined towards calling them e2e. |
| - name: Install poetry | ||
| run: pipx install poetry |
There was a problem hiding this comment.
Do I understand it right that poetry is only used for editing pyproject.toml and poetry.lock here? Because uv does not do it?
If that's the case, I'd just drop a comment above the installation step and call it a day.
| name: Templates Apify integration tests | ||
|
|
||
| on: | ||
| workflow_dispatch: |
There was a problem hiding this comment.
Shouldn't we run this nightly, in addition to this?
There was a problem hiding this comment.
Yes, but I would like to do it in two steps. First merge it with manual trigger and change it to automatic later when it is proven to work well in the repo.
Currently I am testing this workflow on forked repo and even though I had run it many times, it is still kind of a clumsy test setup, so it feels safer to merge it first with only manual trigger.
| crawlee_extras = re.findall(r'crawlee(\[.*\])', pyproject)[0] or '' | ||
|
|
||
| # Inject crawlee wheel file to the docker image and update project to depend on it.""" | ||
| with open(project_path / 'Dockerfile') as f: |
There was a problem hiding this comment.
Patching the lockfile and the Dockerfile seems pretty involved. Can't we replace apify push with make_actor from the SDK integration tests? Then you could just put the wheel inside the build context and pip-install it in the Dockerfile.
There was a problem hiding this comment.
The point of this somewhat complicated way of patching is to ensure it tests both the template package managers and the way they install the project and that it is as similar to user workflow as possible.
By using the make actor similar approach I think I would do slightly different installation steps, but more importantly I would have to make sure that make actor is maintained together with any changes to apify-cli.
Imagine situation where there is some braking change to apify push. If I bypass the command by using make actor approach, then this test will be still passing, while out in the wilds it will start crashing for the people.
There was a problem hiding this comment.
Huh, testing apify push sounds way out of scope even for end-to-end tests. Also, non-backward-compatible changes to the API for building stuff are super unlikely. We already have a make_actor implementation in the SDK, so it would make a lot of sense to me to use the same approach here.
There was a problem hiding this comment.
make_actor is in different repo and is designed for code that is in the string format as the actors are generated on the fly by the test code. Here the actors are generated the way user generates them as a folder structure with files.
I would suggest to start with this approach and iteratively improve it later based on any potential issues we encounter. The tests are written in a way, that this discussed "patch function" is well isolated from the rest of the test, so we can just change the patch function and the rest of the test remains the same.
…#1136) ### Description - Add e2e tests for playwright templates - Add e2e tests for pip-based templates - Fix template issue with pip template that has additional dependencies - Separate all e2e tests in CI to run in it's own runner to allow selective re-triggers (e2e tests work on real network and it will always have some flakiness due to that) - Reduce `max_requests_per_crawl` to 10 in all templates to be aligned with doc examples and to make the tests faster. ### Issues - Closes: #1109

Description
Add
crawleetemplates integration tests withApifyplatform. (onlyParselandBeautifulsouptemplates)Add workflow for manual triggering of integration tests (Will become automatic once it is running for a while to prove stability.)
Issues
Testing
Tested on forked repo:
https://github.com/Pijukatel/crawlee-python/actions/runs/13969844251
uvrelated part of the dockerfile):https://github.com/Pijukatel/crawlee-python/actions/runs/13970390933
parselactor template):https://github.com/Pijukatel/crawlee-python/actions/runs/13970267119
Next steps
Add tests for
Playwrightbased templates.Fix current problems with
Playwrighttemplate that has failing run in default settings.Making the integration tests each run in separate job. Each test can take long time and can be unstable due to the network issues. This brings the necessity to have the possibility to re-trigger individual tests and not the whole test suite.
Since Playwright based crawlers are slower, this will be introduced together with their testing.

Example such run, where only one template test can be re-triggered: