iframe-proxy

Pijukatel · 2024-12-06T09:02:04Z

Description

Show how to integrate Camoufox into PlaywrightCrawler.
Added one docs page and one example code.
Added playwright-camoufox template to cli call crawlee create.
Fixed wrong template indentation when Apify integration was selected in cli.

Issues

Closes: Integrate Camoufox into PlaywrightCrawler #684

Fix inccorect indent in existing project template.

Pijukatel · 2024-12-10T09:37:41Z

janbuchar

Nice job! I'm not sure about the docs prefix though - we also fix a bug in templates and add a new template to boot, so how about 'featorfix`?

janbuchar · 2024-12-11T13:05:40Z

    "project_name": "crawlee-python-project",
    "__package_name": "{{ cookiecutter.project_name|lower|replace('-', '_') }}",
-    "crawler_type": ["beautifulsoup", "parsel", "playwright"],
+    "crawler_type": ["beautifulsoup", "parsel", "playwright", "camoufox"],


I'd prefer to call this playwright-camoufox for example.

janbuchar · 2024-12-11T13:06:52Z

 # % endif
 # % endblock
 # % endfilter
+{% if self.pre_main is defined %}


Interesting trick! Could you use line expressions (# % if ...) here like in the rest of the file?

janbuchar · 2024-12-11T13:08:48Z

+# % endblock
+
+# % block pre_main
+class ExampleCamoufoxPlugin(PlaywrightBrowserPlugin):


Let's leave out the "Example" prefix 🙂

janbuchar · 2024-12-11T13:10:17Z

+import asyncio
+
+#  This external package needs to be installed. It is not included in crawlee.
+from camoufox import AsyncNewBrowser  # type: ignore[import-not-found]


Could we ignore that check on the project level so that the example is cleaner?

janbuchar · 2024-12-11T13:15:04Z

+    @crawler.router.default_handler
+    async def request_handler(context: PlaywrightCrawlingContext) -> None:
+        context.log.info(f'Processing {context.request.url} ...')
+
+        # Extract data from the page using Playwright's API.
+        posts = await context.page.query_selector_all('.athing')
+        data = []
+
+        for post in posts:
+            # Get the HTML elements for the title and rank within each post.
+            title_element = await post.query_selector('.title a')
+            rank_element = await post.query_selector('.rank')
+
+            # Extract the data we want from the elements.
+            title = await title_element.inner_text() if title_element else None
+            rank = await rank_element.inner_text() if rank_element else None
+            href = await title_element.get_attribute('href') if title_element else None
+
+            data.append({'title': title, 'rank': rank, 'href': href})
+
+        # Push the extracted data to the default dataset. In local configuration,
+        # the data will be stored as JSON files in ./storage/datasets/default.
+        await context.push_data(data)
+
+        # Find a link to the next page and enqueue it if it exists.
+        await context.enqueue_links(selector='.morelink')
+
+    # Define a hook that will be called each time before navigating to a new URL.
+    # The hook receives a context parameter, providing access to the request and
+    # browser page among other things. In this example, we log the URL being
+    # navigated to.
+    @crawler.pre_navigation_hook
+    async def log_navigation_url(context: PlaywrightPreNavigationContext) -> None:
+        context.log.info(f'Navigating to {context.request.url} ...')
+
+    # Run the crawler with the initial list of URLs.
+    await crawler.run(['https://news.ycombinator.com/'])


There is a lot of stuff going on that could distract the reader from the actual gist of the example - could we pare it down a bit?

I reduced most of the example and kept only minimal example that still demonstrates the use of the crawler.

janbuchar · 2024-12-11T13:15:42Z

Can you mention the new template here as well?

Add example of how to integrate Camoufox into PlaywrightCrawler.

61462b3

Pijukatel added documentation Improvements or additions to documentation. t-tooling Issues with this label are in the ownership of the tooling team. labels Dec 6, 2024

github-actions Bot assigned Pijukatel Dec 6, 2024

github-actions Bot added this to the 104th sprint - Tooling team milestone Dec 6, 2024

Ignore mypy error on missing import.

81a8f49

Pijukatel changed the title ~~Add example of how to integrate Camoufox into PlaywrightCrawler.~~ docs: Add example of how to integrate Camoufox into PlaywrightCrawler. Dec 6, 2024

Pijukatel added 2 commits December 6, 2024 10:15

Mypy

ba456fa

Mypy

149031d

Pijukatel mentioned this pull request Dec 6, 2024

Integrate Camoufox into PlaywrightCrawler #684

Closed

vdusek changed the title ~~docs: Add example of how to integrate Camoufox into PlaywrightCrawler.~~ docs: Add example of how to integrate Camoufox into PlaywrightCrawler Dec 6, 2024

Pijukatel added 3 commits December 6, 2024 19:00

Add project template for Camoufox

3d73e49

Fix inccorect indent in existing project template.

Remove unused condition in requirements

939e677

Merge remote-tracking branch 'origin/master' into camoufox-example-code

a24f67d

Pijukatel mentioned this pull request Dec 10, 2024

feat: Enable usage of Camoufox in PlaywrightCrawler #782

Closed

Pijukatel added 2 commits December 10, 2024 08:17

Update Dockerfile with comment

d0f2892

Update _pyproject.toml

a67b55b

Pijukatel marked this pull request as ready for review December 10, 2024 09:38

Pijukatel requested review from janbuchar and vdusek December 10, 2024 09:38

janbuchar requested changes Dec 11, 2024

View reviewed changes

Review comments

fca3169

Pijukatel changed the title ~~docs: Add example of how to integrate Camoufox into PlaywrightCrawler~~ feat: Add example of how to integrate Camoufox into PlaywrightCrawler Dec 12, 2024

Pijukatel added the enhancement New feature or request. label Dec 12, 2024

Pijukatel requested a review from janbuchar December 12, 2024 08:15

janbuchar approved these changes Dec 12, 2024

View reviewed changes

Pijukatel merged commit 246cfc4 into master Dec 12, 2024

Pijukatel deleted the camoufox-example-code branch December 12, 2024 09:37

Sunbelt Computer Software

PL/B Language Development and Support

Conversation

Pijukatel commented Dec 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Issues

Uh oh!

Pijukatel commented Dec 10, 2024

Uh oh!

janbuchar left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Pijukatel commented Dec 6, 2024 •

edited

Loading