refactor: RecoverableState abstraction by janbuchar · Pull Request #1172 · apify/crawlee-python · GitHub
Skip to content

refactor: RecoverableState abstraction - #1172

Merged
janbuchar merged 14 commits into
masterfrom
persistent-object-abstraction
Apr 30, 2025
Merged

janbuchar merged 14 commits into
masterfrom
persistent-object-abstraction

Conversation

@janbuchar

@janbuchar janbuchar commented Apr 23, 2025 •

Copy link
Copy Markdown
Collaborator

Key Changes

  • Introduced a new RecoverableState helper class that:

    • Uses Pydantic models for state representation and validation
    • Handles state persistence to a KeyValueStore
    • Automatically registers/deregisters with event system for state persistence
  • Refactored existing components to use the new abstraction:

    • Converted SessionPool to use RecoverableState for state management
    • Converted Statistics to use RecoverableState for state management
    • Converted KeyValueStore to use RecoverableState for autosaved values

Benefits

  • More consistent handling of state persistence across the library
  • Reduced code duplication for state management logic
  • Better separation of concerns between state data and persistence logic

@janbuchar janbuchar added the t-tooling Issues with this label are in the ownership of the tooling team. label Apr 23, 2025
@janbuchar
janbuchar requested review from Pijukatel and vdusek April 23, 2025 13:54
@github-actions github-actions Bot added this to the 113rd sprint - Tooling team milestone Apr 23, 2025
@github-actions github-actions Bot added the tested Temporary label used only programatically for some analytics. label Apr 23, 2025
@janbuchar
janbuchar force-pushed the persistent-object-abstraction branch from 1186ca2 to c1f920c Compare April 24, 2025 09:38
Comment thread src/crawlee/statistics/__init__.py
Comment thread tests/unit/sessions/test_models.py
@janbuchar
janbuchar marked this pull request as ready for review April 24, 2025 14:08
Comment thread tests/unit/crawlers/_playwright/test_playwright_crawler.py
Comment thread src/crawlee/_utils/persistent_object.py Outdated
@janbuchar

This comment has been minimized.

@janbuchar
janbuchar requested a review from Mantisus April 28, 2025 15:37
Comment thread src/crawlee/storages/_key_value_store.py
Comment thread src/crawlee/storages/_key_value_store.py Outdated

@vdusek vdusek left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! I like it, and its usage. Just a few minor notes & questions...

Comment thread src/crawlee/sessions/_models.py
Comment thread src/crawlee/sessions/_session_pool.py Outdated
Comment thread src/crawlee/_utils/persistent_object.py Outdated
Comment thread src/crawlee/_utils/persistent_object.py
Comment thread src/crawlee/_utils/persistent_object.py
Comment thread src/crawlee/storages/_creation_management.py
@janbuchar janbuchar changed the title refactor: PersistentObject abstraction refactor: RecoverableState abstraction Apr 30, 2025
@janbuchar
janbuchar requested review from Pijukatel and vdusek April 30, 2025 09:08

@vdusek vdusek left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@janbuchar
janbuchar merged commit de11d07 into master Apr 30, 2025
@janbuchar
janbuchar deleted the persistent-object-abstraction branch April 30, 2025 09:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Extract state persistence into a mixin-style base class

4 participants