Comparing master...claude/pr-2204-regression-test-90rkc8 · apify/crawlee-python · GitHub
Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: apify/crawlee-python
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: master
Choose a base ref
...
head repository: apify/crawlee-python
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: claude/pr-2204-regression-test-90rkc8
Choose a head ref
Checking mergeability… Don’t worry, you can still create the pull request.
  • 4 commits
  • 3 files changed
  • 1 contributor

Commits on Aug 31, 2026

  1. fix: do not count downtime towards crawler_runtime after migration or…

    … resurrection
    
    When a crawler run was resumed from a persisted state, two issues inflated
    the reported `crawler_runtime`:
    
    - `Statistics.__aenter__` started the periodic logger before updating
      `crawler_last_started_at`, so the first statistics log of the resumed run
      computed the runtime from the previous run's start time and included all
      the downtime between the runs (later stats were computed correctly, which
      made the value appear to "jump back").
    - When the state came from a run that did not finish cleanly (migration,
      abort), `StatisticsState.model_post_init` computed the accumulated runtime
      offset using the current time, permanently baking the downtime between the
      runs into all subsequent statistics. The offset is now computed from the
      moment the state was last persisted instead.
    
    Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01VMHwDx87jV4rnu5LkbpNxy
    claude committed Aug 31, 2026
    Configuration menu
    Copy the full SHA
    b05a9ae View commit details
    Browse the repository at this point in the history

Commits on Sep 1, 2026

  1. fix: report only the accumulated runtime in the initial statistics log

    The first periodic statistics log must report exactly the runtime
    accumulated by previous runs: zero for a fresh start, the previous total
    for a resumed run. `Statistics.__aenter__` now resets the run timestamps
    before starting the periodic logger and sets the new start time after the
    initial log entry is printed.
    
    The accumulated runtime is now also restored from the persisted
    `crawlerRuntimeMillis` value instead of being reconstructed from the last
    run's timestamps, so runs before the most recent one are no longer lost
    from the total, and a state persisted mid-run keeps the correct runtime.
    The timestamp-based reconstruction is kept as a fallback for states
    persisted by older versions. `crawlerRuntimeMillis` is also serialized as
    milliseconds now, matching its name and the other `*Millis` fields,
    instead of an ISO 8601 duration string.
    
    Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01VMHwDx87jV4rnu5LkbpNxy
    claude committed Sep 1, 2026
    Configuration menu
    Copy the full SHA
    29591f2 View commit details
    Browse the repository at this point in the history
  2. fix: parse 'Z'-suffixed timestamps in the statistics persistence test…

    … on Python 3.10
    
    `datetime.fromisoformat` does not accept the 'Z' suffix until Python 3.11,
    which failed the resumed-run test on the 3.10 CI jobs.
    
    Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01VMHwDx87jV4rnu5LkbpNxy
    claude committed Sep 1, 2026
    Configuration menu
    Copy the full SHA
    5c119b1 View commit details
    Browse the repository at this point in the history

Commits on Sep 3, 2026

  1. test: prove runtime loss when resuming from a pre-1.1 statistics state

    PR #2204 restores the accumulated crawler runtime from the persisted
    'crawlerRuntimeMillis' value whenever the key is present, and falls back
    to timestamp-based reconstruction only when the key is absent. However,
    every crawlee release before v1.1.0 persisted 'crawlerRuntimeMillis' as
    a plain model field that nothing ever updated, so those states always
    contain the key with value 0 (verified against crawlee 1.0.2 and 0.6.0
    from PyPI). Resuming from such a state therefore restores a zero runtime
    offset and, because __aenter__ now also resets the run timestamps, the
    entire previous runtime is silently dropped instead of being
    reconstructed from the timestamps as before.
    
    The clean-shutdown test passes on master and fails on the PR head; the
    unclean-shutdown test documents that the stats_persisted_at fallback is
    unreachable for real persisted states of any released version.
    
    Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01BPQjAuUxxHHFv7EqAiSZrM
    claude committed Sep 3, 2026
    Configuration menu
    Copy the full SHA
    d2cfbf1 View commit details
    Browse the repository at this point in the history
Loading