You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit fbab2a8
Browse filesBrowse the repository at this point in the historyBrowse files
fix!: do not count downtime towards crawler_runtime after migration or resurrection
The `crawler_runtime` statistic no longer counts the downtime between
the runs of a resumed crawler, and it accumulates correctly across any
number of persist/resume cycles:
- The first periodic statistics log reports exactly the runtime
accumulated so far: zero for a fresh start, the previously persisted
total for a resumed run.
- The accumulated runtime is stored in the new `accumulated_runtime`
field, persisted under `crawlerRuntimeMillis`. Every persist writes
the live total, so a state persisted mid-run (migration, abort)
carries the runtime up to that moment and a resumed run continues
from the last checkpoint.
- The timestamps of the previous run are cleared before the new run
starts being measured, so they cannot leak the downtime into the
runtime, and a backwards clock jump during a run cannot push the
runtime below the accumulated total.
BREAKING CHANGE: `StatisticsState.crawler_runtime_for_serialization`
is removed; the `crawler_runtime` property is backed by the new
`accumulated_runtime` field. `crawlerRuntimeMillis` is serialized as
a number of milliseconds instead of an ISO 8601 duration string.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VMHwDx87jV4rnu5LkbpNxy
Copy file name to clipboardExpand all lines: docs/upgrading/upgrading_to_v2.md
+4Lines changed: 4 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -17,6 +17,10 @@ We drop support for Python 3.10, which reached its end of life in October 2026.
17
17
18
18
`StatisticsState.crawler_runtime` is computed from the crawler's start and finish timestamps. In v1, assigning to it only emitted a `DeprecationWarning` and the value was ignored. In v2, the property has no setter, so an assignment raises `AttributeError`. Remove any such assignments.
19
19
20
+
## `StatisticsState.crawler_runtime_for_serialization` is removed
21
+
22
+
`StatisticsState.crawler_runtime_for_serialization` is removed, and the persisted `crawlerRuntimeMillis` value is a number of milliseconds instead of an ISO 8601 duration string. Read the runtime from the `crawler_runtime` property instead.
23
+
20
24
## `RequestQueueClient.is_finished` is abstract
21
25
22
26
`RequestQueueClient.is_finished` is now an abstract method. In v1, it fell back to `is_empty`, which ignores requests still being processed and could report the queue as finished too early. A subclass that doesn't implement it raises `TypeError` on instantiation. If you have a custom request queue client, implement `is_finished` so that it returns `True` only when the queue is empty and no requests are in progress. The built-in clients already implement it.
0 commit comments