Description

AutoscaledPool now calls is_finished_function only when none of its own tasks are running and it can't start a new one, either because no task is ready or because a shared concurrency system has no free capacity. Before, it was called on every loop iteration, even with tasks in flight.

• Crawlers no longer query the request manager's is_finished() every half second during a crawl, so storage backends get fewer calls.
• A pool whose is_task_ready_function always resolves to True no longer finishes. It has to resolve to False once the work runs out. The upgrade guide covers this.
• A paused pool doesn't finish until it's resumed.
• crawler.stop() now applies only to the current run. A crawler that was stopped once used to end every later run() immediately.
• After stop() or reaching max_requests_per_crawl, the crawler logs once that it takes no new requests and again when the remaining requests have finished. The stop reason no longer ends with a double period.

Issues

• Follow-up: feat(autoscaling)!: Add ConcurrencySystem to share one concurrency budget between crawlers #2322