- <ApiLink to="class/LocalEventManager">`LocalEventManager`</ApiLink> extends the base event manager for local environments by automatically emitting `SYSTEM_INFO` events at regular intervals. This provides real-time system metrics including CPU usage and memory consumption, which are essential for internal components like the <ApiLink to="class/Snapshotter">`Snapshotter`</ApiLink> and <ApiLink to="class/ConcurrencySystem">`ConcurrencySystem`</ApiLink>.

You can learn more about <ApiLink to="class/Snapshotter">`Snapshotter`</ApiLink> and <ApiLink to="class/ConcurrencySystem">`ConcurrencySystem`</ApiLink> and their configuration in the [Scaling crawlers guide](./scaling-crawlers).

import asyncio

from crawlee import ConcurrencySettings

from crawlee.autoscaling import ConcurrencySystem

from crawlee.crawlers import (

BeautifulSoupCrawler,

BeautifulSoupCrawlingContext,

ParselCrawler,

ParselCrawlingContext,

)

from crawlee.storages import RequestQueue

async def main() -> None:

concurrency_settings = ConcurrencySettings(max_concurrency=10)

# Each crawler needs a request queue of its own, otherwise they would both

# consume the default one and process each other's requests.

first_queue = await RequestQueue.open(alias='first-crawler')

second_queue = await RequestQueue.open(alias='second-crawler')

# One budget for both crawlers, instead of one each.

async with ConcurrencySystem(concurrency_settings) as concurrency_system:

first_crawler = BeautifulSoupCrawler(

concurrency_system=concurrency_system,

request_manager=first_queue,

)

second_crawler = ParselCrawler(

concurrency_system=concurrency_system,

request_manager=second_queue,

)

@first_crawler.router.default_handler

async def first_handler(context: BeautifulSoupCrawlingContext) -> None:

context.log.info(f'Processing {context.request.url}...')

@second_crawler.router.default_handler

async def second_handler(context: ParselCrawlingContext) -> None:

context.log.info(f'Processing {context.request.url}...')

await asyncio.gather(

first_crawler.run(['https://crawlee.dev/']),

second_crawler.run(['https://crawlee.dev/python/']),

)

if __name__ == '__main__':

asyncio.run(main())

import SharedConcurrencySystemExample from '!!raw-loader!roa-loader!./code_examples/scaling_crawlers/shared_concurrency_system_example.py';

The `max_tasks_per_minute` setting in <ApiLink to="class/ConcurrencySettings">`ConcurrencySettings`</ApiLink> controls how many total tasks the crawler can process per minute. The tasks started over the last 60 seconds are counted, so the crawler may spend the whole limit early in the minute, as fast as its concurrency allows. It then waits until the oldest of those starts ages out of the window. By default, this is set to `Infinity`, meaning the crawler can run at full speed, limited only by `max_concurrency`. Use this option if you want to throttle your crawler to avoid overwhelming the target website with continuous requests.

Every crawler runs its requests through an <ApiLink to="class/AutoscaledPool">`AutoscaledPool`</ApiLink>, and the pool asks a <ApiLink to="class/ConcurrencySystem">`ConcurrencySystem`</ApiLink> whether there is room for one more request. The system is what keeps `min_concurrency` requests running even while the machine is overloaded, starts further ones only when there is enough free CPU and memory, and scales the budget up and down as the load changes. To monitor system resources, it leverages the <ApiLink to="class/Snapshotter">`Snapshotter`</ApiLink> and <ApiLink to="class/SystemStatus">`SystemStatus`</ApiLink> classes. If any task raises an exception, the error is propagated and the pool is stopped.

The crawler builds its own system from the `ConcurrencySettings` described above. Build one yourself and pass it as `concurrency_system` to go beyond those settings, or to share a single budget between several crawlers that would otherwise each scale against the whole machine and oversubscribe it. The budget goes to whoever asks for it first, so a crawler that joins a saturated system late may have to wait. Browser crawlers default to a lower `desired_concurrency` than the rest, and a supplied system replaces that default too, so set it in the `ConcurrencySettings` you build the system from. Holding the instance also lets you retune `min_concurrency` and `max_concurrency` while the crawl runs. The lifecycle of such a system belongs to you, so keep it open until every crawler using it has finished.

<RunnableCodeBlock className="language-python" language="python">

{SharedConcurrencySystemExample}

</RunnableCodeBlock>

Both Scrapy and Crawlee have settings to influence the scraping throughput, but they differ in their approach. Crawlee reads <ApiLink to="class/ConcurrencySettings">`ConcurrencySettings`</ApiLink> and scales concurrency automatically based on system resources, as described in the [Scaling crawlers guide](./scaling-crawlers). Use <ApiLink to="class/ConcurrencySettings#__init__">`max_tasks_per_minute`</ApiLink> to cap the request rate. It's close to `DOWNLOAD_DELAY`, but it limits the whole pool rather than adding a fixed delay per domain. Requests aren't spaced out either. They start as fast as the concurrency allows until the cap for the last 60 seconds is reached, and then wait for that window to move on.

Load monitoring and concurrency scaling moved from `AutoscaledPool` to the new `ConcurrencySystem`. A crawler builds a default system from its `concurrency_settings` for each run, so a crawler configured through `concurrency_settings` works as before. To cap the combined concurrency of several crawlers, pass one shared `ConcurrencySystem` to all of them through the new `concurrency_system` option.

The autoscaling classes now live in the public `crawlee.autoscaling` package, which replaces the private `crawlee._autoscaling`. Import `AutoscaledPool`, `ConcurrencySystem`, `ConcurrencyConsumer`, `SystemInfo` and `LoadRatioInfo` from there. `Snapshotter` and `SystemStatus` stay internal.

`AutoscaledPool` now only runs the task loop. If you create a pool yourself, note that:

- `system_status` and `concurrency_settings` are replaced by `concurrency_system`, which has to be active when the pool runs, and `consumer`, which identifies the pool to the system.

- `run_task_function` receives the id the task is booked under.

- `current_concurrency` comes from the concurrency system, so a shared system may count the tasks of other pools too. The pool's own tasks are counted by the new `own_concurrency`.

For example:

from crawlee._autoscaling import AutoscaledPool, Snapshotter, SystemStatus

async with Snapshotter.from_config() as snapshotter:

pool = AutoscaledPool(

system_status=SystemStatus(snapshotter),

concurrency_settings=ConcurrencySettings(max_concurrency=10),

run_task_function=run_task,

is_task_ready_function=is_task_ready,

is_finished_function=is_finished,

)

await pool.run()

from crawlee.autoscaling import AutoscaledPool, ConcurrencyConsumer, ConcurrencySystem

settings = ConcurrencySettings(max_concurrency=10)

async with ConcurrencySystem(settings) as concurrency_system:

pool = AutoscaledPool(

concurrency_system=concurrency_system,

consumer=ConcurrencyConsumer(id='my-pool'),

run_task_function=run_task, # Now called with the task id.

is_task_ready_function=is_task_ready,

is_finished_function=is_finished,

)

await pool.run()

`ConcurrencySettings.max_tasks_per_minute` now caps the tasks started over the last 60 seconds, the same as Crawlee JS. In v1, the pool spaced task starts evenly, `60 / max_tasks_per_minute` seconds apart. In v2, tasks start as fast as the concurrency allows until the cap is reached. The next ones wait until the oldest starts age out of the window. The rate over a minute stays the same, but tasks start in bursts. For details, see [Max tasks per minute](../guides/scaling-crawlers#max-tasks-per-minute).

The value has to be a whole number of at least 1, or infinity, which is the default. A fractional value, such as `0.5` or `7.5`, now raises `ValueError`.

The periodic log of the autoscaling state and the `Waiting for remaining tasks to finish` message are now logged at DEBUG instead of INFO. The crawler's status message at INFO still reports the desired concurrency.

The state log also has a new format. Instead of raw ratios for every resource, it names only the overloaded ones, with the share of time each spent over its threshold and its limit:

To read the concurrency values in code, use `crawler.concurrency_system`.

The autoscaling loggers moved together with the package. `crawlee._autoscaling.autoscaled_pool`, `crawlee._autoscaling.snapshotter` and `crawlee._autoscaling.system_status` are now `crawlee.autoscaling._autoscaled_pool`, `crawlee.autoscaling._snapshotter` and `crawlee.autoscaling._system_status`. The state log, along with the messages about scaling and free capacity, now comes from the new `crawlee.autoscaling._concurrency_system` logger. To configure all of them at once, use the `crawlee.autoscaling` logger.

"""Concurrency settings for a `ConcurrencySystem`."""

if max_tasks_per_minute != float('inf') and (

max_tasks_per_minute < 1 or not float(max_tasks_per_minute).is_integer()

):

raise ValueError('max_tasks_per_minute must be a whole number of at least 1, or infinity')

from ._autoscaled_pool import AutoscaledPool

from ._concurrency_system import ConcurrencySystem

from ._types import ConcurrencyConsumer, LoadRatioInfo, SystemInfo

__all__ = [

'AutoscaledPool',

'ConcurrencyConsumer',

'ConcurrencySystem',

'LoadRatioInfo',

'SystemInfo',

]