Purpose of the change

Remove a hard throughput ceiling on Cohere reranking. CohereReranker.score currently runs the sync ClientV2.rerank call through asyncio.to_thread, which draws from asyncio's default executor: at most min(32, cpu_count + 4) worker threads. Each in-flight rerank pins one of those threads for its full network round trip (typically ~0.3-0.5 s), so rerank throughput is capped at roughly threads / latency RPS regardless of how much concurrency the caller offers:

• 8-core host: 12 threads -> ~24-40 rerank RPS
• even at the 32-thread max: ~65-105 RPS

Beyond the cap, calls queue inside the executor and observed latency grows linearly with load. The default executor is also shared with every other asyncio.to_thread call in the process (e.g. SentenceTransformerEmbedder, CrossEncoderReranker), so saturating it with reranks starves local model inference too.

This PR switches to the Cohere SDK's native async client (AsyncClientV2, httpx-based), making rerank plain event-loop I/O. Concurrency is then bounded by the HTTP connection pool and the API itself, not by a thread pool.

Description

• cohere_reranker.py: score now awaits client.rerank(...) directly; the asyncio.to_thread wrapper is gone. Score mapping and empty/blank input handling are unchanged.
• reranker_manager.py: _build_cohere_reranker constructs AsyncClientV2 (same api_key / base_url kwargs as ClientV2).
• Tests: conftest's cohere_client fixture and the reranker manager tests updated to AsyncClientV2; new keyless unit tests added (see below).

No config or API surface changes; AsyncClientV2 is generated from the same spec as ClientV2 and is already part of the pinned cohere>=5.20.0 dependency.

Measured effect (stub client with 0.2 s simulated latency): 128 concurrent score() calls complete in 0.20 s wall on this branch, vs ~1.7 s when serialized through a 15-thread default executor. The gap widens with load.

Breaking for usage as a library.

Type of change

• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Refactor (does not change functionality, e.g., code style improvements, linting) - performance only; behavior and outputs are unchanged

How Has This Been Tested?

• Unit Test
• Manual verification (list step-by-step instructions)

Unit tests (no API key needed): new test_cohere_reranker_unit.py runs CohereReranker against a stub async client: relevance scores map back to original candidate positions, empty/blank candidates skip the API, blank queries are replaced, request parameters pass through verbatim, client errors are wrapped in ExternalServiceAPIError, and a 64-way concurrency canary fails by timeout if calls ever get serialized through a bounded worker pool. Full reranker + reranker manager suite: 186 passed, ruff clean.

Wire equivalence (no API key needed): pointed both ClientV2 (current production path) and AsyncClientV2 at a local capture HTTP server via base_url and issued the same rerank call. Results: byte-identical requests (POST /v2/rerank, identical JSON body, no header differences including authorization), both parse into equal V2RerankResponse objects, and CohereReranker end-to-end over real HTTP returns correctly mapped scores. Since the async client emits exactly the bytes the sync client already sends in production, server-side behavior is unchanged by this PR.

Integration: the existing test_cohere_reranker.py (integration-marked) is updated to the async client with identical coverage; it needs COHERE_API_KEY to run. Note the pytest-integration workflow does not currently set that secret, so these tests also skip in CI today - the keyless wire-equivalence check above was run in their place.

Test Results:

186 passed, 3 skipped, 54 deselected in 0.32s

wire equivalence probe:
sync:  POST /v2/rerank
async: POST /v2/rerank
body identical: True
header diffs: none
auth header identical: True
parsed responses identical: True
reranker scores (expect [0.1, 0.9, 0.5]): [0.1, 0.9, 0.5]

Checklist

• I have signed the commit(s) within this pull request
• My code follows the style guidelines of this project (See STYLE_GUIDE.md)
• I have performed a self-review of my own code
• My changes generate no new warnings
• I have added unit tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

Screenshots/Gifs

N/A

Further comments

• If very high rerank concurrency exposes connection churn (httpx defaults: 20 keepalive / 100 max connections), the next lever is passing a custom httpx_client to AsyncClientV2; left at defaults here.
• AmazonBedrockReranker has the same to_thread ceiling (boto3 has no async client); out of scope for this PR.

🤖 Generated with Claude Code