Class●Since v0.7
AsyncPromptCache
AsyncPromptCache(
self,
*,
max_size: int = DEFAULT_PROMPT_CACHE_MAX_SIZE,
ttl_seconds: Optional[floatBases
_BasePromptCacheConstructors
Methods
]
=
,
refresh_interval_seconds
:
=
)
Parameters
constructor
__init__
method
stop
Stop async background refresh loop.
Cancels the refresh task and waits for it to complete.
Thread-safe LRU cache with asyncio task refresh.
For use with the asynchronous AsyncClient.
Features:
- In-memory LRU cache with configurable max size
- Asyncio task for refreshing stale entries
- Stale-while-revalidate: returns stale data while refresh happens
- Thread-safe for concurrent access
Example:
async def fetch_prompt(key: str) -> PromptCommit: ... return await client._afetch_prompt_from_api(key) cache = AsyncPromptCache( ... max_size=100, ... ttl_seconds=3600, ... fetch_func=fetch_prompt, ... ) await cache.start() cache.set("my-prompt:latest", prompt_commit) cached = cache.get("my-prompt:latest") await cache.stop()
How often to check for stale entries.