{{ message }}
Repository navigation
Replies: 2 comments 1 reply
|
I have example repo here |
0 replies
|
Hi @neviaumi and thanks for opening this discussion! The Crawlee storage system is currently undergoing a significant refactor - see #1194. With that, you should be able to easily set up a crawler in a way that each MCP "call" (forgive me for not knowing the correct terminology) has its own non-persistent storage (datasets, key-value stores and request queues). In fact, you can already configure the |
1 reply
Answer selected by
neviaumi
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment

Hi @neviaumi and thanks for opening this discussion! The Crawlee storage system is currently undergoing a significant refactor - see #1194. With that, you should be able to easily set up a crawler in a way that each MCP "call" (forgive me for not knowing the correct terminology) has its own non-persistent storage (datasets, key-value stores and request queues).
In fact, you can already configure the
MemoryStorageClientto not dump anything in the filesystem - see the highlighted code in https://crawlee.dev/python/docs/deployment/gcp-cloud-run-functions, for example.