{{ message }}
Repository navigation
create multiple separate memory filesystems #1904
Description
fsspec/implementations/memory.py
class MemoryFileSystem(AbstractFileSystem):
"""A filesystem based on a dict of BytesIO objects
This is a global filesystem so instances of this class all point to the same
in memory filesystem.
"""
store: ClassVar[dict[str, Any]] = {} # global, do not overwrite!
pseudo_dirs = [""] # global, do not overwrite!... ok, but i need multiple different memory filesystems in parallel
assert (
fsspec.filesystem("memory://").store !=
fsspec.filesystem("memory://").store
)this should be as simple as moving store and pseudo_dirs to def __init__
maybe add a new protocol localmemory
Activity
martindurant commentedon Aug 7, 2025
This is a reasonable idea and something I thought of doing in the past. The reason it hasn't happened yet, is that MemoryFS is used extensively in tests - in fact, that is the main use case - and it if often useful not to have to carry variable handles around.
Note that:
would only be true if the filesystem is not cachable, otherwise you get the identical python object:
fsspec.filesystem("memory") is fsspec.filesystem("memory").martindurant commentedon Aug 7, 2025
In short: if you can make the change and no tests fail (or can be easily fixed), let's do it. Note that the documentation will need to be updated.