Search code examples
pythonfsspecpyfilesystem

Does fsspec support virtual filesystems such as pyfileysystem

One of pyfilesystem's main feature is virtual filesystems. E.g.

home_fs = open_fs('~/')
projects_fs = home_fs.opendir('/projects')

I think that is a great feature and was hoping that fsspec has something similar. But I couldn't find an example and I'm not able to get it working.

Solution

  • You might want DirFileSystem, invoked like

    fs = fsspec.implementations.dirfs.DirFileSystem(
    "<root path>", fs=fsspec.filesystem("file")
)

    You can apply this to any filesystem, not only local. root_path needs to be a string that, when you affix further path parts to it, makes a complete path the target filesystem can understand; it may include the protocol (e.g., for HTTP, it must do). In your case, it would be "~" (or the expanded version of this, which would be more explicit).

    Alternatively, you can create an arbitrarily mapped virtual filesystem with ReferenceFileSystem.

    mapping = {"/key1": ["/local/path/file1"],
           "/key2": ["/other/unrelated/path/file"]}
fs = fsspec.filesystem("reference", fo=mapping)

    Here, fs.cat("/key1") would get the contents of "/local/path/file1". You can have those paths be remote, or a mix of different backends, and even byte ranges of target files.