Im using the kind of new io.FS abstraction to walk through a file system and read the first 128 bytes of each file that matches a file extension we have internally.
The files are in a local file system and archive files etc (ZIP and Tar I thunk).
I am using fs.WalkDir, passing in an fs.FS (os.DIR and fstest.MapFS in my tests). When walking I return an array of 'files' (really they are *.pzix and *.pzi files which are our propietary formats). I cannot find a suitable way of using the FS interface to get a few things about the file I am dealing with.
I want to:
I am finding the interfaces in Go a bit confusing coming from Java/C#. I was hoping to operate on the abstractions but I cannot figure out how to get other implementations of the file itself (eg. File interfaces have Stat() and read).
The simplest thing I've found is to store the path and the filename in the array and then when I iterate through the array, determine whether it was a os.Dir or fstest.MapFS but it seems rather counter intuitive:
func collectFiles(f fs.FS, root string) []string {
var files []string
fs.WalkDir(f, ".", func(p string, d fs.DirEntry, err error) error {
if !d.IsDir() { // we also check a few other things in the filename here
f = filepath.Abs(path.Join(root, p))
files = append(files, f)
}
}
return files
}
This gives me:
root = "m://" // mapfs
files = { "m://id-198271.pzi", "m://id-7125-092581.pzix"}
Is there a smarter way for me to deal with the abstractions and not do these things? Because after the array is returned, I have to 'open' the file, read the first 128 bytes (signature) and hash the rest of the file to make sure its 'valid'.
EDIT: To clarify, the collectFiles method is creating our main hit list of files to work on in another method. I was hoping to pass in local system files, zip files and tar files into that method so it can iterate through the files within the archives and add them to the array.
Was hoping there'd be like a File interface I can store in the array instead of a string so that subsequent callers can do f.open() without knowing what the underlying thing is.
Get the filename
p is the name within the file system.
Get the filesize
Get the size by calling fs.Stat(f, p)
Get the openfile method
Open the file with f.Open(p)
Example:
f := os.DirFS("/etc")
fs.WalkDir(f, ".", func(p string, d fs.DirEntry, err error) error {
if !d.IsDir() {
st, _ := fs.Stat(f, p)
r, _ := f.Open(p)
defer r.Close()
// Read prefix
var buf [md5.Size]byte
n, _ := io.ReadFull(r, buf[:])
// Hash remainder
h := md5.New()
_, _ = io.Copy(h, r)
s := h.Sum(nil)
fmt.Printf("%s %d %x %x\n", p, st.Size(), buf[:n], s)
}
return nil
})
For brevity, the example ignores errors. Don't do that in real code.