📁 nefilim: file system as used by traverse

• 1. Introduction
• 2. 🎀 Features
• 3. 🔰 Quick Start
• 4. 🎭 Relative vs Absolute
• 5. 📚 Usage
  • 5.1. 📂 File Systems
    • 5.1.1. ✨ Universal FS
    • 5.1.2. ✨ Traverse FS
    • 5.1.3. ✨ Exists In FS
    • 5.1.4. ✨ Read File FS
    • 5.1.5. ✨ Reader FS
    • 5.1.6. ✨ Make Dir FS
    • 5.1.7. ✨ Move FS
    • 5.1.8. ✨ Change FS
    • 5.1.9. ✨ Copy FS
    • 5.1.10. ✨ Remove FS
    • 5.1.11. ✨ Rename FS
    • 5.1.12. ✨ Write File FS
    • 5.1.13. ✨ Writer FS
• 6. Overwrite Flag
• 7. 💔 Errors
  • 7.1. ⛔ Binary Fs Op Error
  • 7.2. ⛔ Invalid Path Error
  • 7.3. ⛔ Reject Same Directory Move Error
  • 7.4. ⛔ Reject Different Directory Change Error
• 8. Utilities
  • 8.1. 🛡️ EnsureAtPath
  • 8.2. 🛡️ResolvePath
• 9. 💥 Trouble Shooting

1. Introduction

Nefilim is a file system abstraction used internally with snivilised packages for file system operations. In particular, it is the file system used by the directory walker as implemented by the traverse package.

An important note has to be acknowledged about usage of the file systems defined here and their comparison to the ones as defined in the Go standard library.

There are 2 ways of interacting with the file system in Go. The primary way that seems most intuitive would be to use those functions as defined within the os package, eg, we can open a file using os.Open:

os.Open("~/foo.txt")

... or we can use the os.DirFS. However, this method uses a different concept. We can't directly open a file. We first need to create a new file system and to do so, to access the local file system, we would use os.DirFS first:

localFS := os.DirFS("/foo")

where /foo represents an absolute path we have access to. The result is a file system instance as represented by the fs.FS interface. We can now open a file via this instance, but the crucial difference now is that we can now only use relative paths, where the path we specify is relative to the rooted path specified when we created the file system earlier:

localFS.Open("foo.txt")

The file system defined in Nefilim, provides access to the file system in the latter case (ie via a relative file system), but not yet the former, absolute file access (there are plans to create another abstraction that enables this more traditional way of accessing the file system, as denoted by the first example above).

Another rationale for this repo was to fill the gap left by the standard library, in that there are no writer file system interfaces, so they are defined here, primarily for the purposes of snivilised projects, but also for the benefit of third parties. Contained within is an abstraction that defines a file system as required by traverse, but this particular instance only requires a subset of the full set of operations one would expect of a file system, but there is also a Universal File System which will contain the full set of operations, such as Copy, which is currently not required by traverse.

There are also a few minor adjustments and additions that should be noted, such as:

• a slightly different name for creating new directories, Mkdir as defined in the standard packages is replaced by a more user friendly MakeDir. (This is just a minor issue, but having to remember wether the 'd' in Mkdir was capitalised or not, is just friction that I would rather do without.)

• a new Move operation, which is similar to Rename but is defined to separate out the move semantics from rename; ie, Move will only move an item to a different directory. If a same directory move is detected, then this will be rejected with an appropriate error and the client is guided to use Rename instead.

• a new Change operation is defined, that is like Move, but is stricter in that it enforces the use of a name as the to parameter denoting the destination to be in the same directory; ie, it is prohibited to specify another relative directory as the Change operation assumes the destination should reside in the same directory as the source.

The semantics of Rename remains unchanged, so clients can expect consistent behaviour when compared to the standard package.

Other than these changes, the functionality in Nefilim aims to mirror the standard package as much as possible.

2. 🎀 Features

• unit testing with Ginkgo/Gomega
• linting configuration and pre-commit hooks, (see: linting-golang).
• uses 💥 lo

3. 🔰 Quick Start

• To create a universal file system that contains all reader and writer functions:

  import (
    nef "github.com/snivilised/nefilim"
  )

  fS := nef.NewUniversalFS(nef.At{
    Root:      "/Users/marina/dev",
    Overwrite: false,
  })

... creates a file system whose root is /Users/marina/dev

Any operation invoked, should now be done with a path that is relative to this root, eg to open a file:

  if file, err := fS.Open("foo.txt"); err!= nil {
    ...
  }

... will succeed if the file exists at /Users/marina/dev/foo.txt

When creating a file system with writer capabilities, the Overwrite flag can be set within the At struct, which will activate overwrite semantics, that are explained later for each writer operation.

4. 🎭 Relative vs Absolute

There are various file system constructor functions in the form NewXxxFS. Currently, these are all of the relative variety, whereby the client is required to invoke operations with paths that are relative to the root. Nefilim conforms to the semantics of io/fs, so any paths that are not conformant with fs.ValidPath will be rejected.

The key rules for paths to confirm to are:

• must be unrooted
• must not start or end with a '/'
• must not contain '.' or '..' or the empty string, expect for the special case '.' which refers to root
• paths are forward '/' separated only, for all platforms
• characters such as backslash and colon are still valid, but should not be interpreted as path separators

5. 📚 Usage

5.1. 📂 File Systems

_Nefilim comes with predefined interfaces with different capabilities. The interfaces are as narrow as possible, most are single method interfaces. Some interfaces, contain more than 1 closely related methods. Clearly, Nefilim can't provide interfaces for all combination of methods, but the client is free to compose custom ones by combining those defined here by Nefilim.

5.1.1. ✨ Universal FS

Capable of all read and write operations and can be used with traverse if so required. Actually, as previously indicated, traverse doesn't need a UniversalFS, it only requires a TraverseFS, so why would we use a UniversalFS with traverse? Well, within the callback of traverse navigation, we may need to invoke operations, not defined on TraverseFS. But beware, do not invoke operations that would interfere with the currently running navigation session, without making required mitigating actions.

• interface: UniversalFS
• Create: NewUniversalFS

  fS := nef.NewUniversalFS(nef.At{
    Root:      "/Users/marina/dev",
    Overwrite: false,
  })

• Composed of: ReaderFS, WriterFS

---

5.1.2. ✨ Traverse FS

A specialised file system as required for a traverse navigation.

• interface: TraverseFS
• Create: NewTraverseFS

  fS := nef.NewTraverseFS(nef.At{
    Root:      "/Users/marina/dev",
    Overwrite: false,
  })

  result, err := tv.Walk().Configure().Extent(tv.Prime(
    &tv.Using{
      Tree:         "some-path-relative",
      Subscription: enums.SubscribeUniversal,
      Handler: func(node *core.Node) error {
        GinkgoWriter.Printf(
          "---> 🍯 EXAMPLE-REGEX-FILTER-CALLBACK: '%v'\n", node.Path,
        )
        return nil
      },
      GetTraverseFS: func(_ string) tv.TraverseFS {
        return fS
      },
    },
  )).Navigate(ctx)

In the above example, we create a universal file system rooted at /Users/marina/dev. The Tree path set in the tv.Using struct is relative to this root.

• Composed of: MakeDirFS, ReaderFS, WriteFileFS

---

5.1.3. ✨ Exists In FS

A file system that can determine the existence of a path and indicate if its a file or directory.

• interface: ExistsInFS
• Create: NewExistsInFS

  fS := nef.NewExistsInFS(nef.At{
    Root:      "/Users/marina/dev",
  })

• Commands: FileExists, DirectoryExists

💎 FileExists

fS.FileExists("bar/baz/foo.txt")

returns true if /Users/marina/dev/bar/baz/foo.txt exists as a file, false otherwise.

💎 DirectoryExists

fS.DirectoryExists("bar/baz")

returns true if /Users/marina/dev/bar/baz/ exists as a directory, false otherwise.

---

5.1.4. ✨ Read File FS

• interface: ReadFileFS

• Create: NewReadFileFS

  fS := nef.NewReadFileFS(nef.At{
    Root:      "/Users/marina/dev",
    Overwrite: false,
  })

• Composed of: fs.FS
• Commands:

💎 ReadFile

fS.ReadFile("bar/baz/foo.txt")

returns no error if /Users/marina/dev/bar/baz/foo.txt exists as a file, otherwise behaves as fs.ReadFile.

---

5.1.5. ✨ Reader FS

Creates a read only file system.

• interface: ReaderFS
• Create: NewReaderFS

  fS := nef.NewReaderFS(nef.At{
    Root:      "/Users/marina/dev",
  })

• Composed of: fs.StatFS, fs.ReadDirFS, ExistsInFS, ReadFileFS

---

5.1.6. ✨ Make Dir FS

• interface: MakeDirFS
• Create: NewMakeDirFS

  fS := nef.NewMakeDirFS(nef.At{
    Root:      "/Users/marina/dev",
    Overwrite: false,
  })

• Composed of: ExistsInFS
• Commands: MakeDir, MakeDirAll

💎 MakeDir

fS.MakeDir("bar/baz")

behaves as os.Mkdir.

💎 MakeDirAll

fS.MakeDir("bar/baz")

behaves as os.MkdirAll.

---

5.1.7. ✨ Move FS

Comes as part of the UniversalFS only. The Move command is a new operation, that does not exist in the standard library, created to isolate the move semantics of the os.Rename command. os.Rename implements both move and rename semantics combined.

Another problem with os.Rename occurs when moving a file eg:

when a file needs to be moved from bar/file.txt to the directory bar/baz/, invoking so.Rename the intuitive way would be to do as follows:

os.Rename("bar/file.txt", "bar/baz/")

but this will fail with a LinkError. The correct way to achieve the desired result is

os.Rename("bar/file.txt", "bar/baz/file.txt")

ie, the file name has to be replicated in the 'newpath' path.

The Move command challenges this requirement and allows the client to omit the file name from the second parameter and can be achieved as:

fS.Move("bar/file.txt", "bar/baz")

In these examples, we have talked about the new path representing a file, but the same holds true for a directory.

Also, Move really does mean move, the new name is always retained, not renamed, and the new path always represents a different directory from the source.

• interface: MoveFS
• Create: NewUniversalFS

  fS := nef.NewUniversalFS(nef.At{
    Root:      "/Users/marina/dev",
    Overwrite: false,
  })

• Commands: Move

fS.Move("bar/file.txt", "bar/baz")

As described previously, moves /Users/marina/dev/bar/file.txt to /Users/marina/dev/bar/baz/file.txt. However, the behaviour differs depending on the prior existence of bar/baz/file.txt and the value of the overwrite flag.

If the file already exists at the destination and overwrite is true, then the existing file is overwritten, otherwise, an invalid file system operation NewInvalidBinaryFsOpError is returned. This denotes the from and to path and the name of the operation attempted, in this case Move.

5.1.8. ✨ Change FS

Comes as part of the UniversalFS only (implementation pending as of v0.1.2). The Change command is a new operation, that does not exist in the standard library, created to isolate the rename semantics of the os.Rename command.

The Change command imposes a further restriction to os.Rename. In the same way that the Move command rejects setting a destination path that denotes the same directory as the source, Change will reject any attempt to move the item to a different directory. It is purely meant to rename the item in the same location; so Change is Rename but prevents move semantics.

• interface: ChangeFS
• Create: NewUniversalFS

  fS := nef.NewUniversalFS(nef.At{
    Root:      "/Users/marina/dev",
    Overwrite: false,
  })

• Commands: Change

os.Change("bar/from-file.txt", "bar/to-file.txt")

renames /Users/marina/dev/bar/from-file.txt to /Users/marina/dev/bar/to-file.txt. However, the behaviour differs depending on the prior existence of bar/to-file and the value of the overwrite flag.

If the file already exists at the destination and overwrite is true, then the existing file is overwritten, otherwise, an invalid file system operation NewInvalidBinaryFsOpError is returned. This denotes the from and to path and the name of the operation attempted, in this case Change.

---

5.1.9. ✨ Copy FS

... pending

• interface: CopyFS
• Create: tbd

5.1.10. ✨ Remove FS

A file system interface that can delete files and directories

Comes as part of UniversalFS only.

• interface: RemoveFS
• Commands: Remove, RemoveAll

💎 Remove

fS.Remove("bar/baz")

behaves as os.Remove

💎 RemoveAll

fS.RemoveAll("bar/baz")

behaves as os.RemoveAll

---

5.1.11. ✨ Rename FS

A file system interface that can rename/move files and directories

Comes as part of the UniversalFS only.

• interface: RenameFS
• Create: NewUniversalFS

  fS := nef.NewUniversalFS(nef.At{
    Root:      "/Users/marina/dev",
    Overwrite: false,
  })

• Commands: Rename

fs.Rename("from.txt", "bar/baz/to.txt")

behaves as os.Rename, will move file from /Users/marina/dev/from.txt to /Users/marina/dev/bar/baz/to.txt

fs.Rename("from.txt", "to.txt")

behaves as os.Rename, will move file from /Users/marina/dev/from.txt to /Users/marina/dev/to.txt

📍 Note: the overwrite flag is ignored as it is not required by os.Rename

---

5.1.12. ✨ Write File FS

• interface: WriteFileFS
• Create: NewWriteFileFS

  fS := nef.NewWriteFileFS(nef.At{
    Root:      "/Users/marina/dev",
    Overwrite: false,
  })

• Commands: Create, WriteFile

💎 Create

fS.Create("bar/baz")

behaves as os.Create

💎 WriteFile

fS.WriteFile("bar/baz/file.txt")

behaves as os.WriteFile

---

5.1.13. ✨ Writer FS

• interface: WriterFS
• Create: NewWriterFS

  fS := nef.NewNewWriterFS(nef.At{
    Root:      "/Users/marina/dev",
    Overwrite: false,
  })

• Composed of: CopyFS, ExistsInFS, MakeDirFS, RemoveFS, RenameFS, WriteFileFS

---

6. Overwrite Flag

The reader may have observed the presence of the overwrite flag at the construction site, being passed into the NewXxxFS functions and may have wondered why the flag is not passed into the command. This would be a valid observation, but it has been done this way in order to conform to the apis in the standard library. The overwrite flag is purely of the making of Nefilim and the only way to express it, is to pass it in at the time of creating the file system. This means that the client has to make an upfront decision as to what overwrite semantics are required, which is less than desirable, but necessary to avoid incompatibility with the standard packages.

7. 💔 Errors

As nefilim strives to conform to the standard library, commands contained within return the same errors as so defined, eg os.LinkError, os.ErrExist and os.ErrNotExist to name but a few.

However, the custom commands, namely, Move and Change and to some extent, Rename can return new Nefilim defined errors, as described in the following sections.

The errors use idiomatic Go techniques for adding context to source errors by wrapping and also provided are convenience methods for identifying errors that typically invoke errors.Is/As on the client's behalf.

7.1. ⛔ Binary Fs Op Error

IsBinaryFsOpError identifies an error that occurs as a result of a failed invoke of a command that take 2 parameters, typically from and to locations, representing either files or directories. The error also denotes the name of the command to which it relates.

7.2. ⛔ Invalid Path Error

IsInvalidPathError identifies an error that occurs whenever a path fails validation using fs.ValidPath.

7.3. ⛔ Reject Same Directory Move Error

IsRejectSameDirMoveError identifies an error that occurs as a result of a Move attempt to move an item to the same directory.

7.4. ⛔ Reject Different Directory Change Error

IsRejectDifferentDirChangeError an error that occurs as a result of a Change attempt to move an item to a different directory.

(not yet available)

8. Utilities

8.1. 🛡️ EnsureAtPath

EnsurePathAt ensures that the specified path exists (including any non existing intermediate directories). Given a path and a default filename, the specified path is created in the following manner:

• If the path denotes a file (path does not end is a directory separator), then the parent folder is created if it doesn't exist on the file-system provided.
• If the path denotes a directory, then that directory is created.

The returned string represents the file, so if the path specified was a directory path, then the defaultFilename provided is joined to the path and returned, otherwise the original path is returned un-modified.

Note: filepath.Join does not preserve a trailing separator, therefore to make sure a path is interpreted as a directory and not a file, then the separator has to be appended manually onto the end of the path. If vfs is not provided, then the path is ensured directly on the native file system.

[illustrative examples pending]

8.2. 🛡️ResolvePath

ResolvePath performs 2 forms of path resolution. The first is resolving a home path reference, via the ~ character; ~ is replaced by the user's home path. The second resolves ./ or ../ relative path. (The overrides do not need to be provided.)

[illustrative examples pending]

9. 💥 Trouble Shooting

tbd...