Collaborating with Forks

[Shadow53]

@lmburns, I notice you've been doing work in a fork of this project. I was wondering if you are planning on submitting any of those changes for upstream consideration (with relevant issues, as needed) or if you intend to maintain your own personal fork.

I'm opening this discussion here since I did not see a good way to contact you through the forked repo.

[lmburns]

Sorry about the delay in a response to your message. I need to setup GitHub emails differently so I see them every time.

But sure, I would be more than happy to collaborate on this project with you if that is what you wanted to do. I am just in the process of learning Rust and decided that since I use your program, I was going to fork it and add features to it while trying to learn Rust.

I'm not sure the direction that you plan on taking this project, and I see that you're a way better programmer than I am, so if you would like for me to try and work on something and then push the changes upstream for you to review, I'd be more than happy to.

XDG Folder Structure

For my fork, since I currently use macOS (am switching to Linux next week), and I use many many programs that all respect the XDG folder structure, I switched from simply using dirs::config_dir() and other functions for that crate, to using the XDG structure if set, and falling back to the manual $HOME/.config structure for macOS, and the dirs::... functions for all other operating systems.

File Formats

I also added compatibility for 3 file formats, toml, json, and yaml. I personally prefer yaml in the context of this program. It seems to be less repetitive, i.e., in a toml structure the format goes like so:

[hoards]
[hoards.file]
  [hoards.file.config]
    [hoards.file.config.encryption]

And a yaml structure goes in the following:

hoards:
  file:
    config:
      encryption

Just because, I added a way to convert between formats and still need to check whether your combinators works when using a yaml and json structure. I think that it does for yaml. The toml structure goes like:

[envs.itch]
    path_exists = [[
        "/home/shadow53/.itch",
        "/home/shadow53/.local/share/applications/io.itch.itch.desktop"
    ]]

yaml is:

envs:
  itch:
    path_exists:
      - - /home/shadow53/.itch
        - /home/shadow53/.local/share/applications/io.itch.itch.desktop

Added ability to preview configuration file (colorized) and add items through the CLI

This was modified from bat. This feature is not needed at all, but I had planned on creating some sort of way in which the user could add items to the configuration through the command line without ever having to open the configuration file. This would be an option to preview the configuration file. I would like this feature so that way whenever I add a new file that I want to backup, I could add it with ease. There could be a way that this creates an env for a specific executable and adds a path, or something along these lines. I haven't really attempted to work on the feature yet.

shellexpand

I modified the original shellexpand to include expanding nested default values (this is the way that normal shell expansion works). The regular crate will only expand ${ZDOTDIR:-/Users/user/.config/zsh}, the modified version will expand ${ZDOTDIR:-$HOME/.config/zsh}.

Use ignore crate for user options when walking directories

When using the ignore crate, many options are provided that the user can fill out. This includes things such as following symlinks, copying hidden files, only matching a certain pattern, setting a max depth, whether the pattern is a regular expression or a glob, whether the pattern is case-insensitive, and patterns to exclude. I also added a global option that is in the format of a gitignore file, that will globally ignore certain patterns (provided by the ignore crate).

Example configuration:

[global_config]
    ignores = ["exe/"]
[hoards]
[hoards.fish]
    [hoards.config]
        # Whether or not to follow symlinks
        "follow_links" = true
        # Whether or not to skip hidden files
        "hidden" = true
        # Pattern of files to only include
        "pattern" = "*.txt"
        # Whether the pattern is to be interpreted as a regex or a glob
        "regex" = true
        # Max depth to traverse
        "max_depth" = 2
        # Whether the pattern should be case sensitive or not
        "case_sensitive" = false
        # Patterns to exclude (notice they are regex patterns, not like the ignores pattern above)
        "exclude" = ["*patt*"]
    [hoards.fish.confdir]
        "linux|fish" = "/home/shadow53/.config/fish/conf.d"

Encryption

I also started to implement the encryption part of the configuration that you had planned. I used gpgme to set this up and took parts of it from another fork that I am working on prs. This should be modified more and functions that are unnecessary need to be removed. I have the asymmetric encryption down, however, for the symmetric encryption, I am able to use gpg's symmetric flags, but it prompts the user for every file it comes across and I cannot figure out a way to capture a password from the configuration file or from the user through the CLI.

I would like to see how you planned on implementing this.

If any of these features sound like something that would like for me to re-fork your project and implement on that fork to push upstream, I definitely can do this. Like I said, I am learning Rust and have been forking projects to do this, so what I do may not be the best.

[Shadow53]

Still figuring out GitHub Discussions. I replied to the thread but apparently not to you, so I'm leaving this here to make sure you get notified.

[Shadow53]

You mentioned curiosity of how I will implement encryption. I have written a summary of how I am planning on implementing symmetric encryption. Asymmetric is still up in the air, since I need to learn more about how that sort of encryption actually works.

[Shadow53]

If there's something that you think this project would benefit from, I definitely welcome issues and pull requests. I'd like to encourage contributions as well, so feel free to make PRs even if you aren't entirely confident in the implementation -- I'll give you feedback and ask for changes if needed.

XDG Folders

AFAICT the XDG folders are not intended for use on macOS. The use of directories::ProjectDirs ensures that hoard uses the appropriate folders for the system being used. I do not want to make exceptions at this time, in part to keep things simple, though I understand and commend the desire to get used to the Linux-y way of doing things.

File Formats

I understand the desire for YAML (less so for JSON). When I initially announced this project on r/rust, one of the questions I received was if I would support YAML. The answer is no, in part for maintainability reasons (it is easier to enforce one file format) and in part because TOML seems to be more consistent than YAML. It might be more verbose, but I am a person to prefer more verbose to inconsistent parsing. It's why I like using Rust, after all.

Preview Config File/Modify by CLI

This is actually a feature I considered when developing the first version of hoard and decided to scrap to focus on more core features. I am not against this being implemented in the future, though other things will take priority first (e.g., encryption). I may consider a hoard edit command that opens the config file in $EDITOR as a sooner addition, though.

shellexpand

I'd like to see the changes you are talking about. In the initial implementation, I went for simplicity and least surprise when expanding environment variables. I see no immediate issue with using environment variables in default values, though, if implemented well. This would make for a good PR, I think.

Using ignore crate

The current implementation uses the glob crate for ignore patterns and offers no configuration options aside from a list of paths to ignore. I am not against further configuration, so I'll address the examples you gave:

• follow_links: hoard does not currently follow symlinks in the first place, so this option is moot.
• hidden: this can easily be replaced with ".*" or "**/.*" glob rules.
• pattern: the ignore configuration item handles this.
• regex: I'm not opposed to allowing regular expressions, but I think glob handles most use cases and is more readable, if requiring more items in the ignore list.
• max_depth: I think this is intended to ignore everything beyond a given depth. I do not intend to directly support this, since I think it runs against my intend with hoard. Should someone want to use this option, a max_depth = 2 I think would be matched with an ignore pattern of "*/*/*" (since ignoring a directory also ignores its contents).
• case_sensitive: this item is supported by glob and is one that I can see supporting in the future. Currently, hoard uses the default of case-sensitive matching. I can wanting to disable this, though, especially for case-insensitive filesystems like NTFS.
• exclude: This is covered by my notes on pattern and regex, I think.

Encryption

I considered making things "easy" on myself by delegating to gpgme for this, but I am not a fan of the idea of using an external, non-Rust library for this purpose. I have ideas for implementation, but I am holding off on writing the code while I do some more research on how to properly construct cryptographic code like this. It needs to:

• Be secure encryption (both good algorithms and implemented securely)
• Encrypt the file paths as well as the contents
• Ideally be as invisible to the user as possible without sacrificing security
• Be compatible with the existing pre-operation checks

This is a complex enough issue that I have removed it from the plans for the upcoming release so that the release doesn't get delayed more than it needs to be. This issue I will probably not accept PRs for, just because it's something I want to make sure gets done as correctly as I can make it.

Summary

So in terms of things that I would possibly accept PRs for:

• Environment variable expansion
• hoard edit command as described above
• Exposing the option for case sensitivity in glob matching

I'd be happy to (try to) mentor any of these implementations, though I can't say I know what that entails. At the very least, I'm open to helping design what changes to make and performing code review to make sure the code is both good Rust and appropriate for this project.