Solutions for formatter/linter complaints about template syntax

[bradenhilton]

cc @felipecrs

This is a continuation of #1426 (comment), especially #1426 (reply in thread). I'm just going to ramble and see if anything sounds like a good idea. Please do offer your thoughts if you have any.

Have chezmoi create script/env vars to replace template actions with runtime conditionals

• This is currently being implemented so there isn't really anything to brainstorm, just wanted to mention it.

File preprocessors/directives

• Would have an appropriate prefix e.g. chezmoi:delims "# {{" "}}" or chezmoi:replace "# {{" "{{".
  • Can then be placed in language specific comments e.g. // chezmoi:delims "// {{" "}}" or # chezmoi:replace ""# {{" "}}".
• Currently, I can see at least delims and replace/regexReplaceAll being useful, possibly more as well. Maybe a method for forcing a particular encoding or line ending? Not sure.
• It's possible they could break templates if misused.
  • That being said, what doesn't break when misused?
• They would need to be parsed before template execution, as they would alter the data.
  • ASTs would make extracting the comments easier, but there are too many languages to support. So we would probably need to just check if a given line contains a directive or not.
  • A potential optimization could be to mandate that they appear either at the beginning or end of a file, and cease parsing on the first line which doesn't contain a directive, otherwise every line of every template would need to be parsed.
    • Some formatters/linters may complain if certain lines such as shebangs are not at the beginning of the file. I'm currently not aware of any such lines being required at the end of a file.
• The lines containing directives should be removed from the template result.

.chezmoidelims.$FORMAT file

• Can be placed anywhere in the source state, and only acts upon templates within the source state.

• Would have a structure similar to the following:

  .local/share/chezmoi/home/.chezmoidelims.toml:

  ["private_dot_config/powershell/profile.ps1.tmpl"]
      left = "# {{"
      right = "}}"

• A delim can be omitted to use the default value.

delims section in chezmoi config

• Largely similar to the .chezmoidelims file, but with a slightly different structure, perhaps something like:

  [delims]
      "private_dot_config/powershell/profile.ps1.tmpl" = [ "# {{", "}}" ]

Prefixing actions with comments, then fixing up files with a run_after_ script

• This can work for config files.
• Fixing up run_ and modify_ scripts would require editing the source state with a script, which is heavily discouraged.

Some thoughts

• Runtime env vars are a nice solution, but they are only useful for scripts and not config files. They also require extra caution. Imagine accidentally setting PATH to "" because you set it to a variable which has an improper value for whatever reason etc.

• Delims:

  • Seem easier to implement than replacers.

  • Safer, but not very flexible:

    • A left delim of # {{ would result in something like the following multi-action one-liner¹:

      # {{ if eq .chezmoi.os "windows" }}# {{ template "powershell/windows.ps1" }}# {{ end }}

  • Configurable delims could prevent collisions that some users experience with the default ones. An example of this is {{{ and }}} being popular fold markers in vim. Currently users need to replace them with {{ "{{{" }} and {{ "}}}" }}, which makes chezmoi happy, but breaks the folding capability if I recall correctly.

    • The fold markers in vim are configurable, so it's not a huge issue. It would just be a convenience in this case.
    • Not all delim collisions are configurable in the software they originate from, so it would be nice to have a catch-all solution in this case.

• Replacers:

  • Not sure how difficult they would be to implement.

  • More destructive, but also more flexible:

    • Replacing # {{ with {{ would keep templates the same as they are currently (assuming a comment prefix to prevent the formatter/linter complaints)¹:

      # {{ if eq .chezmoi.os "windows" }}{{ template "powershell/windows.ps1" }}{{ end }}

  • Given that replacers edit the template data, I think they should only appear in the template itself. A .chezmoireplace.$FORMAT file or replace config section would create too much separation in my opinion. It also makes any potential support more difficult, whereas I think user configured delims would be more obvious if a template is full of e.g. <<- if .is_headless ->> ... <<- end ->> etc.

• The remaining comments in the template result can likely be reduced to just one by using an intermediate template file and passing its contents through a replace with include. Not a great solution in my opinion.

Footnotes

1. I am aware that the if could be moved into a more general powershell/profle.ps1 template, it was just a random example. ↩ ↩²

[felipecrs]

Just to mention, you can also tinker with some backspaces. Example:

$ chezmoi execute-template '# {{ "\b\bfelipe" }}'
felipe

[bradenhilton]

I think this suffers from the same issue @twpayne outlined in #1426 (reply in thread). It appears to work in the terminal, but not in the actual file:

❯ Get-Content "felipe.txt" -Raw | Format-Hex

   Label: String (System.String) <4AA5BECB>

          Offset Bytes                                           Ascii
                 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
          ------ ----------------------------------------------- -----
0000000000000000 23 20 08 08 66 65 6C 69 70 65 0D 0A             # ��felipe��

[felipecrs]

Makes sense. Thanks!

[halostatue]

Note that chezmoi does not implement its own template engine, but instead uses Go templates. The idea of having .chezmoidelims.$FORMAT for this could work without much difficulty (template.Delims(left, right)). There's a huge gotcha. Delimiters may be specified on the execution of the template, so any included template must also use the same delimiters. Unlike with mustache, delimiters may not be changed mid-execution.

I do not think that replacers make much sense, from my perspective.

[bradenhilton]

Note that chezmoi does not implement its own template engine, but instead uses Go templates.

I am aware of this.

The idea of having .chezmoidelims.$FORMAT for this could work without much difficulty (template.Delims(left, right)). There's a huge gotcha. Delimiters may be specified on the execution of the template, so any included template must also use the same delimiters. Unlike with mustache, delimiters may not be changed mid-execution.

I didn't catch this. Doesn't seem like a huge deal though, one could just set their delims to some obscure combination of characters for all of their templates and never have to worry about it.

I do not think that replacers make much sense, from my perspective.

Would you care to elaborate on this? They require care but also solve the issue perfectly in my opinion. Prefixing template syntax with a language-specific comment stops formatters/linters complaining, then chezmoi removes those comments so the template result is pristine.

[halostatue]

The idea of having .chezmoidelims.$FORMAT for this could work without much difficulty (template.Delims(left, right)). There's a huge gotcha. Delimiters may be specified on the execution of the template, so any included template must also use the same delimiters. Unlike with mustache, delimiters may not be changed mid-execution.

I didn't catch this. Doesn't seem like a huge deal though, one could just set their delims to some obscure combination of characters for all of their templates and never have to worry about it.

Ya. From the godoc for Template.Delims():

Nested template definitions will inherit the settings.

It means that a comment in the template itself would be of limited value.

I do not think that replacers make much sense, from my perspective.

Would you care to elaborate on this? They require care but also solve the issue perfectly in my opinion. Prefixing template syntax with a language-specific comment stops formatters/linters complaining, then chezmoi removes those comments so the template result is pristine.

I see several problems with replacers (especially in-template replacers), but it mostly boils down to the fact that chezmoi does not pre-parse templates in any way, and would probably have several restrictions that should be applied to them so that chezmoi doesn’t have to understand the template format in any way:

• should only be at the top of the template file
• should be one replacer per line
• only affect the top-level template (cannot affect nested templates)
• cannot be specified in nested templates

The reason that it could only affect the top-level template and could not be specified in nested templates is that {{ template "template" }} is not a preprocessor transclusion, but an execution step by the template engine ({{ template "template" }} is equivalent to {{ template "template" nil }}, and most people would actually want {{ template "template" . }}). I do not believe that it is possible to specify a "template preprocessor" that runs whenever a single template gets parsed.

[bradenhilton]

Ya. From the godoc for Template.Delims():

Nested template definitions will inherit the settings.

It means that a comment in the template itself would be of limited value.

The value of the comment comes only from its ability to force formatters/linters to ignore template syntax.

For example, many formatters have configurable rules for bracket/brace/parenthesis spacing, if you enable this and also enable formatting on save, an action can go from {{ template ... }} to { { template ... } } which obviously does not work. This means that you need to format the file, fix the broken template syntax, then save the file without formatting. This is what I do currently and the tedium only grows with time.

The primary issue is that currently you must ensure whatever action you use outputs text after a new line, or the comment will comment out at least part of the text, which can break the template result.

Let's say I have the following PowerShell helper function (which, it just so happens is particularly useful for modify_ scripts on Windows, and so benefits from being a template or include file) saved in a template:

function Read-Stdin {
    [System.Console]::InputEncoding = [System.Text.UTF8Encoding]::new()
    $stdin = [System.Console]::In
    return $stdin.ReadToEnd()
}

If I include this in a templated script I am writing, and prefix it with a comment to prevent linter complaints, and to avoid the brace spacing issue mentioned above, like so:

# ...

#{{ template "helpers/powershell/Read-Stdin.ps1" }}

#...

The template result looks like this:

# ...

#function Read-Stdin {
    [System.Console]::InputEncoding = [System.Text.UTF8Encoding]::new()
    $stdin = [System.Console]::In
    return $stdin.ReadToEnd()
}

# ...

And the script is now broken.

If a newline is prepended to move the content away from the comment, the template result works, but it is now full of empty comments which look ugly.

If I could instead set my left delim to #{{, the template result would look like this:

# ...

function Read-Stdin {
    [System.Console]::InputEncoding = [System.Text.UTF8Encoding]::new()
    $stdin = [System.Console]::In
    return $stdin.ReadToEnd()
}

# ...

There are other workarounds, such as putting all non-templated content in its own file, then including that in a template, which you inform your editor is a plain text file. This prevents the formatter/linter complaints, but that's because they don't act upon plain text files. You also lose language server features. In short, extra intermediate files are required, and a least some parts of files can't take advantage of language servers which can make adding content to them a chore.

If a user does not use any such formatters/linters then they can simply use template syntax in any file without issue.

I agree that the delim inheritance can be a gotcha, but it's almost a complete non-issue unless you're including templates with what would honestly have to be quite absurd levels of nesting. If you're writing a templated script and including helper functions, they should all use the same comment character(s), so the delims can be the same.

I see several problems with replacers (especially in-template replacers), but it mostly boils down to the fact that chezmoi does not pre-parse templates in any way, and would probably have several restrictions that should be applied to them so that chezmoi doesn’t have to understand the template format in any way:

• should only be at the top of the template file

• should be one replacer per line

I'm not sure I follow here, is that not basically what I said in the first post? Chezmoi does not currently pre-parse templates, but it certainly could. Whether or not that is a good idea is up for a debate.

• only affect the top-level template (cannot affect nested templates)

• cannot be specified in nested templates

The reason that it could only affect the top-level template and could not be specified in nested templates is that {{ template "template" }} is not a preprocessor transclusion, but an execution step by the template engine ({{ template "template" }} is equivalent to {{ template "template" nil }}, and most people would actually want {{ template "template" . }}). I do not believe that it is possible to specify a "template preprocessor" that runs whenever a single template gets parsed.

Even with those restrictions, we could do something like have a replacer replace comments in the top level file, and use a new preparseTemplate template function or similar to get the content, no? Again, not saying it's a great idea, but there are ways around some of the text/template restrictions.

[halostatue]

It means that a comment in the template itself would be of limited value.

The value of the comment comes only from its ability to force formatters/linters to ignore template syntax.

I don't think that adding any template pre-processing to Chezmoi would be a good idea—it has too many sharp edges for users. I think that a template override configuration file is a much better idea.

Even with those restrictions, we could do something like have a replacer replace comments in the top level file, and use a new preparseTemplate template function or similar to get the content, no? Again, not saying it's a great idea, but there are ways around some of the text/template restrictions.

To me, the entire concept of replacement instructions is problematic because of the sharp edges that exist.

• It can only work on top-level files.
• It cannot work on any nested template (and the changes to Chezmoi required to make that happen would be almost as large as implementing an entirely new template engine).
• It has a chance of changing templates in unexpected ways.

That's a lot of restrictions that would need to be put into place and I don't envy the documentation (or code changes to support this).

On the other hand, I think that a file-external configuration option to set delimiters would be much better overall.

As an aside, I also think that it might be worthwhile to see if there is another Go-based templating engine that could be added. If there's a .chezmoitemplates.toml file that permits configuration of delimiters, it could also be set up to configure an alternative template engine.

["private_dot_config/powershell/profile.ps1.tmpl"]
  engine = "text/template" # default template engine
  options = {
    left = "# {{",
    right = "}}"
  }

["private_dot_config/powershell/profile.ps1.tmpl"]
  engine = "mustache"
  options = {
    left = "# {{",
    right = "}}"
  }

[bradenhilton]

Thanks for the discussion here. I agree that fixing this is non-trivial and that setting delims is the safer option.

[bradenhilton]

@twpayne I'm extremely hesitant to ask for help on something that is unfinished and isn't exactly my preferred solution, but I'm afraid I've hit my limit when it comes to Go.

I'm attempting to throw together a quick proof of concept for configurable delims just to see if I like the workflow. I'm trying to implement it as a config option (for now, it's just based on how interpreters were implemented) and it works, but the delims aren't inherited when nesting and I'm not sure why.

(This is by no means a complete solution)

❯ git diff HEAD

diff --git a/pkg/chezmoi/delims.go b/pkg/chezmoi/delims.go
new file mode 100644
index 00000000..4ad6e6e8
--- /dev/null
+++ b/pkg/chezmoi/delims.go
@@ -0,0 +1,7 @@
+package chezmoi
+
+// Delims are template delimiters.
+type Delims struct {
+       Left  string `mapstructure:"left"`
+       Right string `mapstructure:"right"`
+}
diff --git a/pkg/chezmoi/sourcestate.go b/pkg/chezmoi/sourcestate.go
index 00a95a62..d14f94ba 100644
--- a/pkg/chezmoi/sourcestate.go
+++ b/pkg/chezmoi/sourcestate.go
@@ -100,6 +100,7 @@ type SourceState struct {
        templates               map[string]*template.Template
        externals               map[RelPath]*External
        ignoredRelPaths         map[RelPath]struct{}
+       delims                  map[string]*Delims
 }

 // A SourceStateOption sets an option on a source state.
@@ -126,6 +127,12 @@ func WithDefaultTemplateDataFunc(defaultTemplateDataFunc func() map[string]any)
        }
 }

+func WithDelims(delims map[string]*Delims) SourceStateOption {
+       return func(s *SourceState) {
+               s.delims = delims
+       }
+}
+
 // WithDestDir sets the destination directory.
 func WithDestDir(destDirAbsPath AbsPath) SourceStateOption {
        return func(s *SourceState) {
@@ -651,7 +658,19 @@ func (s *SourceState) Encryption() Encryption {

 // ExecuteTemplateData returns the result of executing template data.
 func (s *SourceState) ExecuteTemplateData(name string, data []byte) ([]byte, error) {
+       leftDelim := "{{"
+       rightDelim := "}}"
+       extension := path.Ext(strings.TrimSuffix(name, ".tmpl"))
+       delims := s.delims[strings.TrimPrefix(extension, ".")]
+       if len(delims.Left) > 0 {
+               leftDelim = delims.Left
+       }
+       if len(delims.Right) > 0 {
+               rightDelim = delims.Right
+       }
+
        tmpl, err := template.New(name).
+               Delims(leftDelim, rightDelim).
                Option(s.templateOptions...).
                Funcs(s.templateFuncs).
                Parse(string(data))
diff --git a/pkg/cmd/config.go b/pkg/cmd/config.go
index 5314386f..25ce6557 100644
--- a/pkg/cmd/config.go
+++ b/pkg/cmd/config.go
@@ -79,8 +79,9 @@ type ConfigFile struct {
        CacheDirAbsPath    chezmoi.AbsPath                 `mapstructure:"cacheDir"`
        Color              autoBool                        `mapstructure:"color"`
        Data               map[string]any                  `mapstructure:"data"`
+       Delims             map[string]*chezmoi.Delims      `mapstructure:"delims"`
        Format             writeDataFormat                 `mapstructure:"format"`
        DestDirAbsPath     chezmoi.AbsPath                 `mapstructure:"destDir"`
        Interpreters       map[string]*chezmoi.Interpreter `mapstructure:"interpreters"`
        Mode               chezmoi.Mode                    `mapstructure:"mode"`
        Pager              string                          `mapstructure:"pager"`
@@ -1483,6 +1484,7 @@ func (c *Config) newSourceState(
                chezmoi.WithBaseSystem(c.baseSystem),
                chezmoi.WithCacheDir(c.CacheDirAbsPath),
                chezmoi.WithDefaultTemplateDataFunc(c.getTemplateDataMap),
+               chezmoi.WithDelims(c.Delims),
                chezmoi.WithDestDir(c.DestDirAbsPath),
                chezmoi.WithEncryption(c.encryption),
                chezmoi.WithHTTPClient(httpClient),

I'm using the following source state to test:

Source state

chezmoi.toml:

[delims.sh]
    left = "#{{"

test.sh.tmpl:

#{{ template "test.sh" }}

.chezmoitemplates/test.sh:

#{{ template "1" }}

.chezmoitemplates/1:

echo "hello world"

And I get the following output:

test.sh:

#echo "hello world"

The extra comment comes from .chezmoitemplates/test.sh, if I remove it the comment goes away.

@halostatue feel free to offer thoughts also.

[halostatue]

Shouldn’t your chezmoi.toml configuration be fore test.sh or .chezmoitemplates/test.sh?

[halostatue]

Opening this as a draft pull request would make it possible for us to play with the code in situ, as well.

[bradenhilton]

@halostatue it's matching on the file extension, as interpreters does. [delims.sh] matches (or at least I intended for it to match) templates with a .sh extension after stripping the .tmpl extension. Matching on paths (with glob support) would probably be ideal but I wanted to get something working before I explored that.

My understanding is that .chezmoitemplates/test.sh is supposed to inherit the delims from test.sh.tmpl (which are set as exptected).

Re: draft PR, as above, I wanted to wait until I got it working before I did so, as I'm still unsure if this is a good idea and I'm conscious of wasting everyone's time with something that isn't even complete. You make a good point though.

[twpayne]

+1 please open this as a draft PR.

[bradenhilton]

In progress! :)

[twpayne]

Thanks to @bradenhilton and @halostatue, you can now specify per-file template delimiters by including a template directive directly in the template, like:

chezmoi:template:left-delimiter="# {{" right-delimiter=}}

See #2457 for the final implementation.