Added Jinja templating preprocessing engine as well as unit test fixes when running on Windows.

[PredatorVI]

I've added the Jinjava Jinja pre-processing engine to the plugin to allow for embedded Jinja templates.

I'm not yet completely aware of what limitations might exist in Jinjava compared to official Jinja/Jinja2 implementations.

The unit test is minimal (creates 3 pipelines in a loop) but I can add more complex examples over time as needed. The hope is to be able to reduce complexity and duplication on an even larger scale than currently allowed, including (but not limited to):

• importing shared/common code
• looping over entire sections
• taking advantage of defining common values in a single file/location and using those values throughout.

Feedback appreciated.

[ketan]

This is @tomzo's call. But from the surface, this looks good to me.

After @tomzo approves, would you mind updating the readme with pointers to a few example tests showing some of the features you mention?

I'll setup something to do dependency checks on transitive dependencies. To make sure that the dependencies are not using a copyleft license, vulnerabilities etc.

[PredatorVI]

Will do. I am definitely working on more comprehensive examples. It may take a couple of days.

[ketan]

No hurries, take your time. This will go in the next release of GoCD, due in another month or so. - Ketan

…

On Tue, Jan 29, 2019 at 8:57 AM PredatorVI ***@***.***> wrote: Will do. I am definitely working on more comprehensive examples. It may take a couple of days. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#104 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAApZttBWU8E82X5Q_1t9zXapVAfcRSaks5vH7-ZgaJpZM4aXArg> .

[tomzo]

Thanks, this is a great feature.
It would be nice to add examples in readme. Let's wait for that before merging.

Just a thought, maybe it would be better to have files meant for template preprocessing end with gocd.yaml.j2 ?

[PredatorVI]

I wondered about a separate extension, but opted not to based on my experience on how YAML is used in SaltStack or Ansible, or similarly with how a C/C++ compiler will preprocess .c/.cpp files. In both cases, there is no distinction between files containing template directives vs files without template directives.

[ABETES]

Dear all,
dear PredatorVI,

we have successfully applied your much appreciated pull request to our build environment.

The plugin does not yet support the definition of templates that pipelines can reference to.
The Jinja templating engine offers the ability to do exactly that. We would even be able to recursively include modularized template fragments.

But we failed to make use of Jinja's include statement.

test.gocd.yml
pipelines:
  testpipeline:
    group: testpipelines
    materials: 
      ...
    parameters:
      ...
{% include 'testing/test-template.tmpl' %}

test-template.tmpl
  stages:
    - teststage1: 
      jobs:
        testjob:
          tasks:
            - exec:
              command: echo 
              arguments: "test"
...

To solve the "issue" we would like to ask you for the following addition to your work:

In YamlConfigParser you are creating an instance of Jinjava.
This Jinjava does not know where resources are located. Thus the (Jinja-) import or extention of other templates do fail:
//Pre-Process the file through the jinja template engine
Jinjava jinjava = new Jinjava();

The reason is: Jinjava uses the ClasspathResourceLocator.
Usually the files to be included are not located in the Classpath but in the working directory - side-by-side in our case. This would mean: Using the FileLocator implementation instead would make the includes and extention work:

Jinjava jinjava = new Jinjava();
jinjava.setResourceLocator(new FileLocator(baseDir));
whereas baseDir should come from the parseFiles method.

What do you think of the proposal?

Thanks
Thomas
Arne

[PredatorVI]

@ABETES (et. al.),

I'm a little slow getting examples put together but I sort of expected needing a custom resource locator. Thanks for working through that.

It makes sense to use the 'baseDir' as the root context from parseFiles(). This would limit the inclusion scope to a per repo "namespace" and feels like the right way.

My question would be how to handle the cases where parseStream() is called directly (from YamlPluginCLI)? I've not used YamlPluginCli so I don't know what the use cases are and what baseDir should be? Current working directory? Parent directory of 'syntax.file'?

Thoughts?

[arvindsv]

@PredatorVI Good question. I'd say, for now set it to current working directory. Once @marques-work is back from vacation, we might need to rethink the files being sent over from the GoCD server side. I'm not sure if the entire directory is sent to parseStream, but I can see it's processing the files in memory. So, the template files won't be available to the file locator, but we can cross that bridge a little later.

[marques-work]

@PredatorVI Hi there, thanks for the contribution of such a nice feature!

I wanted to shed some light on how parseStream() is used directly. You've seen that there is a usage in YamlPluginCli -- this supports a quick syntax check feature for an external tool.

Most likely, the main use case is probably going to be in the parse-content request, which receives a collection of named string contents that are iteratively passed to parseStream(). Conceptually think of that as a tar archive being sent over the wire. The parse-content request supports a "preflight" API to check an arbitrary collection of pipeline config definitions for both structural validity and dependency satisfaction against a running GoCD server.

Anyhow, you're probably right that we'd need some kind of custom resource locator for this to work in the above case. I haven't looked at Jinjava, but I'm hoping it's not difficult to create one for this use case.

One more restriction to think about here with regards to parse-content and parseStream(). This payload I've described above is typically constructed from a multipart upload during an API request. Generally, most clients do not include the full paths of the files being uploaded, but rather just the filenames. This means this payload will generally be a flattened hierarchy; all files will appear as direct siblings.

Regarding file extensions, I would prefer that the template files have distinct extensions as @tomzo suggested. It's theoretically possible for files to have contents that look like tokens, but aren't intended to be. Perhaps an argument to an exec task might contain a string that looks like a token to Jinja? I'm sure there are more clever examples than this one I mentioned, but my point is that I think we should be explicit when we want our configs rendered vs taken raw. That could lead to unintended consequences and hard-to-debug errors. Not to mention unnecessary processing on raw files.

[tomzo]

but my point is that I think we should be explicit when we want our configs rendered vs taken raw. That could lead to unintended consequences and hard-to-debug errors. Not to mention unnecessary processing on raw files.

Totally agree with all that. Let's be nice and declare what is supposed to be expanded and what is not.

Moreover, after considering the templates fragments used in jinja include or extends, I think we are actually looking at 3 types of files:

• normal gocd.yaml - no expanding needed, but then parsed by plugin to get pipelines and environments.
• gocd.yaml.j2 - a template, which should be expanded, then parsed by plugin to get pipelines and environments.
• gocd.tmpl.j2 - a resource file, which may be referenced in jinja template. This is not intended to be parsed, but must be present in API request, so that later it can be included in jinja resource locator.

@marques-work does that sound right to you?

Generally, most clients do not include the full paths of the files being uploaded, but rather just the filenames. This means this payload will generally be a flattened hierarchy; all files will appear as direct siblings.

@marques-work I think, we should recommend the clients to actually include the full path then. Otherwise there is no way for statements like this {% include 'testing/test-template.tmpl' %} to work in the preflight.

I'd like to get this feature going, since it has stalled a bit.
I think we should implement properly the resource files in separate PR, while also integrating with the gocd-cli and properly testing with preflight API.
As far as I know, @PredatorVI has submitted enough for templates to be expanded already but without external files. We can close the scope of this PR at that.

Here is how I see getting this feature further:

1. @PredatorVI please rebase this as is, and add any docs you have.
2. @PredatorVI let's introduce second pattern for files meant to be used by jinja templates, while others will be parsed as they were before. By default **/*.gocd.yaml.j2.

Then we can work on resource files in next PRs.

@PredatorVI @marques-work Sounds good?

[marques-work]

@tomzo yes, there are potentially 3 types of files that you enumerated that we treat differently.

@marques-work I think, we should recommend the clients to actually include the full path then. Otherwise there is no way for statements like this {% include 'testing/test-template.tmpl' %} to work in the preflight.

I was actually referrring to HTTP clients (like curl) that don't include the full path when constructing a multipart upload payload. For example:

curl -F 'files[]=@/abs/path/to/file1.gocd.yaml' -F 'files[]=@/abs/path/to/file2.gocd.yaml' http://url

curl will not send the absolute path in the POST body, but only the filenames. That is, the above request will only contain file1.gocd.yaml and file2.gocd.yaml in the multipart parts instead of the full path we provide on the command line :(. This presents a problem where we can't resolve files in subdirectories if we issue a curl request to preflight. In the gocd-cli, we might be able to control this better, but not for curl afaik.

@PredatorVI @tomzo this will probably mean that users need to put all files in the same directory as templates if they want to use include or extends.

I agree that we can integrate this first without support for external include and extends support and figure out these solutions later. 👍

[arvindsv]

@marques-work Might be worth checking if this works:

curl -F 'files[]=@/abs/path/to/file1.gocd.yaml;filename=path/to/file1.gocd.yaml' url

[marques-work]

@arvindsv I will try that but I suspect that will create another boundary with field name filename set to the value path/to/file1.gocd.yaml

[marques-work]

@arvindsv it's always nice to be wrong too :)

e$ nc -l 8080
POST / HTTP/1.1
Host: localhost:8080
User-Agent: curl/7.43.0
Accept: */*
Content-Length: 71002
Expect: 100-continue
Content-Type: multipart/form-data; boundary=------------------------f57a6e080bb2fa20

--------------------------f57a6e080bb2fa20
Content-Disposition: form-data; name="files[]"; filename=".idea/workspace.xml"
Content-Type: application/xml

So I suppose we could support hierarchies by putting the burden on the user to override the filenames for each file they post for the API. All the more reason to favor the CLI tool IMO since we could take care of this automatically there.

We'll also need some careful documentation, but at least we know there is a workaround involving a minimal amount of extra typing.