Global Macros for $2,000, Alex.

[joshuataylor]

Hi there,

I'm returning again to creating a project that parses a dbt project, where the original implementation uses Jinja in templates.

Minijinja has been fantastic (it can parse hundreds of files in milliseconds) so far.

One issue I have is how to handle "global macros", ie Macros that are stored in a file that isn't in the same file as the template.

Let me explain, you don't need to understand/care about what dbt is, but let me share some background.

dbt + macros

The way the dbt developer experience works is that you store acros in a folder, then any file can reference that macro without including it. This is incredibly powerful, and macros are used extensively as it's easy.

Macro example

So you could have a macro foobar that looks like this (in macros/foobar.sql)

{% macro foobar(baz) %}
1 as {{ baz }}
{% endmacro %}

Then inside another file you can do this:

select
{{ foobar('x') }},
{{ foobar('y') }},
{{ foobar('z') }}

Which would output
select
1 as x,
Etc

Minijinja

Initial thoughts

Honestly this goes against many of the core ideas and principles of Minijinja - KISS (Keep it simple, stupid).

Workarounds

Right now as a proof of concept I'm including any referenced macros at the head of the file when parsing the SQL file.

Note this was done as a proof of concept and is terrible idea!

Steps

1. Preparse all macros in the directory, validate correct, etc. Create a rust list containing all macro names.
2. Parse the file using normal rust reading, check if the file contains a reference to a known macro. If it does, add the contents of the macro file to the top of the file.
3. Parse the file using Minijinja.

If a file is foo.sql and includes a reference to something that looks like a macro, I do a dirty hack where I check for what looks like a macro, then check if it's an existing function or a known macro.

Implementation

Unsure of this, didn't want to put ideas which might lead others astray.

I would love a way to fallback from loading functions to loading a macro I could define ahead of time.

Bounty

I am willing to put a $2,000AUD bounty on this. This has been living rent free in my head for too long.

Am I crazy? Yes. Am I serious? Also yes.

[mitsuhiko]

What makes this more annoying than necessary to implement is this issue: #329

If it were not for that, you could pass a macro to another template like so:

use minijinja::{Environment, context};

let env = Environment::new();
let tmpl = env
    .template_from_str("{% macro m(a) %}{{ a }}{% endmacro %}")
    .unwrap();
let (_, state) = tmpl.render_and_return_state(()).unwrap();
let m = state.lookup("m").unwrap();
assert_eq!(m.get_attr("name").unwrap().as_str(), Some("m"));
let rv = m.call(&state, args!(42)).unwrap();
assert_eq!(rv.as_str(), Some("42"));

let tmpl2 = env.template_from_str("{{ m(23) }}").unwrap();
let rv = tmpl2.render(context! { m }).unwrap();
assert_eq!(rv, "23");

Unfortunately today this just will just fail with cannot call this macro. template state went away. as there is strongly held reference to the state. So even if the state is actually not unloaded, the macro cannot be called. I'm not sure how to best address that limitation as lifetimes make this very tricky to resolve.