Nexus

Nexus is a dependency injection framework that leverages Pathom’s attribute resolving powers to automatically infer the data and lifecycle needs of a system. The goal is to keep developer toil to a minimum and programming fun, while keeping the overall system tightly bound together.

1. Components specify the data they need and are used like a function with their arguments automatically passed to them, Fulcro-style.
2. The system specifies exactly which components it wants – these are parts of your application that are useful in themselves, not just to support some other part of your program – like a webserver or a Kafka Streams application.
3. You provide a minimal amount of initial information, a flat map (representing a Pathom entity) populated by e.g. (System/getenv) calls.
4. Pathom makes the magic happen, getting both sides exactly what they asked for and turning up your system, without any effort from the programmer!

Rationale

Pros

• Learning efficiency. 99% just Pathom, nothing more to learn, no additional cognitive overhead.
• Ergonomics. Not having to pass data around and automatic suspend/resume lets you iterate faster.
• Performance. Uses Pathom’s parallel runner to turn up the system in parallel.
• Debugging. Plug in Pathom Viz for an graphical overview of complex topologies and traces (caveat: Viz may get slow/broken when trying to write large values via Transit, which components often are)
• Extensibility. Complex behavior can be injected via Pathom plugins (Nexus itself is mostly just a plugin).
• Scope. Feed components with data over the network, through federated EQL parsers instead of config files or environment variables with Pathom 3’s dynamic resolvers.

Cons

• You will need to learn Pathom, and Pathom is a large dependency to take on just for this one purpose. But there’s a great tutorial and it really is useful, probably Clojure’s killer app (and I try all of these).

Status

Lightly tested. I migrated from a ~100 line Integrant/Aero config having been frustrated with how slow the reloads were getting and saw no way of optimizing Integrant, and have been pleased with how much faster the development experience with Nexus is. It fades into the background and lets me write as if any component already has all the data it needs, without round-trips to a master config.

Motivation (why not Integrant?)

Dependency injection is a good practice, but it is cumbersome to work with, as writing a function that relates to external state in any way entails a round-trip to the controller where you do some manual and redundant data entry. Consider this example of an Integrant function that closes over some configuration info about an external API:

(defmulti ig/init-key ::call-api [_ {:keys [addr secret]}]
  (fn [args] (http/get addr (assoc args :secret secret))))

The inversion of control is good to have here, because it means we can easily stub it out for tests and such.

But say we had some function, happily pure..

(defn do-thing [foo bar]
  (do-something foo)
  (do-something-else bar))

Called from some other component:

(defmulti ig/init-key ::call-other [_ {:keys [foo]}]
  (fn [args]
    (do-thing foo "bar")
    (do-other-thing)))

If we one day decided that do-thing needs to call that API, we have to turn it into an Integrant key (or pass around a big map, which is arguably even worse) so it can have access to call-api:

(defmulti ig/init-key ::do-thing [{:keys [call-api]}]
  (fn [foo bar]
    (call-api {:arg "foo"})
    (do-something foo)
    (do-something-else bar)))

And then update the Integrant config:

{:my.ns/call-api {:addr "foo" :secret "bar"}
 :my.ns/do-thing {:call-api #ig/ref :my.ns/call-api}
 :my.ns/call-other  {:do-thing #ig/ref :my.ns/do-thing
                     :foo      "foo"}}

And then change call-other:

(defmulti ig/init-key ::call-other [_ {:keys [do-thing foo]}]
  (fn [args]
    (do-thing foo "bar")
    (do-other-thing)))

The Integrant config is eventually going to control a lot of the program, putting many unrelated concerns all in one place and forcing the programmer to be explicit to an end that doesn’t really merit the effort.

Using Nexus

With Nexus, we can instead do it like this:

(ns my.ns
  (:require [com.nivekuil.nexus :as nx]))

(nx/def call-api [{::keys [addr secret]}]
  {}
  (fn [args] (http/get addr (assoc args :secret secret))))

(nx/def do-thing [{::keys [call-api]}]
  {}
  (fn [foo bar]
    (call-api {:arg "foo"})
    (do-something foo)
    (do-something-else bar)))

(nx/def call-other [{::keys [do-thing foo]}]
  {}
  (fn [args]
    (do-thing foo "bar")
    (do-other-thing)))

(nx/def server [{::keys [call-api call-other opts]}]
  {}
  (start-server (register-apis [call-api call-other]) opts))

do-thing itself knows that it needs call-api, and call-other is left alone. There is no need to alter a config map to add a component.

To start this system, we call nx/init with a config map (really a Pathom entity) and the targets we want to turn up. Note that the target is specified by a keyword corresponding to the fully-qualified name of the symbol in nx/def.

(def sysenv
  (nx/init #:my.ns{:addr   "foo"
                   :secret (slurp "secret.txt")
                   :foo    "foo"
                   :opts   {:port 80}}
           [:my.ns/server]))

As you can see, initializing our system only needs to provide top-level information and specify top-level components. Ultimately, the system is trying to start a server. The programmer does not specify all the data the server needs to start, because the server component itself already knows what it needs, and so on recursively. If any component on the critical path does not have all the data it needs, the system will not start and an error will be thrown.

We also may want to do some cleanup action when we want to stop the system. This is done with a ::nx/halt key whose value is a function that takes the return value of the nx/def block and does something to it.

(nx/def server [{::keys [call-api call-other opts]}]
  {::nx/halt stop-server} ;; single arg function
  (start-server (register-apis [call-api call-other])))

Then we can halt the system with the value returned from nx/init

(nx/halt! sysenv)

Note that nx/def is essentially just pco/defresolver. The map that follows the args is the same one where you normally would place ::pco/input and most Pathom attributes are valid. The exception is ::pco/output as the returned attribute is always derived from the resolver name.

Resetting

For developer convenience, there is also a nx/reset function (for REPL/reloaded workflow usage, see below) that you can call on the sysenv returned from nx/init along with the targets (just like p.eql/process), which will skip components that have not changed. A component has changed when either its inputs or its body has changed. Unlike Integrant, Nexus components are defined by a macro so Nexus can actually detect when the source code of a component has changed for suspend/resume invalidation purposes!

This caching is on by default. To always reload a component (something you want if it references a protocol, since those are recreated on load) you can set the keyword ::nx/cache? to false in the component options map. To turn off the cache by default, set the same keyword in the env (using :env-transform in init) to false.

While this opinionated behavior should save a decent amount of effort (cf. Integrant’s readme) it is not as flexible and efforts to improve it should it prove deficient are welcome.

Debugging

A resolver with ::nx/debug? will log its value whenever it is evaluated.

Compared to Integrant

Integrant complects the shape of your system with the information it needs. With Integrant, you provide a tree, represented by a map or EDN file. With Nexus, you take the component-local code you’ve already written (like the genetic code stored in a seed), give it the conditions under which it will sprout (a flat map of usually namespace-qualified attributes), and Pathom grows the tree for you.

In Integrant, structure is nested and components take unqualified arguments. In Nexus, structure is flat and components take qualified arguments, i.e. names with globally consistent referents. Whereas Integrant uses hierarchical relationships, a Nexus config is just a Pathom entity, which provides all the up-front information necessary for any component to connect to any other component in a flat map. There is no external schema that pre-determines the connections that can be made.

Nexus really is just a few lines of code making use of Pathom. This means if you know Pathom, you already know 99% of Nexus. If you don’t, then you might want to learn it anyway because Pathom is an incredibly versatile tool, and quite addicting to use once you get into the logic programming mode of thinking.

We can also take advantage of all Pathom’s features. Notably, parallel runs (not yet in Pathom 3), plugins (Nexus itself is mostly just a plugin), and Pathom Viz, for a top-level graphical view of the system and to trace execution times to debug slow-starting components.

That brings me to the most significant reason I wrote Nexus, out of anger with Integrant. Integrant can do suspend/resume, but it has to be manually wired in, and if you do it based on inputs to the component then the component will not refresh even if you change the body of the component itself, if the inputs haven’t changed. This forces you be careful about inconsequential things, juggling several more things during development, and is not fun.

Integrant is often used with Aero, which has its merits but more advanced usage entails thinking in an M-expression style DSL based on EDN tags. As such, plain Clojure actually ends up being more data-centric than what is superfically EDN, insofar as the appeal of data over code is a matter of data being more reusable, more general, or less opinionated about its environment. So while there is value in having a cut down language for configuration, I found Aero’s benefits too marginal to justify this extra weight. You can still use Nexus with Aero if you choose to, of course.

Namespaces

deps.edn (sha only for now)

{:deps
 {com.nivekuil/nexus {:git/url "https://github.com/nivekuil/nexus"
                      :sha     "..."}}}

require

(:require [com.nivekuil.nexus :as nx])

clj-kondo

In .clj-kondo/config.edn:

{:lint-as {com.nivekuil.nexus/def clojure.core/defn}}

Reloaded workflow

In .dir-locals.el

(cider-ns-refresh-after-fn . "user/go")

In dev/user.clj

(clojure.tools.namespace.repl/set-refresh-dirs "src/main")

(defn go []
  (let [config  (system/config :dev) ;; the initial config map/pathom entity
        targets [:app.server/server :app.logger/logger]]
    (nx/go config targets {})))

Logging

To get a better idea of what Nexus is doing under the hood, set this somewhere in your repl: (alter-var-root #'nx/log? (constantly true))

Caveats

The ::nx/halt cleanup function is just an entry in the Pathom resolver config, like ::pco/output, ::pco/input etc. This is evaluated at resolver definition time and does not close over the input – it only takes the resolver return value!

Take care when relying on dynamic requires for dependencies alone, without requiring the namespace explicitly. This can be convenient for breaking cyclical dependencies but can cause problems with AOT and tools.refresh. You may need to explicitly require namespaces or things can be missing at runtime (though Nexus will fail quickly and loudly).

You might run into stale caches. The reset logic currently looks at the body of `nx/def`s to determine whether it should be reloaded. If the code inside `nx/def` references something outside, and only that outside thing changes, Nexus will not know to reload the `nx/def`s that are referencing that outside thing. You can avoid this by making `nx/def`s reference nothing (and therefore close over nothing) outside its own declared inputs, or figure out a way to improve this library so that it can account for external references.

TODO:

• cljs? should be easy since Pathom natively supports it
• think about how derived components would work – good enough already?
• can ::nx/halt close over params?