DifferentiationInterface instead of individual backends?

[gdalle]

Following our conversation with @lxvm today, opening this issue to keep track of a potential DI integration here (pun intended). What would be the major hurdles?

[ChrisRackauckas]

I don't think DI has a rules system that applies rules to all backends?

[gdalle]

No, for that we would still borrow individual rule systems. But inside of these rules we compute derivatives, and it can be with an arbitrary backend, not only the one we define the rule inside

[lxvm]

As @gdalle said, we would have to implement rules in each rules system, such as ChainRules.jl, but perhaps these could all call a single implementation of the pullback/pushforward. Then I would have to implement the pushback/pullforward using DI with a backend selected with the pre-existing sensealg keyword in the init function. As for hurdles, I think they include implementing the Leibniz integral rule for derivatives w.r.t. parameters and the limits of integration for all the integral function APIs. Few of the algorithms support a discretize then differentiate approach, and the rest only work with differentiate then discretize. Our broken derivative tests are a good place to track progress.