Perf and Pre-Allocation

[willtebbutt]

This PR

1. makes a function generated, because type inference sometimes falls over for it in some situations which matter, and
2. moves the pre-allocation functionality in-house. It's largely a generalisation of stuff that's already been written in DI.jl, but having it here lets us test it thoroughly here, and then get DI.jl to just use it. This increases the level of confidence that it's possible for me to have that when a user uses DI.jl, they're getting the best possible level of performance.

Todo:

• a few more tests for value_and_pullback!! with a cache
• mark these new functions as "experimental" (I think the design is fine, but I don't want to commit to it yet)
• verify locally that I'm able to use these functions to substantially simplify the implementation of the Mooncake extension in DI.jl.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Files with missing lines	Coverage Δ
src/fwds_rvs_data.jl	96.84% <100.00%> (ø)
src/interface.jl	94.82% <100.00%> (+3.16%)	⬆️
src/tangents.jl	86.98% <100.00%> (ø)

... and 2 files with indirect coverage changes

[github-actions]

Performance Ratio:
Ratio of time to compute gradient and time to compute function.
Warning: results are very approximate! See here for more context.

┌────────────────────────────┬──────────┬─────────┬─────────────┬─────────┐
│                      Label │ Mooncake │  Zygote │ ReverseDiff │  Enzyme │
│                     String │   String │  String │      String │  String │
├────────────────────────────┼──────────┼─────────┼─────────────┼─────────┤
│                   sum_1000 │     70.7 │     1.0 │        5.61 │ missing │
│                  _sum_1000 │      6.7 │  1260.0 │        33.2 │ missing │
│               sum_sin_1000 │     2.24 │     1.7 │        10.6 │ missing │
│              _sum_sin_1000 │     2.58 │   279.0 │        12.9 │ missing │
│                   kron_sum │     54.9 │    3.31 │       187.0 │ missing │
│              kron_view_sum │     53.5 │    8.57 │       198.0 │ missing │
│      naive_map_sin_cos_exp │     2.51 │ missing │        7.09 │ missing │
│            map_sin_cos_exp │     2.83 │    1.54 │        6.12 │ missing │
│      broadcast_sin_cos_exp │     2.58 │    2.26 │        1.45 │ missing │
│                 simple_mlp │      7.6 │    3.27 │        12.0 │ missing │
│                     gp_lml │     11.4 │    4.49 │     missing │ missing │
│ turing_broadcast_benchmark │     3.18 │ missing │        24.9 │ missing │
│         large_single_block │     4.01 │  4660.0 │        30.9 │ missing │
└────────────────────────────┴──────────┴─────────┴─────────────┴─────────┘