Benchmarking

We provide benchmarks to encourage collaboration and competition in making actuarial software run faster than ever before.

Containerized benchmarks

We currently only have 1 benchmark, we are working to expand the benchmarks. Open a discussion in this repository if you have any thoughts to share.

Time measurement is currently the best of 3 runs.

benchmark	classification	container	A100-SXM4-40GB	H100-SXM5-80GB
BasicTerm_ME 100 Million	recursive PyTorch	link	15.8284s	7.205s
BasicTerm_ME 100 Million	compiled iterative JAX	link	3.448s	1.551s

Notes

• BasicTerm_ME 100 Million
  • You can find lifelib's modelpoint file with 10,000 modelpoints as model_point_table.xlsx. We use these modelpoints, but repeat them 10,000 times for 100,000,000 modelpoints.

GitHub-hosted runners

These benchmarks run in GitHub-hosted runners in GitHub actions. Used for benchmarks that are not computationally intensive.

Benchmarks in this repository:

• basic_term_benchmark: Replicate the cashflows of the LifeLib BasicTerm model
  • Python LifeLib BasicTerm_M
  • Julia using LifeSimulator
  • Python using recursive formulas with PyTorch and NumPy
  • Python using matrix operations (no recursion) on PyTorch arrays and NumPy arrays
• exposures: Create date partitions for experience studies
  • Julia ExperienceAnalysis
  • R actxps
• mortality: Read SOA mortality tables and use them in a simple calculation
  • Julia MortalityTables
  • Python Pymort

The below results are generated by the benchmarking scripts in the folders for each language. These scripts are run automatically by GitHub Actions and populate the results below.

basic_term_benchmark:
- Julia array basic_term:
    minimum time: TrialEstimate(30.279 ms)
    result: 1.4489630534602132e7
  Julia recursive basic_term:
    minimum time: TrialEstimate(84.812 ms)
    result: 1.4489630534602132e7
- Python array numpy basic_term_m:
    minimum time: 83.45314299992879 milliseconds
    result: 14489630.534603368
  Python array pytorch basic_term_m:
    minimum time: 51.54931000004126 milliseconds
    result: 14489630.534603368
  Python lifelib basic_term_m:
    minimum time: 618.0503439999256 milliseconds
    result: 14489630.534601536
  Python recursive numpy basic_term_m:
    minimum time: 63.08064000006652 milliseconds
    result: 14489630.534603368
  Python recursive pytorch basic_term_m:
    minimum time: 75.6699999999455 milliseconds
    result: 14489630.53460337
basic_term_me_benchmark:
- Python heavylight numpy basic_term_me:
    minimum time: 354.6492100000478 milliseconds
    result: 215146132.0684811
  Python lifelib basic_term_me:
    minimum time: 1191.792300999964 milliseconds
    result: 215146132.06848112
  Python recursive numpy basic_term_me:
    minimum time: 309.30894900006933 milliseconds
    result: 215146132.0684814
exposures:
- Julia ExperienceAnalysis.jl:
    minimum time: TrialEstimate(29.659 ms)
    num_rows: 141281
- R actxps:
    min: 486.248656 ms
    num_rows: 141281
mortality:
- Julia MortalityTables.jl:
    minimum time: TrialEstimate(229.507 μs)
    result: 1904.4865526636793
- Python PyMort:
    minimum time: 9.425531999909253 milliseconds
    result: 1904.4865526636793
savings_benchmark:
- Julia Benchmarks savings:
    minimum time: TrialEstimate(119.226 ms)
    result: 3.507113709040273e12
- Python lifelib cashvalue_me_ex4:
    minimum time: 596.2715760000492 milliseconds
    result: 3507113709040.141
  Python recursive numpy cashvalue_me_ex4:
    minimum time: 543.3022800000344 milliseconds
    result: 3507113709040.124