More frameworks and model parameter settings?

[wang-boyu]

There is a Rust ABM framework named krABMaga and I wonder how Agents.jl performs as compared to it. They have implemented the example models used here in their examples repo.

They also have frameworks comparisons similar to yours (code and webpage). In their comparison, the models were run with more number of agents settings, from 1k, 2k, up to 100k.

So instead of the table shown here, would it be better to have more model runs and plot all results as lines? Similar to what krABMaga has in their website but with lines instead of points.

I noticed that the github workflow takes much time to run, so this might be a bottleneck.

cc @rht

[Datseris]

More frameworks are super welcomed, provided that it is possible to run them on CI (which I guess with Rust shouldn't be a problem). Plotting things as line is also great. What we feel is best is to find a solution to save the CSV file of all benchmarks somehow during CI runs (which we haven't figured out yet). Once we have a saved CSV file, not only we have a history of runs from previously tagged editions, but we can also present the output however we wish, e.g., tables or lines.

If you or someone else wants to put together a PR that adds krABMaga for at least one model, and adds CI runs for it that would be great!

[Datseris]

p.s.: 100k agents sounds really high though to do with all frameworks within CI time. I wouldn't be so sure that this is something possible to do with NetLogo during the CI limiting time (I think 6hrs? @Tortar knows better). Is it necessary to go so high? If you are looking for scaling with number of agents, going from 10^2 to 10^4 or 10^5 would anyways reveal the scaling.

[wang-boyu]

What we feel is best is to find a solution to save the CSV file of all benchmarks somehow during CI runs

These perhaps could be useful (but I'm not certain)?

• Storing workflow data as artifacts
• Downloading workflow artifacts

If you or someone else wants to put together a PR that adds krABMaga for at least one model, and adds CI runs for it that would be great!

Unfortunately I've got zero experience with Rust. Perhaps @Tortar or @rht would like to give a try?

[Datseris]

Probably the developers of the software itself are the most knowledgeable of rust CI to do this!

[Tortar]

The big constraint is that to be able to run all models on the same machine we should run all jobs for a model in the same workflow, so we have a 6hrs limit for each model to run all the tests as @Datseris correctly noticed. I'd also like to stay inside the 3hrs limit at maximum for now so that if new frameworks join the comparison we can have time also for them.

So we can do something like this I think to improve the matter:

• Have three version, with something similar in orders as 10^2, 10^3, 10^4 agents each so that we can easily see scaling anyway
• To gain some time we could see if running the models for less steps (e.g. 50 instead of 100) change something by a lot, otherwise reduce the steps
• See if we can find a way not to restart Netlogo headless for each of the 100 iterations for each models, since this takes something like 1hrs alone with three versions of the model.

And surely it would be cooler to have lines at some point, but we first need to store persistently the results in a place different from CI log. Also notice that in the benchmarks on the webpage you linked @wang-boyu the results for Agents.jl are wrong since the setup is not correct, and besides the version used of each frameworks is not specified so we can't even say anything anyway since for example many things in Agents.jl became faster in newer releases.

[rht]

What we feel is best is to find a solution to save the CSV file of all benchmarks somehow during CI runs (which we haven't figured out yet).

These perhaps could be useful (but I'm not certain)?

Storing workflow data as artifacts
Downloading workflow artifacts

The artifact is only stored up to 90 days. A solution that could work is to create another Git repo as a storage, configure this repo (by adding personal access token of that storage repo to this repo's secret), then push the benchmark result to the storage repo.
Edit: to clarify, with a storage Git repo, the benchmark result can be stored indefinitely. Additionally, this method relies less on GitHub-specific stacks.

Regarding with setting up krABMaga, their benchmark repo has the models code and instruction to run them, which is an optimum way to run the code. There is hopefully no distortion in benchmark result when ported, but we could ping the krABMaga devs when the PR is ready.