leaderboard v2.0: disagreement in scores between v2.0 and v1.0

[KennethEnevoldsen]

56,72375Some results seem to not agree with each other between the old and new leaderboard e.g. the ranking and scores for the Law tab look quite different

Here is an example:

For gritlm, multilingual-e5-large-instruct, multilingual-e5-base at least the models (generally) agree:

The tasks seem to match between v1 and v2.

Seems like scores on both benchmarks are:

scores	task
35.29	AILACasedocs
41.8	AILAStatutes
20.61	GerDaLIRSmall
64.22	LeCaRDv2
82.05	LegalBenchConsumerContractsQA
95	LegalBenchCorporateLobbying
44.18	LegalQuAD
70.64	LegalSummarization
56.723	avg

@x-tabdeveloping is there an issue with rounding here? it should be 56.72 not 56.73 (minor though)

Additionally not sure why the mean retrieval is not the same as the mean(task)?

originally posted to #1317