MSMARCO eval split in MTEB English (classic) benchmark

[aashka-trivedi]

Using the benchmarks object to evaluate the classic MTEB benchmark (MTEB(eng, classic)) uses the test split for all datasets. For MS-MARCO, the paper states that the dev set is to be used instead, leading to mismatch in previously reported leaderboard numbers and scores calculated using the instructions in the docs:

tasks = mteb.get_benchmark("MTEB(eng, classic)") # or use a specific benchmark

evaluation = mteb.MTEB(tasks=tasks)
evaluation.run(model, output_folder="results")

[isaac-chung]

@aashka-trivedi thanks for bringing this up! In a previous version where we still have run_mteb_english.py, the implementation was correct ('test' for every task except for MSMARCO). We certainly welcome a PR to fix this issue :)

CC @KennethEnevoldsen