Evaluate multiple models/datasets/languages using the CLI directly

[mehdidc]

Issue #43

So multiple ways to do that with this PR (will add doc in the README later) .

For models, we can provide list of pretrained model names in the form of 'model,pretrained' (so model and pretrained are comma separated). For datasets, we can provide a list of datasets. For languages, we can provide a list of languages.
Example:

clip_benchmark --pretrained_model  ViT-B-32-quickgelu,laion400m_e32 ViT-L-14,laion400m_e32  \
--dataset cifar10 cifar100 --dataset_root "clip_benchmark_datasets/{dataset}" --language en jp \
--verbose --output "{dataset}_{pretrained}_{model}_{language}_{task}.json"

Note that --dataset_root and --output can be now in the form of a template that depends on the dataset/model/language/task (for --output) and dataset name (for --dataset_root).

We can also provide files with models or datasets list (one per line):

clip_benchmark --pretrained_model  benchmark/models.txt \
--dataset benchmark/datasets.txt --dataset_root "clip_benchmark_datasets/{dataset}"  \
--verbose --output "{dataset}_{pretrained}_{model}_{language}_{task}.json"

We can also provide model collection names (openai, openclip_base, openclip_full are supported) or dataset collection names (vtab, vtab+, retrieval, imagenet_robustness are supported):

clip_benchmark --pretrained_model  openai openclip_base  --dataset vtab+ retrieval \
--dataset_root "clip_benchmark_datasets/{dataset}" --verbose \
--output "{dataset}_{pretrained}_{model}_{language}_{task}.json"

(openclip_base is the same as benchmark/models.txt, while openclip_full is using OpenCLIP's open_clip.list_pretrained_models())

The evaluation is sequential, but we can think in the future how to do evaluation in parallel in multiple GPUs (out of the scope of this PR).

@rom1504 what do you think, anything else we needed?

[rom1504]

great!

What do you think about putting the same explanation as this PR description directly into the readme ?

[rom1504]

about what else we need, I think in future PRs those would be great to increase usability some more:

• generate important diagrams from the notebook automatically. much easier to read a picture than a csv!
• put all datasets in wds #52 to make it even easier to get datasets

[mehdidc]

great!

What do you think about putting the same explanation as this PR description directly into the readme ?

Yes good idea

[mehdidc]

about what else we need, I think in future PRs those would be great to increase usability some more:

* generate important diagrams from the notebook automatically. much easier to read a picture than a csv!

* [put all datasets in wds #52](https://github.com/LAION-AI/CLIP_benchmark/issues/52) to make it even easier to get datasets

Alright thanks, will have a look at these afterwards

[mehdidc]

Added README, instructions on how to run the zero-shot benchmark without run.sh (with also multilingual benchmark). Will do some tests to find out if there any issues, but otherwise I believe it's fine.

[rom1504]

Let's maybe integrate this in main cli ?

[mehdidc]

Thanks, done, looks indeed better like that. Perharps we should do the same for clip_benchmark_export_wds (in another PR).

[rom1504]

Looks great

[mehdidc]

Ok, I ran again the full VTAB+ and retrieval benchmark with the new CLI, all worked fine, I just see some variation (compared to current benchmark.csv) in the numbers in the 3rd digit after decimal point, and a few times in the 2nd digit after decimal point, perhaps it is because of AMP (which I use now, didn't use before). Variation in 3rd digit after decimal point is probably okay, in the second digit I'd be more worried, will need to investigate the effect of AMP after this PR.

Below I show the delta between current and benchmark that I reran:

Seems to happen in fer2013, mnist, renderedsst2, diabetic retinopathy, kitti, pcam, with diabetic retinopathy being the worse.

Retrieval looks fine:

[rom1504]

Looks good to me, let's merge it !

[mehdidc]

Ok then let's merge