put all datasets in wds

[rom1504]

follow up of #47

• put all datasets at hf
• then to reference all the datasets links in a file
• and adapt run.sh to have an option to use webdataset as a source, so it works for all datasets
• important to have enough shards (at least 4)

[djghosh13]

Will leave a note here to also add an option to download (cache) when loading from HF. Mainly, I need to add another CLI option/parameter somewhere, since both source URL (currently specified by --dataset_root) and destination download path should be specified.

[djghosh13]

Another related note: adding some form of support for different languages of classnames and templates to avoid needing to duplicate the whole dataset for multilingual eval

[rom1504]

@djghosh13 would definitely be great to do this issue if you're still interested

[usuyama]

+1 for wds!

I tried HF datasets for images before, but somehow didn't like that much. Maybe because Arrow wasn't very flexible / intuitive for images.

[djghosh13]

Definitely still interested! What are your thoughts on the implementation of this point?

Another related note: adding some form of support for different languages of classnames and templates to avoid needing to duplicate the whole dataset for multilingual eval

Everything else should be straightforward once I actually get around to it.

[rom1504]

We already have support for that, check the code

Currently the path that was taken is to put the other languages prompt and classnames directly in this repo

[djghosh13]

Hm, yeah, I guess do we want to use the same procedure for wds? Currently, the wds loader expects a classnames.txt and zeroshot_classification_templates.txt in the same folder/HF repo as the data.

[rom1504]

I think it makes sense to put in HF/wds the same as in the original source, so the English classnames and prompts for most datasets
And to put new content that we provide from other ways (so for example that's multilingual classnames/prompt lists) directly in clip benchmark

Can think of it as an override

The reasoning is that we cannot add more things in the original source, so doing it that way (rather than adding more languages in all the wds) will keep the source of truth in a single place (clip benchmark repo) for both original source and wds formats.

[rom1504]

done