Webdataset updates

[djghosh13]

(Mostly) addresses issues #52 and #67

• Support for conversion and evaluation of retrieval datasets (clip_benchmark_export_wds --retrieval)
• More complete API for converting other classification/retrieval datasets, with import clip_benchmark.webdataset_builder)
• (Not in commit) All VTAB+ and retrieval datasets have been uploaded to HF
• Readmes have been updated appropriately, default suggestion in benchmark/README.md is to use webdataset

Not completed:

• Multilingual support, i.e., overriding default classnames & templates with other languages
• Converting voc2007_multilabel

[djghosh13]

Note: I've only tested the benchmark code with a single model, so I haven't run the complete experiments. There are some minor differences in numbers from what @rom1504 gave me, but no differences from what I get with the original datasets when I run them myself.

[rom1504]

Nice, will check it out

[mehdidc]

Really cool, thanks @djghosh13! For differences in numbers, might be related to this issue #59

[djghosh13]

I see, yeah, I think the differences I saw were also in the 0.001s range.

[rom1504]

Are datasets getting cached locally? Where and is it tweakable ?

[djghosh13]

I forgot to add this to the readme. By default, no, but there is a new --wds_cache_dir parameter in the CLI which is directly passed to Webdataset(cache_dir=) if a path is given.

[djghosh13]

I hadn't actually tested it before, but it looks like it will save the .tar files inside the specified cache directory in a subdirectory that's named like datasets_clip-benchmark_wds_vtab-cifar10_resolve_main_test (for example)

[rom1504]

ok if it doesn't by default, it's good

[rom1504]

I'll test this

[rom1504]

ok so one thing here
not caused by your pr, but I really think we should put the main eval command in the readme at the beginning, not hidden in benchmark/ folder

[rom1504]

#76 a minor point, but quite nice for UX

[rom1504]

yeah this is much faster than the file based option

will run to the end and compare numbers, if all good will merge

[rom1504]

ok it does work, let's go