Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

document how to run retrieval and generative eval using wds #91

Closed
rom1504 opened this issue Apr 26, 2023 · 3 comments
Closed

document how to run retrieval and generative eval using wds #91

rom1504 opened this issue Apr 26, 2023 · 3 comments

Comments

@rom1504
Copy link
Contributor

rom1504 commented Apr 26, 2023

@rom1504
Copy link
Contributor Author

rom1504 commented Apr 26, 2023

maybe something like clip_benchmark eval --dataset=mscoco_captions --dataset_root="https://huggingface.co/datasets/clip-benchmark/wds_{dataset_cleaned}/tree/main" --task=mscoco_generative --model=coca_ViT-L-14 --output=result.json --batch_size=256 --pretrained=model.pt

@mehdidc
Copy link
Collaborator

mehdidc commented Jul 7, 2023

Yes the following works fine for retrieval:

clip_benchmark eval --dataset=wds/mscoco_captions --dataset_root="https://huggingface.co/datasets/clip-benchmark/wds_{dataset_cleaned}/tree/main" --task=zeroshot_retrieval --model=ViT-B-32 --output=result.json

for generative something similar with task mscoco_generative does not seem to work, will fix it in a PR and add docs for both.

@mehdidc
Copy link
Collaborator

mehdidc commented Jul 7, 2023

Done

@mehdidc mehdidc closed this as completed Jul 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants