-
Notifications
You must be signed in to change notification settings - Fork 82
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Overwritten zeroshot templates for webdataset eval #109
Comments
Thanks a lot @djghosh13 ! I haven't noticed this, I think it would make sense to overwrite only if custom option is provided (second option). Will do a PR mentioning the issue in datacomp as well. So basically order of precedence would be: 1 - templates from The other thing I am thinking of, is to (optionally) write the classnames/templates that ended up beging used in the JSON dump to have full transparency. |
* fix overwritten zeroshot templates issue #109. Thanks to @djghosh13. * handle non classification/linear probe case * comments * simplify * support dumping the classnames and templates that are used for evaluation * fix tests * just comments * just comments
Fixed. |
#98 added better support for custom zeroshot prompt templates. However, the exact implementation results in behavior which conflicts with webdataset datasets (e.g. "wds/cifar10") stored on Huggingface or locally. Specifically:
en_zeroshot_classification_templates.json
build_wds_dataset
loads classnames and templates from the source (local filesystem or Huggingface repo) (line 731)ds.templates = templates
overwrites the templates on line 45This affects all wds datasets, though it probably hasn't changed results in the majority of cases, where the dataset name and templates match the defaults in
en_zeroshot_classification_templates.json
.A fix should probably either:
ds.templates
if it already exists; ords.templates
only if thecustom_template_file
parameter is set (i.e. custom template takes precedence over webdataset template)The text was updated successfully, but these errors were encountered: