Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[mieb] Task filtering by modality supported by models #1633

Merged
merged 5 commits into from
Dec 26, 2024

Conversation

isaac-chung
Copy link
Collaborator

@isaac-chung isaac-chung commented Dec 26, 2024

Fixes #1632

  • Added ModelMeta.modalities, which defaults to ["text"] to be backwards compatible. VLMs will have ["image","text"].
  • Added logic in MTEB.run for exact match between task and model modalities
  • Added modality model meta to relevant models

Here is the new log, showing that moco only supports images.

## Evaluating 1 tasks:
Any2AnyRetrieval
    - VidoreSyntheticDocQAAIRetrieval, t2i


INFO:mteb.evaluation.MTEB:

********************** Evaluating VidoreSyntheticDocQAAIRetrieval **********************
INFO:mteb.evaluation.MTEB:nyu-visionx/moco-v3-vit-b only supports ['image'], but the task modalities are ['text', 'image'].

Checklist

  • Run tests locally to make sure nothing is broken using make test.
  • Run the formatter to format the code using make lint.

cc @Muennighoff @gowitheflow-1998

@KennethEnevoldsen
Copy link
Contributor

I Would imagine the ideal message to be:

INFO:mteb.evaluation.MTEB:nyu-visionx/moco-v3-vit-b only supports the modalities ["images"], but the task category is t2i and include both the modalities ["text", "image"].

@isaac-chung
Copy link
Collaborator Author

@KennethEnevoldsen thanks but the PR was still in draft, and we had the same thought! It should be ready now.

@isaac-chung isaac-chung marked this pull request as ready for review December 26, 2024 10:44
@isaac-chung
Copy link
Collaborator Author

Thanks for the quick reviews folks. Merry Christmas! 🎄🎁

@isaac-chung isaac-chung merged commit 10ba68d into mieb Dec 26, 2024
10 checks passed
@isaac-chung isaac-chung deleted the task-filtering-by-modality-for-models branch December 26, 2024 12:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants