Skip to content

Issues: EvolvingLMMs-Lab/lmms-eval

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

102 Open 52 Closed
102 Open 52 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Calculate metrics after saving output
#288 opened Sep 29, 2024 by Mohamed-Dhouib
Try on qwen2vl-72b, out of memory
#285 opened Sep 28, 2024 by ChinChyi
Bug on Qwen2-VL over videomme bug Something isn't working
#283 opened Sep 27, 2024 by ThisisBillhe
2
Bug on Qwen2-VL
#282 opened Sep 25, 2024 by yusalei
2
Batch GPT is not working help wanted Extra attention is needed
#279 opened Sep 24, 2024 by NimrodShabtay
2
[new model] qwen2-vl support
#264 opened Sep 19, 2024 by jxgu1016
3
How to set model precision when evaluating and what's the default precision? Such as "fp16" or "loat_in_8bit"
#263 opened Sep 18, 2024 by transcend-0
1
Issue of GPT4o evaluation with dense frame sampling strategy
#256 opened Sep 15, 2024 by HYUNJS
video-mme (short/medium/long) enhancement New feature or request
#254 opened Sep 14, 2024 by orrzohar
@choiszt
5
[Result] Can you provide some results on benchmarks?
#248 opened Sep 13, 2024 by Chenfeng1271
4
[BUG] Commit #0d02bad may break srt_api by mis-use of asyncio
#243 opened Sep 11, 2024 by Maxwell-Lyu
2 tasks done
4
No llavavid module in the latest LLaVA-NeXT
#242 opened Sep 11, 2024 by YitaoLiu1996
2
Tasks were not found: activitynetqa
#239 opened Sep 10, 2024 by jby20180901
9
Encountering task missing issue even when specifying task from list mentioned at current_tasks.md
#233 opened Sep 6, 2024 by rodosingh
task_manager.list_all_tasks() is not showing outputs properly.
#232 opened Sep 6, 2024 by rodosingh
Cannot find a valid ref in kc/worldqa
#230 opened Sep 6, 2024 by tychen-SJTU
2 tasks
1
[logging] The program is not responding during multi-gpu evaluation
#222 opened Sep 3, 2024 by ssmisya
1
AttributeError: 'LlavaQwenConfig' object has no attribute 'mm_newline_position'
#219 opened Sep 2, 2024 by linxid
2 tasks
name 'get_model_name_from_path' is not defined
#214 opened Aug 27, 2024 by jby20180901
2
LiveBench doesn't work.
#211 opened Aug 26, 2024 by jyC23333
question about RefCOCO|+|g. Is it a region caption task?
#210 opened Aug 25, 2024 by yuecao0119
1
PiPy / Github version number mismatch
#208 opened Aug 24, 2024 by tyler-romero
[Request] Loosen dependency version constraints, such as for transformers
#207 opened Aug 24, 2024 by tyler-romero
Wrongly treating mvbench as an image task
#205 opened Aug 23, 2024 by hulianyuyy
3
Llava-v1.5 and Llava-v1.5-hf Performes on MMMU differs a lot!
#203 opened Aug 22, 2024 by hxhcreate
1
ProTip! Add no:assignee to see everything that’s not assigned.