Skip to content

Issues: EvolvingLMMs-Lab/lmms-eval

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

138 Open 63 Closed
138 Open 63 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Failure with Running vqav2_test on llava-v1.5-7b Model
#413 opened Nov 17, 2024 by AdrianHuang2002
Documentation for used metrics ?
#412 opened Nov 16, 2024 by Mohamed-Dhouib
About Multi-Image Inference
#410 opened Nov 15, 2024 by jun0wanan
[Bug] --cache_requests failed to deserialize cached requests
#405 opened Nov 12, 2024 by nanocm
1
mvbench-missing videos
#403 opened Nov 7, 2024 by orrzohar
2
Evaluating LLaVA OneVision 72B - memory, speedup, multinode
#400 opened Nov 6, 2024 by orrzohar
4
LLaVA_OneVision class lacks parameter "mm_newline_position"
#395 opened Nov 5, 2024 by Richar-Du
1
Got wrong answer when test nextqa/longvideobench with llava_vid
#393 opened Nov 4, 2024 by wanxinzzz
3
Error when testing QwenVL2: 'NoneType' object is not subscriptable
#388 opened Nov 1, 2024 by Mohamed-Dhouib
5
llava_hf inference is extremely slow
#387 opened Nov 1, 2024 by luomancs
1
Evaluation speed
#385 opened Nov 1, 2024 by si0wang
1
Error running LLava-next
#384 opened Oct 31, 2024 by ZihaoZheng98
1
Unable to Conduct Video Evaluation on InternVL2
#378 opened Oct 29, 2024 by helloworld01001
3
Evaluation Performance on LLaVA-Lora
#375 opened Oct 28, 2024 by XindiWu
1
I can't reproduce many original performances for Qwen_VL. Who can help me?
#374 opened Oct 28, 2024 by xiaochengsky
2 tasks
2
[Function Behavior] loglikelihood Function Returns Loss Instead of Log Likelihood for Multiple Choice Selection
#370 opened Oct 28, 2024 by j1u2l3i4a5n
@kcz358
1
[Bug] Wrong results for many datasets due to a bug in the evaluation of tasks with multiple targets (affecting tasks like docvqa and metrics like anls) bug Something isn't working
#363 opened Oct 27, 2024 by elronbandel
@pufanyi
3
Question Regarding "N/A" Output for longvideobench_test_i in lmms_eval
#361 opened Oct 27, 2024 by kusen-okawa
1
llama_hf on pope benchmark gives zero results.
#356 opened Oct 26, 2024 by giobin
3
Weights and Biases Integration
#351 opened Oct 25, 2024 by arman-kazemi
2
add COIN?
#348 opened Oct 25, 2024 by hasanar1f
RUN - ERROR - Dataset A-OKVQA is not valid, will be skipped.
#339 opened Oct 22, 2024 by max-yue
Bug: Unable to calculate metrics from saved predictions using --predict_only and --from_log bug Something isn't working help wanted Extra attention is needed
#337 opened Oct 21, 2024 by zjwu0522
1
VideoMME CUDA out of memory
#336 opened Oct 21, 2024 by yhy-2000
2
[Feature Request] Update Datasets Version, so that lmms-eval can be used in Offline Environment enhancement New feature or request
#335 opened Oct 18, 2024 by jungle-gym-ac
4
ProTip! Exclude everything labeled bug with -label:bug.