-
Notifications
You must be signed in to change notification settings - Fork 158
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Does vlmeval support multi card inference and batch size > 1? #32
Comments
Hi, @John-Ge ,
|
Thanks for your relpy! |
The authors of LLaVA have tried to create the beta-version of batch inference https://github.com/haotian-liu/LLaVA/issues/754 |
Hi, @darkpromise98 , we will try to include this feature into VLMEvalKit recently. |
That's great ! |
haotian-liu/LLaVA#754 (comment) BTW, I find sglang may not support lora+base model. I train llava with lora. If possible, I hope you could support load base model and merge lora weights and deploy it for evaluation. |
Hi, @John-Ge @darkpromise98 , I have reviewed the request. I'm sorry that I may not implement this feature on my own for the following reasons:
BTW, I'm willing to review and merge it VLMEvalKit main branch if someone is willing to create a PR (might be relatively heavy) about it. |
Does vlmeval support multi card inference and batch size > 1?
The text was updated successfully, but these errors were encountered: