New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

VLM是否支持并行推理？ #1330

Open

youjiaSHTU opened this issue Jul 9, 2024 · 1 comment

youjiaSHTU commented Jul 9, 2024

Describe the feature
Please describe the feature requested here(请在这里描述需求)

目前infer的方式和deploy部署，都是串行的。是否有某种方式可以实现并行对多条数据进行同时处理（为了加快速度）。
例如，我需要使用一个自己训练的模型（minicpm-v2.5-chat），对大约一万条文字提问+每个提问对应的一张图片进行推理并保存结果，最快的方式是什么？

gjd2017 commented Jul 16, 2024

同问，多模态大模型批量推理

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment