[LLM] add deploy server #9581
Open
[LLM] add deploy server #9581
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for your contribution! |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## develop #9581 +/- ##
===========================================
+ Coverage 52.97% 52.98% +0.01%
===========================================
Files 703 703
Lines 110981 110982 +1
===========================================
+ Hits 58788 58809 +21
+ Misses 52193 52173 -20 ☔ View full report in Codecov by Sentry. |
ZHUI
reviewed
Dec 9, 2024
|
|
||
| <h1 align="center"><b><em>大模型服务化部署</em></b></h1> | ||
|
|
||
| *该部署工具是基于英伟达Triton框架专为服务器场景的大模型服务化部署而设计。它提供了支持gRPC、HTTP协议的服务接口，以及流式Token输出能力。底层推理引擎支持连续批处理、weight only int8、后训练量化（PTQ）等加速优化策略，为用户带来易用且高性能的部署体验。* |
| *该部署工具是基于英伟达Triton框架专为服务器场景的大模型服务化部署而设计。它提供了支持gRPC、HTTP协议的服务接口，以及流式Token输出能力。底层推理引擎支持连续批处理、weight only int8、后训练量化（PTQ）等加速优化策略，为用户带来易用且高性能的部署体验。* | ||
|
|
||
| # 快速开始 | ||
|
|
| # 下载模型 | ||
| wget https://paddle-qa.bj.bcebos.com/inference_model/Meta-Llama-3-8B-Instruct-A8W8C8.tar | ||
| mkdir Llama-3-8B-A8W8C8 && tar -xf Meta-Llama-3-8B-Instruct-A8W8C8.tar -C Llama-3-8B-A8W8C8 | ||
|
|
|
|
||
|
|
||
| ``` | ||
| from fastdeploy_client.chatbot import ChatBot |
There was a problem hiding this comment.
emm，还是需要fd吗？fd的安装会不会有问题，版本要求，兼容性要求呢？
ZHUI
reviewed
Dec 10, 2024
|
|
||
|
|
||
| ``` | ||
| from fastdeploy_client.chatbot import ChatBot |
| REQUIRED_PACKAGES = fin.read() | ||
|
|
||
| setuptools.setup( | ||
| name="fastdeploy-client", |
There was a problem hiding this comment.
可以换一个 paddle 或者 paddlenlp 之类名字？
ZHUI
reviewed
Dec 10, 2024
|
|
||
| docker run --gpus all --shm-size 5G --network=host --privileged --cap-add=SYS_PTRACE \ | ||
| -v ${MODEL_PATH}:/models/ \ | ||
| -dit registry.baidubce.com/paddlepaddle/fastdeploy:llm-serving-cuda123-cudnn9-v1.2 \ |
There was a problem hiding this comment.
能否去除掉特定docker的依赖。依赖特定docker的话，会使得易用性降低。
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
New features
PR changes
Others
Description
add llm deploy server