README_en.md

mnn-llm

Chinese

Example Projects

• cli: Compile using the command line, for Android compilation refer toandroid_build.sh
• web: Compile using the command line, runtime requires specifyingweb resources
• android: Open with Android Studio for compilation; APK download:
• ios: Open with Xcode for compilation; 🚀🚀🚀This sample code is 100% generated by ChatGPT🚀🚀🚀
• other: Added capabilities for text embedding, vector querying, document parsing, memory bank, and knowledge base 🔥.

模型支持

For exporting the llm model to ONNX or mnn, please usellm-export

Current supported models：

model	onnx-fp32	mnn-quant
chatglm-6b
chatglm2-6b
chatglm3-6b
codegeex2-6b
Qwen-7B-Chat
Baichuan2-7B-Chat
Llama-2-7b-chat
internlm-chat-7b
Yi-6B-Chat
deepseek-llm-7b-chat
Qwen-1.8B-Chat
phi-2
bge-large-zh
TinyLlama-1.1B-Chat
Qwen1.5-0.5B-Chat
Qwen1.5-1.8B-Chat
Qwen1.5-4B-Chat
Qwen1.5-7B-Chat

Other version:

• Qwen-1_8B-Chat-int8：

Performance

CPU 4-thread speed: prefill / decode tok/s

model	android(f16/32)	macos (f32)	linux (f32)	windows (f32)
qwen-1.8b-int4	100.21 / 22.22	84.85 / 19.93	151.00 / 35.89	117.30 / 33.40
qwen-1.8b-int8	99.95 / 16.94	67.70 / 13.45	118.51 / 24.90	97.19 / 22.76
chatglm-6b-int4	17.37 / 6.69	19.79 / 6.10	34.05 / 10.82	30.73 / 10.63
chatglm2-6b-int4	26.41 / 8.21	20.78 / 6.70	36.99 / 11.50	33.25 / 11.47
chatglm3-6b-int4	26.24 / 7.94	19.67 / 6.67	37.33 / 11.92	33.61 / 11.21
qwen-7b-int4	14.60 / 6.96	19.79 / 6.06	33.55 / 10.20	29.05 / 9.62
baichuan2-7b-int4	13.87 / 6.08	17.21 / 6.10	30.11 / 10.87	26.31 / 9.84
llama-2-7b-int4	17.98 / 5.17	19.72 / 5.06	34.47 / 9.29	28.66 / 8.90

Tested system and device information is as follows

os	device	CPU	Memory
android	XiaoMi12	Snapdragon 8gen1	8 GB
macos	MacBook Pro 2019	Intel(R) Core(TM) i7-9750H	16 GB
linux	PC	Intel(R) Core(TM) i7-13700K	32GB
windows	PC	Intel(R) Core(TM) i7-13700K	32GB

Downloading INT4 Models

# <model> like `chatglm-6b`
# linux/macos
./script/download_model.sh <model>

# windows
./script/download_model.ps1 <model>

Building

Current build status:

System	Build Statud
Linux
Macos
Windows
Android

Local Compilation

# linux
./script/build.sh

# macos
./script/build.sh

# windows msvc
./script/build.ps1

# android
./script/android_build.sh

The default backend used is CPU. If you want to use a different backend, you can add a MNN compilation macro within the script:

• cuda: -DMNN_CUDA=ON
• opencl: -DMNN_OPENCL=ON

4. Execution

# linux/macos
./cli_demo qwen-1.8b-int4 # cli demo
./web_demo qwen-1.8b-int4 ../web # web ui demo

# windows
.\Debug\cli_demo.exe qwen-1.8b-int4
.\Debug\web_demo.exe qwen-1.8b-int4 ../web

# android
adb push libs/*.so build/libllm.so build/cli_demo /data/local/tmp
adb push model_dir /data/local/tmp
adb shell "cd /data/local/tmp && export LD_LIBRARY_PATH=. && ./cli_demo qwen-1.8b-int4"

Reference

• chatglm-6b
• chatglm2-6b
• chatglm3-6b
• codegeex2-6b
• Baichuan2-7B-Chat
• Qwen-7B-Chat
• Qwen-VL-Chat
• Qwen-1.8B-Chat
• Llama-2-7b-chat-ms
• internlm-chat-7b
• phi-2
• bge-large-zh
• TinyLlama-1.1B-Chat-v0.6
• Yi-6B-Chat
• Qwen1.5-0.5B-Chat
• Qwen1.5-1.8B-Chat
• Qwen1.5-4B-Chat
• Qwen1.5-7B-Chat
• cpp-httplib
• chatgpt-web
• ChatViewDemo
• nlohmann/json