The nm-vllm packages published in this repository are Neural Magic Enterprise Editions of vLLM. Packages are versioned Python wheels and Docker images. These are released as "production level" official releases and "beta level" nightly releases.
Official releases are made at the discretion of Neural Magic, but typically track with vllm releases. These wheels are available via the official PyPI as well as Neural Magic's PyPI.
Nightly builds are released every night given green runs in automation. The wheels are available at Neural Magic's PyPI.
Please see how we are doing with our benchmark results here.
The nm-vllm PyPI package includes pre-compiled binaries for CUDA (version 12.1) kernels. For other PyTorch or CUDA versions, please compile the package from source.
Install it using pip:
pip install nm-vllm --extra-index-url https://pypi.neuralmagic.com/simpleTo utilize the weight sparsity features, include the optional sparse dependencies.
pip install nm-vllm[sparse] --extra-index-url https://pypi.neuralmagic.com/simpleThe nm-vllm-ent container registry includes premade docker images.
Launch the OpenAI-compatible server with:
MODEL_ID=Qwen/Qwen2-0.5B-Instruct
docker run --gpus all --shm-size 2g ghcr.io/neuralmagic/nm-vllm-ent:latest --model $MODEL_IDNeural Magic maintains a variety of optimized models on our Hugging Face organization profiles: