How can I run the realtime model locally on my linux machine?

[CyberTimon]

Hello

Ultravox looks really interesting.

Is it possible to run this model on my own linux machine, which has 2x RTX 3090 24GB GPU's?
I couldn't find a python inference server or something similar.

Thank you!
Kind regards,
Timon

[johnwick123f]

@CyberTimon I believe vllm and transformers have support for ultravox but vllm is faster. The vllm script to run ultravox.

https://github.com/vllm-project/vllm/blob/661a34fd4fdd700a29b2db758e23e4e243e7ff18/examples/offline_inference_audio_language.py#L23