How to make quantised models work faster on CPU machines

[TheRabidWolverine]

The assumption is that quantised models are better suited on CPU machines than their non-quantised counterparts. So what governs the performance of quantized versions of the embeddings (both during embedding generation as well as runtime inferencing) on these CPU machines? Number of CPU cores? Or RAM? Or both? Will a machine with more cores be able to parallel process higher number of queries with good performance, or does it require RAM as well to do that? Please advise.

[NirantK]

Hey @TheRabidWolverine, couple of things:

1. It's not always necessary that quantization works better for CPU only. One can quantize models for CUDA or Apple runtimes too. It's just that we've chosen to prefer CPU since we had difficulty setting up tests for CUDA and Apple runtimes.
2. What governs the speed gain — performance? Primarily, the model size. Quantized models do two things: It's less operations and hence the model size is smaller
3. FastEmbed does use more CPU processes — that's because for larger datasets, we do data parallel processing. That can have a RAM impact as well, since we load more data into RAM for parallel processing.

I hope this answers your questions? Please feel free to ask more follow up questions!

[TheRabidWolverine]

Thanks Nirant. Do more cores necessarily make the performance faster?Sent from my iPhoneOn 23-Feb-2024, at 10:35 AM, Nirant ***@***.***> wrote: Closed #132 as resolved. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>