Can we use any CausalLM model from HuggingFace in 4bit in Spark-NLP?

[AayushSameerShah]

🙋🏻‍♂️ Hie there!

JSL has an amazing collection of models, but for me I am willing to use instruction tuned models like mosaicml/mpt-7b-instruct which can understand the instructions and perform appropriately.

😕 The issue

In our program, we are willing to use the less amount of resources as possible. Thus, if I can use any model in 4bit quantization by using this library, this would be wonderful.

Will you please help me through this?
Thank you soo much 🙏🏻

[maziyarpanahi]

Hi,

Not currently, we only have TensorFlow engine and it's not compatible with 4-bit. We also don't have any annotators that accept CasualLM like Falcon or MPT.

We are working on adding ONNX to the mix and after that we can start adding support for CasualLM.