v0.7.0: LLaVA Multimodal LLM Support

Congratulations on 20k stars 🎉 We are the 1st of the GitHub Trending at Apr. 23rd 🔥 Follow us at X

New features

• Support SFT/PPO/DPO/ORPO for the LLaVA-1.5 model by @BUAADreamer in #3450
• Support inferring the LLaVA-1.5 model with both native Transformers and vLLM by @hiyouga in #3454
• Support vLLM+LoRA inference for partial models (see support list)
• Support 2x faster generation of the QLoRA model based on UnslothAI's optimization
• Support adding new special tokens to the tokenizer via the new_special_tokens argument
• Support choosing the device to merge LoRA in LlamaBoard via the export_device argument
• Add a Colab notebook for getting into fine-tuning the Llama-3 model on a free T4 GPU
• Automatically enable SDPA attention and fast tokenizer for higher performance

New models

• Base models
  • OLMo-1.7-7B
  • Jamba-v0.1-51B
  • Qwen1.5-110B
  • DBRX-132B-Base
• Instruct/Chat models
  • Phi-3-mini-3.8B-instruct (4k/128k)
  • LLaVA-1.5-7B
  • LLaVA-1.5-13B
  • Qwen1.5-110B-Chat
  • DBRX-132B-Instruct

New datasets

• Supervised fine-tuning datasets
  • LLaVA mixed (en&zh) by @BUAADreamer in #3471
• Preference datasets
  • DPO mixed (en&zh) by @hiyouga

Bug fix

• Fix #2093 #3333 #3347 #3374 #3387