v2.2.1

English Version

New Features

1. Multimodal: Supported a large number of multimodal datasets and restructured the multimodal architecture. Some models now support grounding tasks.
2. Web-ui: Added support for RLHF, evaluation, and quantization.
3. Evaluation Functionality: Refactored the evaluation functionality, now using OpenCompass internally, supporting over 50 evaluation datasets.
4. Deployment Functionality: VLLM infer_backend now supports multimodal models.
5. Agent Training: Refactored the construction, training, and deployment of agent datasets, making the agent pipeline more complete.
6. Human Alignment: Added alignment algorithms such as KTO and CPO, and refactored the human alignment code.

New Models

1. openbuddy-llama3-70b
2. Deepseek-coder-v2
3. llava1.5, llava1.6, llava-next-video
4. gemma2
5. Florence
6. phi3-4k
7. internlm2.5, xcomposer2.5
8. internvl2
9. codegeex4
10. mistral-7b-instruct-v0.3

New Datasets

1. Over 30 foundational multimodal datasets, including GQA, RefCOCO, and Llava-pretrain.
2. Swift-mix general mixed dataset.
3. Video-chatgpt video dataset.

中文版本

新功能

1. 多模态：支持了非常多的多模态数据集，并重构了多模态架构，部分模型开始支持grounding任务
2. Web-ui：支持了RLHF、评测和量化
3. 评测功能：进行了重构，内部使用了OpenCompass，支持50+评测集
4. 部署功能：VLLM infer_backend支持多模态模型
5. Agent训练：重构了Agent数据集构造、训练、部署，Agent链路更加完整
6. 人类对齐：增加了KTO、CPO等对齐算法，并重构了人类对齐的代码

新模型

1. openbuddy-llama3-70b
2. Deepseek-coder-v2
3. llava1.5, llava1.6, llava-next-video
4. gemma2
5. Florence
6. phi3-4k
7. internlm2.5, xcomposer2.5
8. internvl2
9. codegeex4
10. mistral-7b-instruct-v0.3

新数据集

1. GQA、RefCOCO、Llava-pretrain等30+多模态基础数据集
2. swift-mix通用混合数据集
3. video-chatgpt视频数据集

What's Changed

• fix py38 by @Jintao-Huang in #1152
• refactor rlhf by @hjh0119 in #1090
• Fix/web UI 0617 by @tastelikefeet in #1158
• Add more datasets by @tastelikefeet in #1065
• support openbuddy-llama3-70b by @Jintao-Huang in #1162
• Add Deepseek coder v2 model by @tastelikefeet in #1164
• Update web-ui labels by @tastelikefeet in #1168
• fix hf download by @tastelikefeet in #1169
• fix kto bugs by @hjh0119 in #1173
• Fix preprocessor by @tastelikefeet in #1174
• refactor ui by @tastelikefeet in #1178
• fix DPO by @hjh0119 in #1181
• fix bugs by @Jintao-Huang in #1171
• 在argument.py添加hccl参数，用于NPU环境下训练使用 by @jiaozhentian in #1165
• Custom agent dataset training support by @hjh0119 in #1184
• RLHF UI by @tastelikefeet in #1182
• fix kto by @hjh0119 in #1187
• update agent doc by @hjh0119 in #1189
• Fix device map in limited mem by @tastelikefeet in #1190
• Fix sk will be uploaded by @tastelikefeet in #1192
• Fix dataset concatenation by @tastelikefeet in #1193
• Fix glm4v dataloader by @CjangCjengh in #1183
• fix glm4v images by @Jintao-Huang in #1194
• [TorchAcc] Cache the compiled results and remove some xla flags by @anw90 in #1160
• Fix vqa dataset loading by @tastelikefeet in #1195
• [TorchAcc] Add acc_steps arguments to speed up torchacc training by @anw90 in #1196
• fix minicpm-v data_collator by @Jintao-Huang in #1197
• Add a tuner to train a part of the model by @tastelikefeet in #1200
• Fix ConversationPreprocessor loading by @tastelikefeet in #1201
• Fix map cache on multi node by @tastelikefeet in #1203
• refactor multi-modal by @Jintao-Huang in #1212
• support llava 1.5 by @Jintao-Huang in #1217
• add in_browswer by @tastelikefeet in #1220
• [TorchAcc] Add USE_TORCH_XLA=0 flag for native swift scripts by @anw90 in #1221
• Fix glm4v batch_size>1 by @Jintao-Huang in #1223
• Add debug log support by @tastelikefeet in #1226
• Fix qlora deploy by @Jintao-Huang in #1224
• Add new dataset by @tastelikefeet in #1227
• fix refcoco by @tastelikefeet in #1232
• Fix bugs by @tastelikefeet in #1241
• fix bugs by @Jintao-Huang in #1242
• refactor inference by @Jintao-Huang in #1245
• Add more datasets by @tastelikefeet in #1246
• Support gemma2 by @Jintao-Huang in #1247
• Support gemma2 by @Jintao-Huang in #1248
• add push to hub tracker by @tastelikefeet in #1249
• support Florence series model by @hjh0119 in #1244
• update eval by @Jintao-Huang in #1251
• reset snapshot_download to modelscope by @tastelikefeet in #1254
• Feat/open compass by @tastelikefeet in #1213
• fix output empty will be filtered by @tastelikefeet in #1256
• add phi3 models by @tastelikefeet in #1259
• update dataset preprocess by @Jintao-Huang in #1257
• fix rlhf by @tastelikefeet in #1261
• fix rlhf by @Jintao-Huang in #1260
• Support llava1.6 mistral by @Jintao-Huang in #1268
• Add new dataset by @tastelikefeet in #1270
• support vllm & multimodal by @Jintao-Huang in #1229
• Fix bugs by @Jintao-Huang in #1275
• fix llamapro by @tastelikefeet in #1276
• fix xcomposer deploy by @Jintao-Huang in #1280
• Fix/0703 2 by @tastelikefeet in #1284
• Fix llamapro merge and quantized training by @tastelikefeet in #1289
• update lr_scheduler by @Jintao-Huang in #1288
• support internlm2.5 by @Jintao-Huang in #1291
• fix vllm by @Jintao-Huang in #1292
• fix florence & deploy by @Jintao-Huang in #1293
• fix mini-internvl-phi3 template by @Jintao-Huang in #1298
• fix florence by @hjh0119 in #1301
• Refactor UI and fix some bugs by @tastelikefeet in #1300
• fix dependency by @tastelikefeet in #1306
• support codegeex4 by @Jintao-Huang in #1305
• support internvl2 by @hjh0119 in #1304
• support llava video by @Jintao-Huang in #1307
• fix docs by @Jintao-Huang in #1309
• support lr_scheduler_kwargs by @Jintao-Huang in #1310
• Fix internvl2 template by @hjh0119 in #1308
• Fix bugs by @tastelikefeet in #1311
• support warmup_stable_decay by @Jintao-Huang in #1312
• Support xcomposer2.5 by @Jintao-Huang in #1287
• Fix bugs by @tastelikefeet in #1319

New Contributors

• @jiaozhentian made their first contribution in #1165
• @CjangCjengh made their first contribution in #1183
• @anw90 made their first contribution in #1160

Full Changelog: v2.1.1...v2.2.1