Skip to content

Latest commit

 

History

History
152 lines (112 loc) · 7.04 KB

README.md

File metadata and controls

152 lines (112 loc) · 7.04 KB
Raw

CatVision

🤗 models

Introduction

A multimodal large-scale model, characterized by its open-source nature, closely emulates the functionalities of the GPT4V/Qwen-VL-Plus model. Built upon the foundation of Qwen-72b-Chat, CatVision in handling inputs that combine both images and text. This model is designed to effectively follow instructions for output formats, benefiting from the strengths of Qwen72b.

一个开源多模态大模型，紧密模拟了GPT4V/Qwen-VL-PLUS系列模型的功能。该模型建立在Qwen-72b-Chat的基础上，可以处理包含交错的图文输入。该模型从Qwen72b的优势中受益，旨在有效地遵循输出格式指令。

Our model performs close to the closed-source Qwen-VL-PLUS on many datasets and significantly surpasses the performance of the open-source model Qwen-VL-7B-Chat.

我们的模型在很多数据集上，接近闭源的Qwen-VL-PLUS的效果，并大幅超过开源模型Qwen-VL-7B-Chat的效果。

  • Our training approach consisted of two stages, inspired by LLava1.5. In the initial stage, we trained the visual encoder + perceptual resampler, and in the second stage, we focused on training the large language model + perceptual resampler with instructional data. To overcome limited computational resources (32xA100-80G), we used Lora for training in both stages.

受LLava1.5启发，我们的训练分为两个阶段：在初始阶段，我们训练了视觉编码器+感知重采样器；在第二阶段，我们专注于使用视觉指令数据训练大型语言模型+感知重采样器。为了克服有限的计算资源(32xA100-80G)，我们在两个阶段都使用了Lora进行培训。

  • During the first stage, our training data included samples from ShareGPT4V and CC12M. As we progressed to the second stage, our training dataset encompassed ShareGPT4V fine-tune data, LVIS Instruct4V, OCR data, InforGraphics/Chart QA data, and data sourced from region descriptions in VG.

在第一阶段，我们的训练数据包括来自ShareGPT4V和CC12M的样本。第二阶段，我们的训练数据集包括ShareGPT4V微调数据、LVIS Instruct4V、OCR数据、信息图表问答数据以及从VG区域描述中获取的数据。

  • The visual encoding part is inherited from Qwen-VL-Chat, i.e., Openclip ViT-bigG.

视觉编码部分继承自Qwen-VL-Chat，即Openclip ViT-bigG。

  • We are continuously collecting instruction data, optimizing the model, and looking forward to supporting more tasks.

我们正在持续收集指令数据，优化模型，期待能支持更多的功能。

Quick Start

from transformers import AutoTokenizer, AutoModelForCausalLM, AutoConfig
tokenizer = AutoTokenizer.from_pretrained(
    pretrained_model_name_or_path="huizhang0110/CatVision",
    model_max_length=8192,
    padding_side="left",
    trust_remote_code=True
)
config = AutoConfig.from_pretrained(
    pretrained_model_name_or_path="huizhang0110/CatVision",
    trust_remote_code=True
)
model = AutoModelForCausalLM.from_pretrained(
    pretrained_model_name_or_path="huizhang0110/CatVision",
    config=config,
    device_map="auto", 
    trust_remote_code=True
).eval()
query = "<img>demo.jpg</img>\n介绍一下这张图像！"
response, history = model.chat(
    tokenizer,
    query=query,
    history=None,
)

Benchmark

Our model achieved favorable results on the many leaderboards.

Model Val (900) Test (11K)
Gemini Ultra 59.4 ----
GPT4V 56.8 55.7
Gemini Pro 47.9 ----
Yi-VL-34B 45.9 41.6
Qwen-VL-PLUS 45.2 40.8
CatVision 45.9 40.1
Macro-VL 41.2 40.4
InfiMM-Zephyr-7B 39.4 35.5
Yi-VL-6B 39.1 37.8
SVIT 38.0 34.1
LLaVA-1.5-13B 36.4 33.6
Emu2-Chat 36.3 34.1
Qwen-VL-7B-Chat 35.9 32.9
Model Val (900) Test (11K)
GPT-4V(ision) (Playground) 42.5 43.7
Qwen-VL-PLUS* 39.5 36.8
CatVision 39.6 ----
Yi-VL-34B 36.2 36.5
Yi-VL-6B 35.8 35.0
Qwen-VL-7B-Chat 30.7 31.3
InternVL-Chat-ViT-6B-Vicuna-7B 26.4 26.7
InternVL-Chat-ViT-6B-Vicuna-13B 27.4 26.1
CogAgent-Chat 24.6 23.6
Emu2-Chat 23.8 24.5
Chinese-LLaVA 25.5 23.4
VisCPM 25.2 22.7
mPLUG-OWL2 20.8 22.2
Frequent Choice 24.1 26.0
Random Choice 21.6 21.6
Model mmbench_cn (test) mmbench_cn (dev) mmbench_en (test) mmbench_zh (dev) ccbench
Qwen-VL-PLUS(BASE) 83.3 83.2 82.7 81.5 77.6
GPT4v 77.0 75.1 74.4 75.0 46.5
Qwen-VL-PLUS 67.0 66.2 70.7 69.6 55.1
CatVision 70.9 71.8 70.2 71.6 49.8
Qwen-VL-Chat 61.8 60.6 56.3 56.7 41.2
Model Perception Cognition
GPT4v 1409.43 517.14
Qwen-VL-PLUS 1681.25 502.14
CatVision 1560.90 366.43
Qwen-VL-Chat 1487.57 360.71
  • Open Compress

wait

  • Show Case

图像描述

图像描述

信息图表

图像问答

区域理解

区域理解

Citation

@misc{CatVision,
  author = {[email protected]},
  title = {CatVision},
  year = {2024},
  publisher = {huggingface},
  howpublished = {\url{https://huggingface.co/huizhang0110/CatVision}}
}

logo