Skip to content

Latest commit

 

History

History
242 lines (189 loc) · 18.4 KB

README_EN.md

File metadata and controls

242 lines (189 loc) · 18.4 KB
Raw

XVERSE-65B

🤗 Hugging Face ｜ ModelScope ｜ 💬 WeChat

中文 | English

Update Information

  • [2024/03/25] Released the XVERSE-65B-Chat GGUF and GPTQ quantification models, supporting llama.cpp and vLLM inference of the XVERSE-65B-Chat model on MacOS/Linux/Windows systems.
  • [2023/12/14] Released the XVERSE-65B-Chat model, which is an aligned version of the XVERSE-65B base model.
  • [2023/12/08] Released the XVERSE-65B-2 base model. This model builds upon its predecessor through Continual Pre-Training, reaching a total training volume of 3.2 trillion tokens. It exhibits enhancements in all capabilities, particularly in mathematics and coding skills, with a 20% improvement on the GSM8K benchmark and a 41% increase on HumanEval.
  • [2023/11/29] Update model architecture and additional pre-training data information.
  • [2023/11/24] Update the related information of the pre-training data.
  • [2023/11/06] Released the XVERSE-65B base model.

Model Introduction

XVERSE-65B is a multilingual large language model, independently developed by Shenzhen Yuanxiang Technology. The models released this time is the base model XVERSE-65B. Its key features are as follows:

  • Model Structure: XVERSE-65B uses the mainstream Decoder-only Transformer network structure, supports 16k context length, which can meet the need of longer multi-round dialogues, knowledge question-answering, and summarization. This makes the model more versatile in application scenarios.
  • Training Data: The model has been thoroughly trained on a diversified and high-quality dataset consisting of 2.6 trillion of tokens, including more than 40 languages such as Chinese, English, Russian, and Spanish. The sampling ratio of different types of data is finely set, which makes the performance of Chinese and English excellent, and also takes into account the effect of other languages.
  • Tokenization: Based on the BPE (Byte-Pair Encoding) algorithm, a tokenizer with a vocabulary size of 100,534 has been trained using hundreds of gigabytes of language data. This tokenizer is capable of supporting multilingual without the need for additional vocabulary expansion.
  • Training Framework: The training utilizes FlashAttention2 for accelerated computation, and on top of 3D parallelism, virtual pipeline technology is applied to reduce the excessive bubble rate caused by longer pipelines and 16k context windows. This achieves a peak computational efficiency within the industry-leading range in the petaflop-scale cluster. Concurrently, through continuous optimization of cluster infrastructure operations, resource scheduling, training frameworks, and the scheduling platform, a highly stable, low-interruption, and robust fault-tolerant training system has been developed, enhancing the effective weekly training rate to 98.6%.

The models sizes, architectures and learning rate of XVERSE-65B are showed as follows:

params d_model n_heads n_layers d_ff learning rate
65B 8192 64 80 22016 1.5e−4

Introduction of Pre-training Data

During the pre-training phase, XVERSE-65B primarily utilized 7 different types of data. The following table shows a comparison of the pre-training datasets of XVERSE-65B with some other well-known models:

Data Type GPT31 Llama2 BLOOM3 PaLM4 Chinchilla5 Gopher6 MT-NLG7 XVERSE-65B
Web Pages Y Y Y Y Y Y Y Y
Code Y Y Y Y Y Y Y
Encyclopedia Y Y Y Y Y Y Y
Books Y Y Y Y Y Y Y
Academic Papers Y Y Y
QA Y Y Y Y Y

Note: 'Y' indicates that the data type was used.

The sampling ratios of different data types during the pre-training phase are as follows:

Web Pages Code Encyclopedia Books Academic Papers QA Other
Proportion (%) 72.91 7.09 4.81 5.62 6.55 1.15 1.87

During the pre-training phase, XVERSE-65B primarily used 41 kinds of natural language, and the following table shows the proportion of different languages in the pre-training data:

Language Proportion (%) Language Proportion (%) Language Proportion (%) Language Proportion (%) Language Proportion (%) Language Proportion (%)
en 54.91 pl 0.48 hu 0.19 ar 0.12 fa 0.07 sl 0.05
zh 31.09 it 0.36 ko 0.18 ro 0.11 hi 0.07 et 0.04
ja 3.22 pt 0.34 sv 0.15 bg 0.10 no 0.07 lv 0.03
ru 3.15 cs 0.27 el 0.14 th 0.10 ca 0.06 sr 0.03
de 1.52 uk 0.24 fi 0.14 da 0.09 iw 0.06 ta 0.03
es 0.91 tr 0.23 id 0.13 mr 0.08 lt 0.05 kk 0.02
fr 0.73 nl 0.20 vi 0.13 sk 0.08 ms 0.05

Note: Reference to the abbreviations of different languages: ISO_639-1

For the Code data, the following table shows the proportion of different programming languages:

Programming Language Proportion (%) Programming Language Proportion (%) Programming Language Proportion (%) Programming Language Proportion (%) Programming Language Proportion (%) Programming Language Proportion (%)
PHP 17.06 Go 3.38 Shell 0.74 PowerShell 0.23 Arduino 0.13 R 0.04
JavaScript 15.65 Rust 2.33 Haskell 0.46 Groovy 0.21 Assembly 0.13 ABAP 0.01
Java 15.18 Ruby 1.61 Common Lisp 0.43 Pascal 0.20 Clojure 0.12 COBOL 0.0022
Python 14.64 Swift 1.40 Perl 0.34 FORTRAN 0.19 Cuda 0.12 Verilog 0.0001
TypeScript 6.55 Kotlin 1.40 CSS 0.32 Elixir 0.17 VHDL 0.09
C 4.84 Scala 1.08 Julia 0.32 Solidity 0.16 Emacs Lisp 0.08
C++ 4.68 Dart 0.95 Visual Basic 0.25 F# 0.14 Objective-C++ 0.08
C# 3.44 SQL 0.76 OCaml 0.24 Erlang 0.14 Crystal 0.06

Model Evaluation

To comprehensively assess the performance of the model, we conducted extensive testing across a range of standard datasets, including C-Eval, CMMLU, Gaokao-Bench, MMLU, GAOKAO-English, AGIEval, RACE-M, CommonSenseQA, PIQA, GSM8K and HumanEval. These evaluations spanned multiple capabilities of the model, specifically including Chinese question answering, English question answering, language comprehension, common sense questioning, logical reasoning, mathematical problem-solving, and coding ability. The results of the evaluations are as follows:

Capability Dimension Dataset XVERSE-65B-2 XVERSE-65B Llama1-65B Llama2-70B Falcon-180B GPT-3.5 GPT-4
Chinese QA C-Eval 5-shot 72.4 68.6 38.8 49.9 54.2 54.4 68.7
CMMLU 5-shot 75.1 72.6 40.6 53.6 57.2 53.9 71.0
Gaokao-Bench1 5-shot 76.9 73.9 38.9 51.4 50.5 - -
English QA MMLU 5-shot 74.4 70.8 63.4 68.9 70.5 70.0 86.4
GAOKAO-English1 5-shot 86.6 85.3 67.0 76.6 63.3 - -
Chinese & English QA AGIEval1 5-shot 66.2 61.8 42.4 51.4 51.3 - -
Language Understanding RACE-M 0-shot 90.7 90.6 67.9 81.5 87.6 85.6 93.7
Common Sense QA CommonSenseQA 7-shot 81.1 79.8 74.0 78.5 82.4 80.2 88.3
Reasoning PIQA 0-shot 79.4 80.4 82.8 82.8 85.3 81.7 89.2
Math GSM8K 4-shot 72.6 60.3 50.9 56.8 62.6 57.1 92.0
Coding HumanEval 0-shot 37.8 26.8 23.7 29.9 - 48.1 67.0

1: Tests are conducted only on single-answer multiple-choice questions, thus excluding fill-in-the-blanks, open-ended questions, and multiple-answer multiple-choice questions.

For all the comparison models mentioned above, we prioritize the disclosure of their officially published results. In the absence of official data, we refer to the reported outcomes from OpenCompass Leaderboard. Results not covered by the aforementioned sources are derived from our own evaluation pipline.
For MMLU, we adopt the evaluation tools provided by the authors, C-Eval, AGIEval, GAOKAO-Bench, GAOKAO-English are the same as MMLU. For the remaining evaluation datasets, the OpenCompass is employed for evaluation.

Usage

Hardware requirements

The following table lists the hardware resources required for inference and fine-tuning on XVERSE-65B:

Type Kind Memory GPU
XVERSE-65B Training LoRA with ZeRO-3 1500GB 8*A800 80G
XVERSE-65B Inference BF16/FP16 500GB 2*A800 80G

Environment Setup

  1. Clone this repository:
git clone https://github.com/xverse-ai/XVERSE-65B
cd XVERSE-65B
  1. Install the dependencies using pip:
pip install -r requirements.txt

Loading with Transformers

The XVERSE-65B model can be loaded for inference using the following code:

>>> import torch
>>> from transformers import AutoTokenizer, AutoModelForCausalLM
>>> tokenizer = AutoTokenizer.from_pretrained("xverse/XVERSE-65B")
>>> model = AutoModelForCausalLM.from_pretrained("xverse/XVERSE-65B", trust_remote_code=True, torch_dtype=torch.bfloat16, device_map='auto')
>>> model = model.eval()
>>> inputs = tokenizer('北京的景点：故宫、天坛、万里长城等。\n深圳的景点：', return_tensors='pt').input_ids
>>> inputs = inputs.cuda()
>>> generated_ids = model.generate(inputs, max_new_tokens=64, eos_token_id=tokenizer.eos_token_id, repetition_penalty=1.1)
>>> print(tokenizer.batch_decode(generated_ids, skip_special_tokens=True))

Web Demo

The following code can be used to start a web server. By entering the access address in the browser, you can perform inference with the XVERSE-65B model:

python chat_demo.py --port='port' --model_path='/path/to/model/' --tokenizer_path='/path/to/tokenizer/'

Fine-tuning

XVERSE-65B allow developers to fine-tune for improved performance. Here, we attempted to use LLaMA-Factory for compatible fine-tuning training with XVERSE-65B, and tested it in an environment with 8 * Nvidia A800 80 GB + DeepSpeed. Below, we provide the fine-tuning method using LoRA with ZeRO-3.

Environment Setup

Download the LLaMA-Factory project and [install dependencies] (https://github.com/hiyouga/LLaMA-Factory#getting-started) as required.

Training

Training launch script:

Replace model_path with your own model path.

Both XVERSE-65B and XVERSE-65B-Chat are trained based on bfloat16. It is recommended to use bfloat16 for fine-tuning training.

deepspeed --num_gpus 8 src/train_bash.py \
    --deepspeed deepspeed.json \
    --stage sft \
    --model_name_or_path model_path  \
    --do_train \
    --dataset alpaca_gpt4_zh \
    --template default \
    --finetuning_type lora \
    --lora_target q_proj,v_proj \
    --output_dir  output_model_path \
    --overwrite_cache \
    --per_device_train_batch_size 4 \
    --gradient_accumulation_steps 4 \
    --lr_scheduler_type cosine \
    --logging_steps 1 \
    --save_steps 1000 \
    --learning_rate 5e-5 \
    --num_train_epochs 3.0 \
    --plot_loss \
    --bf16

deep_speed.json parameter settings：

{
    "train_micro_batch_size_per_gpu":"auto",
    "gradient_accumulation_steps":"auto",
    "gradient_clipping":"auto",
    "zero_allow_untested_optimizer":true,
    "fp16":{
        "enabled":false
    },
    "bfloat16":{
        "enabled":true
    },
    "zero_optimization":{
        "stage":3,
        "allgather_partitions":true,
        "reduce_scatter":true,
        "overlap_comm":false,
        "contiguous_gradients":true
    }
}

Limitations and Disclaimer

Like all other Large Language Models (LLMs), XVERSE-65B may produce inaccurate, biased, or otherwise offensive content under certain circumstances. Therefore, please use the content generated by the model with caution and refrain from disseminating harmful content. Before deploying any application of XVERSE-65B, developers should conduct safety tests and optimization of the model according to its specific application.

We strongly warn against the use of the XVERSE-65B model for producing or spreading harmful information, or conducting any activities that might harm the public, national, or social security, or violate regulations. We assume no responsibility for any problems arising from the use of the XVERSE-65B model, whether it be data security issues, public opinion risks, or any risks and issues caused by misunderstanding, misuse, dissemination, or non-compliance with the model.

Open Source License

The use of the source code in this repository must follow the Apache-2.0 open-source license, while the use of the model weights of XVERSE-65B needs to adhere to the Model License Agreement.

The XVERSE-65B model weights are fully open to academic research and support free commercial use. To apply for a commercial license, please fill in the application form. For other questions or collaborations, please contact [email protected].

Footnotes

  1. GPT3 Paper: Language Models are Few-Shot Learners ↩

  2. LLaMA Paper: LLaMA: Open and Efficient Foundation Language Models ↩

  3. BLOOM Paper: BLOOM: A 176B-Parameter Open-Access Multilingual Language Model ↩

  4. PaLM Paper: PaLM: Scaling Language Modeling with Pathways ↩

  5. Chinchilla Paper: Training Compute-Optimal Large Language Models ↩

  6. Gopher Paper: Scaling Language Models: Methods, Analysis & Insights from Training Gopher ↩

  7. MT-NLG Paper: Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model ↩