minicpm-v-v2_5-chat 微调显存溢出

[lyc728]

训练脚本

--model_type minicpm-v-v2_5-chat \
    --model_id_or_path /data/MiniCPM-V/pretrained/MiniCPM-Llama3-V-2_5 \
    --dataset /data/swift/finetune/train_0703.jsonl \
    --ddp_find_unused_parameters true \
    --output_dir /data/swift/us_desc/ \
    --batch_size 1 \
    --lora_target_modules ALL \
    --gradient_accumulation_steps 16 \
    --gradient_checkpointing true \
    --use_flash_attn true \
    --num_train_epochs 5 \
    --save_strategy "steps" \
    --save_steps 500 \
    --sft_type "lora" \
    --save_total_limit 2 \
    --ddp_backend nccl \
    --save_only_model false \

[babla9]

I had the same issue occur on an 80gb A100.

[Jintao-Huang]

这个问题有点难解, 一定要训练vision encoder部分吗

使用的是main分支么

[lyc728]

sion encoder部

你昨天说把--lora_target_modules ALL \这个注释是一样的，走到验证就会溢出，使用的是main分支，才拉的

[lyc728]

指定多块GPU CUDA_VISIBLE_DEVICES=0,1,2,3

指定了的

[KasLoot]

Same problem and I posted a issue report on both github and wechat group weeks ago and no one response...

Anyway, I have figured out a way to "fix" this problem: set --evaluation_strategy no --batch_size 32. Any train batch size not 32 and has evaluation process will trigger the program to use 10 GB * eval_batch_size GPU resource. And do eliminate the evaluation process because it loads 30GB+ GPU memory from nowhere when finishing evluation process.

Another issuse when tranning is that: even though the above setup can fix the issue temporarily, it can still have memory leak every 50 steps when the program is trying to calculate some sort of GPU usage and epoch number kind of stuff.

[babla9]

@Jintao-Huang @tastelikefeet any thoughts on what the issue could be here? This is one of the better performance for gpu size tradeoffs models, so would be great to fine-tune.

[lyc728]

现在发现新的问题，我有一个能训练的数据集，现在对数据集进行采样了一部分，发现训练开始时就会超显存