Bug on Qwen2-VL over videomme

[ThisisBillhe]

It seems like qwen_vl_utils will occupy excessive memory when pre-process, until it get killed by system.

Traceback before it get killed:

  File "/mnt/workspace/lmms-eval-main/lmms_eval/models/qwen2_vl.py", line 228, in generate_until
    image_inputs, video_inputs = process_vision_info(messages)
  File "/mnt/workspace/anaconda3/envs/qwen2vl/lib/python3.10/site-packages/qwen_vl_utils/vision_process.py", line 219, in process_vision_info
    video_inputs.append(fetch_video(vision_info))
  File "/mnt/workspace/anaconda3/envs/qwen2vl/lib/python3.10/site-packages/qwen_vl_utils/vision_process.py", line 125, in fetch_video
    video, audio, info = io.read_video(
  File "/mnt/workspace/anaconda3/envs/qwen2vl/lib/python3.10/site-packages/torchvision/io/video.py", line 323, in read_video
    vframes_list = [frame.to_rgb().to_ndarray() for frame in video_frames]
  File "/mnt/workspace/anaconda3/envs/qwen2vl/lib/python3.10/site-packages/torchvision/io/video.py", line 323, in <listcomp>
    vframes_list = [frame.to_rgb().to_ndarray() for frame in video_frames]
KeyboardInterrupt
Model Responding:   0%|                                                                                                                                                                                  | 0/2700 [02:36<?, ?it/s]

[Luodian]

Hi thanks for raising this issue, can you help to confirm how many frames would be in this list in your case?

vframes_list

[Luodian]

I think a temporary solution is to put image list and let the code to inside this part in qwen_vl_utils

https://github.com/kq-chen/qwen-vl-utils/blob/fdefae03a8dfee0ec11274b77437db8951eb67fd/src/qwen_vl_utils/vision_process.py#L192C1-L201C22