Issue of GPT4o evaluation with dense frame sampling strategy

[HYUNJS]

Thanks for open-sourcing this great evaluation framework!

I've encountered an issue while using GPT-4o on Video-MME and was wondering if anyone has already found a solution for it.

When attempting to evaluate GPT-4o with a dense frame sampling strategy (e.g., 512 frames) in low-resolution mode, I got an invalid request error. I’ve verified that the number of tokens is within the 128K context length limit, and I've tried both normal API and Batch API requests, but the error happens.

Specifically, I’m prompting GPT-4o with 338 frames in low-resolution mode, interleaved with some text prompts between the frames. The following error occurs:
openai.BadRequestError: Error code: 400 - {'error': {'message': 'Too many images in request: 251, maximum allowed: 250.', 'type': 'invalid_request_error', 'param': None, 'code': None}}