[MiniGPT4] Add MiniGPT4 to SHARK

[Abhishek-Varma]

-- This is the first installment of MiniGPT4 in SHARK.

Signed-off-by: Abhishek Varma [email protected]

Download the vmfbs from SHARK Tank

The script currently has changed the vmfb names so it needs to be modified like the latest fp16 Vision Model's vmfb : minigpt4_vision_model_fp16_cuda.vmfb after downloading accordingly (I'm working on refactoring this all and get rid of the manual downloading - this is WIP patch).

Discalimer: This is a pretty big WIP patch

[powderluv]

Lets call this "MultiModal" in the UI since I assume we will want to switch out LLMs etc in the future.

[Abhishek-Varma]

Hi @powderluv - we can merge this and handle bugs in subsequent PR.

1. I've used source build of IREE to compile first/second llama in fp16 for CUDA.
2. I've verified it works in both CLI and WebUI in Linux CUDA.
3. Have cloned the llama model, used for initialising the first/second, as a workaround - because after compiling the first llama for fp16 - because we change fx graph, it was crashing for second llama.

In the subsequent PR - I'd try to handle your comment + automate certain aspect of running this script which currently requires manual intervention + try to get rid of other part of this still huge a code base.

Also, I'll have to try for int8 - let me know your thoughts here as well!

[powderluv]

Please lint and we can merge it in and iterate.

[Abhishek-Varma]

Please lint and we can merge it in and iterate.

Hi @powderluv
I've formatted the said files.
We can merge this in.

[powderluv]

Is this ready to merge ?

[Abhishek-Varma]

Is this ready to merge ?

Hi @powderluv .
This can be merged. Because I was working on a separate branch for int8 and shared that yesterday - I've added the extra set of commits on top of the fp16/fp32 commit.

The CI failure looks unrelated.