Look forward to the GGUF version of this model

#1
by jian2023 - opened

Maybe it can be used in ollama

Currently using the v2 gguf, though this is a significant improvement and having a gguf of this would be greatly appreciated.
The strong OCR abilities of this model would be very helpful for making documents accessible.
This model is excellent and all your kind efforts and contributions to the community are greatly appreciated.

OpenBMB org

The GGUF format will be released soon😃

OpenBMB org

Thank you all for the valuable feedback! We really appreciate it. We are working on GGUF and ollama for MiniCPM-Llama3-V 2.5, which will be available soon. We hope the model can help the community and the people in need.

OpenBMB org
edited May 23

MiniCPM-Llama3-V 2.5 can run with llama.cpp now! See our fork of llama.cpp for more detail.

and here is our model in gguf format.
https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-gguf
@ibrahimkettaneh

thanks @Cuiunbo what are vram requirements for each of " ggml-model-Q4_K_M.gguf - mmproj-model-f16.gguf "

OpenBMB org

For memory consumption, mmproj-model-f16.gguf is about 1g+, and ggml-model-Q4_K_M.gguf is 5g.
But we optimize to run vit first, then free it, so that the overall consumption peaks at only 5g.
@xzintsux

OpenBMB org

We have supported Ollama! Please visit our GitHub for more usage info! https://github.com/OpenBMB/MiniCPM-V

Sign up or log in to comment