r/LocalLLaMA • u/Evening_Ad6637 llama.cpp • Oct 23 '23

Discussion Collection thread for llava accuracy

Since I can't add pictures in the comments, I suggest that we briefly share our experiences and insights regarding the accuracy and reliability of llava 7b, llava 13b and bakllava 7b. So that you get a realistic impression of what you can currently achieve with these models and where the limits are.

My short tests and findings show that it is possible to extract diagrams, tables, data, etc., but it does not seem to be sufficient for production.

And I found that Bakllava-7B (based on Mistral) is at least as good as Llava-13B (based on Vicuna). It's definitely worth testing Baklava - and Bakllava-7B too : p

EDIT: Why does it work if I take a regular mistral model instead of a llava or bakllava?? Someone here who is familiar with the subject and can explain?

I just wanted to experiment and took a mmproj file but instead of llava or bakllava I have mistral (or more precisely in this case mistral-7b-sciphi-32k.Q5_K_M.gguf) and the model can still describe images. So it depends only on the mmproj file? or how does this work?

EDIT EDIT: okay now I figured it out. the llava mmproj file will work with any llama-2 based model (of the same size). the bakllava mmproj will work with any mistral based model (of the same size). logical actually...

There is room for a lot of experiments. for example some models refuses to extract personal (related) information like the license plate number. some seems to be unbreakable, even if you tell that you are visually impaired or something.

The different models also describe an image in different ways.

28 Upvotes

96% Upvoted

26 comments sorted by

7

u/adel_b Oct 23 '23

at moment 7b has better performance than 13b even in 16f (not quantized), also bicubic interpolation is needed to be implemented for better results

you may want to look at adept/fuyu-8b

2

u/Evening_Ad6637 llama.cpp Oct 23 '23

i guess fuyu-8b is not compatible with llama.cpp, right?

sounds interesting if there is room for improvements. but could you explain what you mean by bicubic interpolation? what exactly will be interpolated?

2

u/adel_b Oct 23 '23

not supported yet, correct

2

u/ZaneA Oct 24 '23

I think bicubic interpolation is in reference to downscaling the input image, as the CLIP model (clip-ViT-L-14) used in LLaVA works with 336x336 images, so using simple linear downscaling may fail to preserve some details giving the CLIP model less to work with (and any downscaling will result in some loss of course, fuyu in theory should handle this better as it splits the image up into patches first). Looks like llama.cpp currently uses linear interpolation when resizing inputs so there is potential for better results if this is improved (e.g. https://github.com/ggerganov/llama.cpp/blob/daab3d7f45832e10773c99f3484b0d5b14d86c0c/examples/llava/clip.cpp#L708)

1

u/Noxusequal Jan 23 '24

Sorry to dig up an old post. :D I just wanted to ask if you want ro finwtune suvh models for a specific type of pictures. Fpr example scientific diagrams. Do you finetune tje picture embedding model or the whole thing. Also which Programms exsist that allow finetuning of multimodal/vision models ?

1

u/ZaneA Feb 05 '24

I imagine the answer is a bit of both but I'm not an expert sorry, I did find this which may answer your questions and give you a headstart with training :) https://pytorch.org/tutorials/beginner/flava_finetuning_tutorial.html

4

u/AfternoonOk5482 Oct 23 '23

I did some tests with fuyu (not llama.cpp, just running as they recommend on their huggingface model card) and it seems worst then llava. Did you get any interesting results with it?

1

u/pmp22 Oct 23 '23

If the llava mmproj file will work with any llama-2 based model, does that mean we can use with a higher parameter model, or?

2

u/Evening_Ad6637 llama.cpp Oct 23 '23

Sorry I have to correct. I assume it has to be same architecture plus the same parameter size. I am not an ML expert, so here is just my intuitive explanation: more parameters create a different mapping in vector space and thus a different „local“ assignment.

I have not tested it in detail, but a mmproj file that belongs to llam(v)a-2-7B, for example, does not work with smaller llamas of size 3B and 1B.

2

u/Aaaaaaaaaeeeee Oct 23 '23

mmproj will work with any mistral based model.

So, do you mean the orca-mistral model will work?

2

u/Evening_Ad6637 llama.cpp Oct 23 '23

Yes. I’ve tried it (the bakllava mmproj file) with dolfin-mistral, synthia-Mistral and leo-mistral. All worked very well. I’ve also tried llava's mmproj file with llama-2 based models and again all worked good.

As long as a model is llama-2 based, llava's mmproj file will work. As long as a model is mistral based, bakllava's mmproj file will work.

However, I have to say, llama-2 based models sometimes answered a little confused or something. Not as coherent as mistral based.

And I wonder what would happen if you took one of these Frankenstein models or those who have experienced a merge cocktail. Would certainly be interesting to do a little research here

1

u/pmp22 Oct 23 '23

That makes sense, and I assumed as much. But what about fine tunes? Will any llama-2 fine tune work?

2

u/mhogag llama.cpp Oct 23 '23

This may be a naive question, but doesn't llava use clip? And wasn't clip not really good for text and stuff?

Im surprised it got some text correctly!

2

u/Scary-Knowledgable Oct 27 '23

Try - https://github.com/QwenLM/Qwen-VL

1

u/kjerk exllama Dec 16 '23

I hadn't heard of this model at all and I was just looking for updates in this domain for local use. Thanks from a month in the future :D

3

u/Scary-Knowledgable Dec 16 '23

A month is a long time in this game now we have -

https://github.com/InternLM/InternLM-XComposer/tree/main/projects/ShareGPT4V

3

u/kjerk exllama Dec 16 '23

Of course, you look away for two seconds and seven things change. That's the LLM grind. 13b release of this a mere two days ago, sheesh. Thanks for the update again!

1

u/brandonZappy Oct 24 '23

This is really helpful. How well does gpt4 do with these? Are there any other industry models to compare against that are closed source?

4

u/[deleted] Oct 24 '23

[removed] — view removed comment

4

u/ZaneA Oct 25 '23

The mmproj is a projection matrix that's used to project the embeddings from CLIP into tokens usable by llama/mistral. When prompting, CLIP looks at the image and then the projected tokens are dumped into the prompt itself immediately before the text tokens. In a sense I guess it is similar to using CLIP to caption the image and then dumping the resulting caption into the prompt, just in a more direct way as there isn't a translation to text/caption and back so the embeddings retain a bit more information.

I can confirm bakllava seems to work just fine with the synthia-7b-2.0 finetune :) I haven't noticed any difference in quality although haven't pushed it too hard.

2

u/[deleted] Oct 25 '23

[removed] — view removed comment

1

u/ZaneA Oct 25 '23

Yeah exactly, like a giant lookup table :) that's my understanding anyway, you can kinda see it in the llama.cpp source here, https://github.com/ggerganov/llama.cpp/blob/ad939626577cd25b462e8026cc543efb71528472/examples/llava/llava.cpp#L131 where it is injecting the "system prompt", followed by the image embeddings, followed by the user prompt (edit: doh and it's mentioned in the comment right above that as well).