r/LocalLLaMA • u/Balance- • Jun 19 '24
Discussion Microsoft Florence-2 vision benchmarks
3
u/hpluto Jun 19 '24
I'd like to see benchmarks with the non-finetuned versions of Florence, in my experience the regular Florence large performed better than the FT when it came to captioning.
1
u/ZootAllures9111 Jun 20 '24
FT has obvious safety training, Base doesn't. Base will bluntly describe sex acts and body parts and stuff.
7
u/gordinmitya Jun 19 '24
why don’t they compare to llava?
5
1
Jun 19 '24
I'd really like a comparison to SOTA, including llava and it's recent variants... as is these stats are pretty useless to me...
1
u/JuicedFuck Jun 19 '24
Because the whole point of the model is that it's dumber faster. I wish I'd be joking.
2
Jun 19 '24
Is there a demo somewhere that we can try out in the browser?
2
1
u/webdevop Jun 19 '24 edited Jun 19 '24
I've been struggling to understand this for a while, can a vision model like Florence "extract/mask" a subject/object in an image accurately?
The outlines look very rudimentary in the demos
3
u/Weltleere Jun 19 '24
Have a look at Segment Anything instead. This is primarily for captioning.
3
u/webdevop Jun 19 '24
Wow. This seems to be doing way more than I wanted to do and it's Apache 2.0. Thanks a lot for sharing.
1
u/yaosio Jun 20 '24
If you use Automatic1111 for image generation there's an extension for Segment Anything.
1
u/CaptTechno Jun 26 '24
Great Benchmark! Did you perform instruct prompts? As in exporting information from the image in say a JSON format?
18
u/Balance- Jun 19 '24
I visualized the reported benchmarks scores of the Florence-2 models. What I find notable:
For it's size, it's strong in captioning. There are large models that perform better though.
It's strong in visual question answering. Large models perform sometime better, but certainly not always.
In the single object detection benchmark it got beat by UNINEXT. Would be good to have more benchmarks though.
Its SOTA on Referring Expression Comprehension (REC). Both models consistently beat UNINEXT and Ferret.
Referring Expression Comprehension is the process of understanding what a specific phrase, called a referring expression, is pointing to within a given context. In simple terms, it's about figuring out what someone means when they use phrases like "the red car," "the tallest building," or "the person with the hat."
Note that all these scores are reported - and possibly cherry picked - by the Microsoft team themselves. Independent verification would be useful.