---
license: apache-2.0
language:
- en
metrics:
- accuracy
pipeline_tag: image-text-to-text
---

# Introduction

We use the powerful [TinyLLaVA Factory](https://github.com/TinyLLaVA/TinyLLaVA_Factory) to create a super small image-text-to-text model with only 296M params.

The goal is to make it possible to run LLaVA models on edge devices (with few gigabytes of memory).

For LLM and vision tower, we choose [OpenELM-270M-Instruct](apple/OpenELM-270M-Instruct) and [facebook/dinov2-small](facebook/dinov2-small), respectively.

# Result

[POPE](https://tinyllava-factory.readthedocs.io/en/latest/Evaluation.html#pope):

| Category    | # Samples | TP   | FP  | TN   | FN  | Accuracy | Precision | Recall | F1 Score | Yes Ratio |
|-------------|------------|------|-----|------|-----|----------|-----------|--------|----------|-----------|
| Adversarial | 3000       | 1264 | 575 | 925  | 236 | 0.7297   | 0.6873    | 0.8427 | 0.7571   | 0.613     |
| Popular     | 3000       | 1264 | 301 | 1199 | 236 | 0.8210   | 0.8077    | 0.8427 | 0.8248   | 0.5217    |
| Random      | 2910       | 1264 | 290 | 1120 | 236 | 0.8192   | 0.8134    | 0.8427 | 0.8278   | 0.5340    |

[TEXTVQA](https://tinyllava-factory.readthedocs.io/en/latest/Evaluation.html#textvqa)

Samples 5000, Accuracy 27%

[SCIENCEQA](https://tinyllava-factory.readthedocs.io/en/latest/Evaluation.html#scienceqa)

Samples 4241, Correct: 1725, Accuracy: 40.64%, IMG-Accuracy: 36.54%