---
library_name: transformers
pipeline_tag: image-text-to-text
tags:
- vision
license: mit
language:
- en
base_model:
- microsoft/Florence-2-base
---

# PTA-1: Controlling Computers with Small Models

PTA (Prompt-to-Action) is a vision language model for computer use applications based on Florence-2. 
With less than 300M parameters it beats larger models in GUI text and element localization.
This allows low latency computer automations with local execution.

**Model Input:** Screenshot + description_of_target_element

**Model Output:** BoundingBox for Target Element


## How to Get Started with the Model

Use the code below to get started with the model.


```python
from PIL import Image
from transformers import AutoProcessor, AutoModelForCausalLM 

device = "cuda:0" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

model = AutoModelForCausalLM.from_pretrained("AskUI/PTA-1", torch_dtype=torch_dtype, trust_remote_code=True).to(device)
processor = AutoProcessor.from_pretrained("AskUI/PTA-1", trust_remote_code=True)

task_prompt = "<OPEN_VOCABULARY_DETECTION>"
prompt = task_prompt + "description of the target element"

image = Image.open("path to screenshot")

inputs = processor(text=prompt, images=image, return_tensors="pt").to(device, torch_dtype)

generated_ids = model.generate(
    input_ids=inputs["input_ids"],
    pixel_values=inputs["pixel_values"],
    max_new_tokens=1024,
    do_sample=False,
    num_beams=3,
)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=False)[0]

parsed_answer = processor.post_process_generation(generated_text, task="<OPEN_VOCABULARY_DETECTION>", image_size=(image.width, image.height))

print(parsed_answer)
```


## Evaluation

**Note:** This is a first version of our evaluation with 999 samples (333 samples from each dataset). 
We are still running all models on the full test sets. We are seeing +-5% deviations for a subset of the models we have already evaluated.

| Model                                      | Parameters | Mean   | agentsea/wave-ui | AskUI/pta-text | ivelin/rico_refexp_combined |
|--------------------------------------------|------------|--------|------------------|----------------|-----------------------------|
| AskUI/PTA-1                                | 0.27B      | 79.98  | 90.69*           | 76.28          | 72.97*                      |
| anthropic.claude-3-5-sonnet-20241022-v2:0  | -          | 70.37  | 82.28            | 83.18          | 45.65                       |
| agentsea/paligemma-3b-ft-waveui-896        | 3.29B      | 57.76  | 70.57*           | 67.87          | 34.83                       |
| Qwen/Qwen2-VL-7B-Instruct                  | 8.29B      | 57.26  | 47.45            | 60.66          | 63.66                       |
| agentsea/paligemma-3b-ft-widgetcap-waveui-448 | 3.29B   | 53.15  | 74.17*           | 53.45          | 31.83                       |
| microsoft/Florence-2-base                  | 0.27B      | 39.44  | 22.22            | 81.38          | 14.71                       |
| microsoft/Florence-2-large                 | 0.82B      | 36.64  | 14.11            | 81.98          | 13.81                       |
| EasyOCR                                    | -          | 29.43  | 3.9              | 75.08          | 9.31                        |
| adept/fuyu-8b                              | 9.41B      | 26.83  | 5.71             | 71.47          | 3.3                         |
| Qwen/Qwen2-VL-2B-Instruct                  | 2.21B      | 23.32  | 17.12            | 26.13          | 26.73                       |
| Qwen/Qwen2-VL-2B-Instruct-GPTQ-Int4        | 0.90B      | 18.92  | 10.81            | 22.82          | 23.12                       |


\* Models is known to be trained on the train split of that dataset.

The high benchmark scores for our model are partially due to data bias. 
Therefore we expect users of the model to fine-tune it according to the data distributions of their use case. 


#### Metrics

Click success rate is calculated as the number of clicks inside the target bounding box. 
If a model predicts a target bounding box instead of a click coordinate, its center is used as its click prediction.