Create readme usage and benchmark
Browse files
README.md
ADDED
@@ -0,0 +1,59 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
## Requirements
|
2 |
+
|
3 |
+
```bash
|
4 |
+
pip install pythainlp
|
5 |
+
pip install git+https://github.com/openai/CLIP.git
|
6 |
+
```
|
7 |
+
|
8 |
+
## Usage
|
9 |
+
|
10 |
+
Encode a text by
|
11 |
+
```python
|
12 |
+
from transformers import AutoModel
|
13 |
+
|
14 |
+
text = 'หมากำลังวิ่งในสนามหญ้า'
|
15 |
+
model = AutoModel.from_pretrained("patomp/thai-light-multimodal-clip-and-distill", trust_remote_code=True)
|
16 |
+
|
17 |
+
embeddings = model(text)
|
18 |
+
print("Text features shape:", embeddings.shape)
|
19 |
+
|
20 |
+
```
|
21 |
+
|
22 |
+
Encode an image by
|
23 |
+
```python
|
24 |
+
import torch
|
25 |
+
import clip
|
26 |
+
import requests
|
27 |
+
from PIL import Image
|
28 |
+
|
29 |
+
device = "cuda" if torch.cuda.is_available() else "cpu"
|
30 |
+
model, preprocess = clip.load("ViT-B/32", device=device)
|
31 |
+
|
32 |
+
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
|
33 |
+
image = Image.open(requests.get(url, stream=True).raw)
|
34 |
+
image = preprocess(image).unsqueeze(0).to(device)
|
35 |
+
|
36 |
+
with torch.no_grad():
|
37 |
+
image_features = model.encode_image(image)
|
38 |
+
|
39 |
+
print("Image features shape:", image_features.shape)
|
40 |
+
```
|
41 |
+
|
42 |
+
## Benchmark
|
43 |
+
|
44 |
+
On the test set of [Thai MS COCO 2014 dataset](https://huggingface.co/datasets/patomp/thai-mscoco-2014-captions)
|
45 |
+
|
46 |
+
| Model \ Metrics | text-find-image recall@1 | text-find-image recall@10 | image-find-text recall@1 | image-find-text recall@10 | # text samples per second* |
|
47 |
+
| :--- | --- | --- | --- | --- | --- |
|
48 |
+
| **Multilingual Encoder** | | | | | |
|
49 |
+
| [clip-ViT-B-32-multilingual-v1](https://huggingface.co/sentence-transformers/clip-ViT-B-32-multilingual-v1) | 0.075 | 0.242 | 0.096 | 0.286 | - |
|
50 |
+
| [XLM-Roberta-Large-Vit-B-32](https://huggingface.co/M-CLIP/XLM-Roberta-Large-Vit-B-32) | **0.226** | **0.565** | **0.265** | **0.596** | 20 |
|
51 |
+
| **Thai Encoder (WangchanBERTa-based)** | | | | | 48 |
|
52 |
+
| [Thai-Cross-CLIP](https://github.com/vikimark/Thai-Cross-CLIP) | 0.167 | 0.475 | 0.197 | 0.523 | |
|
53 |
+
| **Thai Encoder (Thai2Fit-based)** | | | | | |
|
54 |
+
| [thai-light-multimodal-clip-and-distill](https://huggingface.co/patomp/thai-light-multimodal-clip-and-distill) | 0.082 | **0.328** | 0.118 |**0.401**| 450 |
|
55 |
+
| [thai-light-multimodal-distill](https://huggingface.co/patomp/thai-light-multimodal-distill) | **0.084** | 0.319 | 0.122 |**0.401**| 450 |
|
56 |
+
|
57 |
+
## Reference
|
58 |
+
|
59 |
+
Some part of this content referenced from https://huggingface.co/M-CLIP/XLM-Roberta-Large-Vit-B-32
|