Spaces:
Running
Running
Update README.md
Browse files
README.md
CHANGED
|
@@ -10,72 +10,6 @@ pinned: true
|
|
| 10 |
|
| 11 |
We decided to create an organization to collect the latest (and useable) models for the Hungarian specific finetuned LLMs (Whisper, Bart, LLama, etc). Feel free to join our organization and push your models.
|
| 12 |
|
| 13 |
-
## About the models
|
| 14 |
-
Hungarian language specific compare test results (on Google/flerus):
|
| 15 |
-
|
| 16 |
-
| Original models | WER | CER | Normalized_WER | Normalized_CER | Database | Split | Runtime |
|
| 17 |
-
|:--------------------------------------------------------------------------------------------------------|:-------|:------|:-----------------|:-----------------|:--------------|:--------|:----------|
|
| 18 |
-
| [openai/whisper-tiny](https://huggingface.co/openai/whisper-tiny) | 102.46 | 50.31 | 103.37 | 50.19 | google/fleurs | test | 60.44 |
|
| 19 |
-
| [openai/whisper-base](https://huggingface.co/openai/whisper-base) | 89.08 | 41.3 | 93.13 | 41.56 | google/fleurs | test | 89.66 |
|
| 20 |
-
| [openai/whisper-small](https://huggingface.co/openai/whisper-small) | 48.67 | 15.1 | 45.55 | 15.39 | google/fleurs | test | 175.03 |
|
| 21 |
-
| [openai/whisper-medium](https://huggingface.co/openai/whisper-medium) | 32.49 | 9.58 | 29.04 | 10.05 | google/fleurs | test | 393.56 |
|
| 22 |
-
| [openai/whisper-large](https://huggingface.co/openai/whisper-large) | 28.2 | 7.77 | 24.76 | 8.31 | google/fleurs | test | 675.77 |
|
| 23 |
-
| [openai/whisper-large-v2](https://huggingface.co/openai/whisper-large-v2) | 23.14 | 5.94 | 19.83 | 6.48 | google/fleurs | test | 772.64 |
|
| 24 |
-
| [openai/whisper-large-v3](https://huggingface.co/openai/whisper-large-v3) | 18.88 | 4.56 | 15.48 | 5.2 | google/fleurs | test | 667.66 |
|
| 25 |
-
| Finetuned models | | | | | | | |
|
| 26 |
-
| [Hungarians/whisper-small-cv17-hu](https://huggingface.co/Hungarians/whisper-small-cv17-hu) | 188.94 | 75.87 | 188.21 | 77.32 | google/fleurs | test | 472.43 |
|
| 27 |
-
| [Hungarians/whisper-tiny-cv16-hu-v3](https://huggingface.co/Hungarians/whisper-tiny-cv16-hu-v3) | 75.9 | 50.61 | 85.55 | 50.91 | google/fleurs | test | 65.17 |
|
| 28 |
-
| [Hungarians/whisper-tiny-cv16-hu-v2](https://huggingface.co/Hungarians/whisper-tiny-cv16-hu-v2) | 72.13 | 41.71 | 71.13 | 41.45 | google/fleurs | test | 50.48 |
|
| 29 |
-
| [Hungarians/whisper-tiny-cv16-hu-final](https://huggingface.co/Hungarians/whisper-tiny-cv16-hu-final) | 68.43 | 38.24 | 64.07 | 38.14 | google/fleurs | test | 41.48 |
|
| 30 |
-
| [Hungarians/whisper-tiny-cv16-hu](https://huggingface.co/Hungarians/whisper-tiny-cv16-hu) | 64.7 | 28.02 | 60.9 | 27.7 | google/fleurs | test | 42.35 |
|
| 31 |
-
| [Hungarians/whisper-tiny-hu-cleaned](https://huggingface.co/Hungarians/whisper-tiny-hu-cleaned) | 59.67 | 26.01 | 54.72 | 25.73 | google/fleurs | test | 33.72 |
|
| 32 |
-
| [Hungarians/whisper-tiny-cv17-hu](https://huggingface.co/Hungarians/whisper-tiny-cv17-hu) | 58.76 | 24.86 | 56.1 | 24.72 | google/fleurs | test | 39.57 |
|
| 33 |
-
| [sarpba/whisper-tiny-cv18-hu-cleaned](https://huggingface.co/sarpba/whisper-tiny-cv18-hu-cleaned) | 52.74 | 24.02 | 50.09 | 23.91 | google/fleurs | test | 40.16 |
|
| 34 |
-
| [Hungarians/whisper-base-cv16-hu-v2](https://huggingface.co/Hungarians/whisper-base-cv16-hu-v2) | 51.41 | 20.97 | 46.79 | 20.93 | google/fleurs | test | 70.57 |
|
| 35 |
-
| [Hungarians/whisper-base-hu-cleaned](https://huggingface.co/Hungarians/whisper-base-hu-cleaned) | 51.38 | 20.05 | 46.54 | 20.14 | google/fleurs | test | 70.84 |
|
| 36 |
-
| [Hungarians/whisper-base-cv16-hu](https://huggingface.co/Hungarians/whisper-base-cv16-hu) | 50.06 | 17.71 | 44.83 | 17.44 | google/fleurs | test | 65.49 |
|
| 37 |
-
| [Hungarians/whisper-medium-cv16-hu](https://huggingface.co/Hungarians/whisper-medium-cv16-hu) | 49.77 | 24.98 | 47.79 | 25.4 | google/fleurs | test | 498.53 |
|
| 38 |
-
| [Hungarians/whisper-base-cv16-hu-final](https://huggingface.co/Hungarians/whisper-base-cv16-hu-final) | 48.37 | 16.28 | 43.84 | 16.31 | google/fleurs | test | 67.07 |
|
| 39 |
-
| [Hungarians/whisper-base-cv17-hu](https://huggingface.co/Hungarians/whisper-base-cv17-hu) | 45.61 | 14.95 | 40.79 | 14.94 | google/fleurs | test | 64.15 |
|
| 40 |
-
| [sarpba/whisper-base-cv18-hu-cleaned](https://huggingface.co/sarpba/whisper-base-cv18-hu-cleaned) | 42.09 | 13.67 | 36.66 | 13.53 | google/fleurs | test | 54.7 |
|
| 41 |
-
| [Hungarians/whisper-small-cv16-hu-v2](https://huggingface.co/Hungarians/whisper-small-cv16-hu-v2) | 41.07 | 13.16 | 36.59 | 13.21 | google/fleurs | test | 201.28 |
|
| 42 |
-
| [Hungarians/Whisper-small-hu-cleaned](https://huggingface.co/Hungarians/Whisper-small-hu-cleaned) | 39.12 | 13.91 | 41.15 | 14.11 | google/fleurs | test | 274.09 |
|
| 43 |
-
| [Hungarians/whisper-small-cv16-hu](https://huggingface.co/Hungarians/whisper-small-cv16-hu) | 37.5 | 11.31 | 32.54 | 11.35 | google/fleurs | test | 608.28 |
|
| 44 |
-
| [Hungarians/whisper-small-cv16-hu-v1.5](https://huggingface.co/Hungarians/whisper-small-cv16-hu-v1.5) | 35.61 | 10.99 | 30.33 | 11.04 | google/fleurs | test | 605.69 |
|
| 45 |
-
| [Hungarians/whisper-medium-hu-cleaned](https://huggingface.co/Hungarians/whisper-medium-hu-cleaned) | 26.26 | 6.8 | 21.97 | 7.31 | google/fleurs | test | 442.53 |
|
| 46 |
-
| Our best models | | | | | | | |
|
| 47 |
-
| [sarpba/whisper-tiny-cv18-hu-cleaned](https://huggingface.co/sarpba/whisper-tiny-cv18-hu-cleaned) | 52.74 | 24.02 | 50.09 | 23.91 | google/fleurs | test | 40.16 |
|
| 48 |
-
| [sarpba/whisper-base-cv18-hu-cleaned](https://huggingface.co/sarpba/whisper-base-cv18-hu-cleaned) | 42.09 | 13.67 | 36.66 | 13.53 | google/fleurs | test | 54.7 |
|
| 49 |
-
| [sarpba/whisper-small-cv18-hu-cleaned](https://huggingface.co/sarpba/whisper-small-cv18-hu-cleaned) | 29.75 | 9.23 | 25.19 | 9.38 | google/fleurs | test | 281.95 |
|
| 50 |
-
| [sarpba/whisper-medium-cv18-hu-cleaned](https://huggingface.co/sarpba/whisper-medium-cv18-hu-cleaned) | 23.89 | 6.79 | 19.81 | 7.3 | google/fleurs | test | 541.17 |
|
| 51 |
-
| [Hungarians/whisper-large-v2-hu-cleaned](https://huggingface.co/Hungarians/whisper-large-v2-hu-cleaned) | 21.82 | 5.51 | 18.39 | 6.15 | google/fleurs | test | 725.31 |
|
| 52 |
-
AZ UTOLSÓ HÁROM SOR INT8 KVANTÁLT MODELL EREDMÉNYE.
|
| 53 |
-
|
| 54 |
-
## Quant loss examle
|
| 55 |
-
| Model | WER | CER | Normalized_WER | Normalized_CER | Database | Split | Runtime |
|
| 56 |
-
|:----------------------------------------------------------|:------|:------|:-----------------|:-----------------|:--------------|:--------|:----------|
|
| 57 |
-
| Hungarians/whisper-base-cv17-hu | 45.61 | 14.95 | 40.79 | 14.94 | google/fleurs | test | 243.97 |
|
| 58 |
-
| float16 | 50.55 | 21.01 | 46.81 | 20.99 | google/fleurs | test | 301.41 |
|
| 59 |
-
| float32 | 49.69 | 20.77 | 47.38 | 20.74 | google/fleurs | test | 339.15 |
|
| 60 |
-
| int8_float32 | 46.71 | 16.67 | 42.51 | 16.51 | google/fleurs | test | 246.06 |
|
| 61 |
-
| int8_float16 | 46.5 | 17.13 | 42.23 | 16.92 | google/fleurs | test | 242.12 |
|
| 62 |
-
| int8_bfloat16 | 45.7 | 15.06 | 41.03 | 15.04 | google/fleurs | test | 148.05 |
|
| 63 |
-
| bfloat16 | 45.6 | 15 | 40.88 | 14.97 | google/fleurs | test | 144.87 |
|
| 64 |
-
| int8 | 45.54 | 16.55 | 42.4 | 16.44 | google/fleurs | test | 236.97 |
|
| 65 |
-
|
| 66 |
-
As you can see the INT8 quant have better points form original modell.
|
| 67 |
-
|
| 68 |
-
|
| 69 |
-
Lower value is better!
|
| 70 |
-
|
| 71 |
-
For Homeassistant faster-whisper need to use, the int8, fp16, fp32 modells, from subfolders.
|
| 72 |
-
|
| 73 |
-
# Some Hungarian info bellow:
|
| 74 |
-
|
| 75 |
-
A kész nodellek mindíg itt vannak, az én (sarpba) repómban a félkész, vagy kisérleti stádiumu cuccok vannak.
|
| 76 |
-
|
| 77 |
-
Hosassistant faster-whisperhez az almappákban lévő int8, fp16, fp32 ct2 quantised (ezt nem tom hogy kéne magyarul írni :)) modelleket tudjátok használni a legegyszerűbben cociweb [custom_whisper](https://github.com/cociweb/custom_whisper.git) addonjával.
|
| 78 |
-
|
| 79 |
## Közösség
|
| 80 |
|
| 81 |
Ha szeretnél csatlakozni a magyar nyelvű társalkodó csoportunkhoz ahol kérdezhetsz, megoszthatod a tapasztalataidat, vagy egy, a magyar LLM szakértőiből álló csoport tagja szeretnél lenni, csatlakozz FB csoportunkhoz: [Hungarian-LLM](https://www.facebook.com/groups/hungarian.llm).
|
|
|
|
| 10 |
|
| 11 |
We decided to create an organization to collect the latest (and useable) models for the Hungarian specific finetuned LLMs (Whisper, Bart, LLama, etc). Feel free to join our organization and push your models.
|
| 12 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 13 |
## Közösség
|
| 14 |
|
| 15 |
Ha szeretnél csatlakozni a magyar nyelvű társalkodó csoportunkhoz ahol kérdezhetsz, megoszthatod a tapasztalataidat, vagy egy, a magyar LLM szakértőiből álló csoport tagja szeretnél lenni, csatlakozz FB csoportunkhoz: [Hungarian-LLM](https://www.facebook.com/groups/hungarian.llm).
|