|
--- |
|
license: cc-by-4.0 |
|
--- |
|
|
|
# Clean ConceptNet Data for All Languages |
|
|
|
## Data Details |
|
|
|
For our project on [Retrofitting Glove embeddings for Low Resource Languages](https://github.com/pyRis/retrofitting-embeddings-lrls/tree/main?tab=readme-ov-file), we extracted all data from the [ConceptNet](https://github.com/commonsense/conceptnet5/wiki/Downloads) database for 304 languages. The extraction process involved several steps to clean and analyze the data from the official ConceptNet dump available [here](https://s3.amazonaws.com/conceptnet/downloads/2019/edges/conceptnet-assertions-5.7.0.csv.gz). |
|
|
|
The final extracted dataset, available in another [HuggingFace repo](https://huggingface.co/datasets/DGurgurov/conceptnet_all), was used for training the graph embeddings using PPMI and consequently applying SVD on the co-occurence statistics of PPMI between the words. |
|
|
|
We generate graph embeddings for 72 languages present in both CC100 and ConceptNet. |
|
|
|
### Dataset Structure |
|
|
|
Each file is a txt file with a word / phrase and corresponding embedding separated with a space. |
|
|
|
Use the following function to read in the embeddings: |
|
|
|
```python |
|
def read_embeddings_from_text(file_path, embedding_size=300): |
|
"""Function to read the embeddings from a txt file""" |
|
embeddings = {} |
|
with open(file_path, 'r', encoding='utf-8') as file: |
|
for line in file: |
|
parts = line.strip().split(' ') |
|
embedding_start_index = len(parts) - embedding_size |
|
phrase = ' '.join(parts[:embedding_start_index]) |
|
embedding = np.array([float(val) for val in parts[embedding_start_index:]]) |
|
embeddings[phrase] = embedding |
|
return embeddings |
|
``` |
|
|
|
### Language Details |
|
|
|
| Language Code | Language Name | Vocabulary Size| |
|
| --- | --- | --- | |
|
| af | Afrikaans | 12973 | |
|
| sc | Sardinian | 573 | |
|
| yo | Yoruba | 2283 | |
|
| gn | Guarani | 131 | |
|
| qu | Quechua | 5156 | |
|
| li | Limburgish | 485 | |
|
| ln | Lingala | 4109 | |
|
| wo | Wolof | 1196 | |
|
| zu | Zulu | 2758 | |
|
| rm | Romansh | 3919 | |
|
| ht | Haitian Creole | 2699 | |
|
| su | Sundanese | 2514 | |
|
| br | Breton | 11665 | |
|
| gd | Scottish Gaelic | 14418 | |
|
| xh | Xhosa | 2504 | |
|
| mg | Malagasy | 26575 | |
|
| jv | Javanese | 4919 | |
|
| fy | Frisian | 7608 | |
|
| sa | Sanskrit | 5789 | |
|
| my | Burmese | 4875 | |
|
| ug | Uyghur | 998 | |
|
| yi | Yiddish | 8054 | |
|
| or | Oriya | 109 | |
|
| ha | Hausa | 802 | |
|
| la | Latin | 848943 | |
|
| sd | Sindhi | 143 | |
|
| so | Somali | 593 | |
|
| ku | Kurdish | 9737 | |
|
| pa | Punjabi | 4488 | |
|
| ps | Pashto | 1087 | |
|
| ga | Irish | 29459 | |
|
| am | Amharic | 1909 | |
|
| km | Khmer | 3466 | |
|
| uz | Uzbek | 5224 | |
|
| ky | Kyrgyz | 3574 | |
|
| cy | Welsh | 13243 | |
|
| gu | Gujarati | 4427 | |
|
| eo | Esperanto | 91074 | |
|
| sw | Swahili | 9131 | |
|
| mr | Marathi | 5545 | |
|
| kn | Kannada | 3415 | |
|
| ne | Nepali | 4224 | |
|
| mn | Mongolian | 6740 | |
|
| si | Sinhala | 2062 | |
|
| te | Telugu | 18707 | |
|
| be | Belarusian | 14871 | |
|
| mk | Macedonian | 28935 | |
|
| gl | Galician | 52824 | |
|
| hy | Armenian | 23434 | |
|
| is | Icelandic | 40287 | |
|
| ml | Malayalam | 6750 | |
|
| bn | Bengali | 7306 | |
|
| ur | Urdu | 8476 | |
|
| kk | Kazakh | 13700 | |
|
| ka | Georgian | 25014 | |
|
| az | Azerbaijani | 13277 | |
|
| sq | Albanian | 16262 | |
|
| ta | Tamil | 9064 | |
|
| et | Estonian | 20088 | |
|
| lv | Latvian | 30059 | |
|
| ms | Malay | 88416 | |
|
| sl | Slovenian | 89210 | |
|
| lt | Lithuanian | 21184 | |
|
| he | Hebrew | 27283 | |
|
| sk | Slovak | 21657 | |
|
| el | Greek | 39667 | |
|
| th | Thai | 94281 | |
|
| bg | Bulgarian | 171740 | |
|
| da | Danish | 46600 | |
|
| uk | Ukrainian | 27682 | |
|
| ro | Romanian | 36206 | |
|
|
|
|
|
### Licensing Information |
|
|
|
This work includes data from ConceptNet 5, which was compiled by the |
|
Commonsense Computing Initiative. ConceptNet 5 is freely available under |
|
the Creative Commons Attribution-ShareAlike license (CC BY SA 3.0) from |
|
http://conceptnet.io. |
|
|
|
### Citation Information |
|
|
|
``` |
|
@misc{gurgurov2024lowremrepositorywordembeddings, |
|
title={LowREm: A Repository of Word Embeddings for 87 Low-Resource Languages Enhanced with Multilingual Graph Knowledge}, |
|
author={Daniil Gurgurov and Rishu Kumar and Simon Ostermann}, |
|
year={2024}, |
|
eprint={2409.18193}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.CL}, |
|
url={https://arxiv.org/abs/2409.18193}, |
|
} |
|
|
|
@paper{speer2017conceptnet, |
|
author = {Robyn Speer and Joshua Chin and Catherine Havasi}, |
|
title = {ConceptNet 5.5: An Open Multilingual Graph of General Knowledge}, |
|
conference = {AAAI Conference on Artificial Intelligence}, |
|
year = {2017}, |
|
pages = {4444--4451}, |
|
keywords = {ConceptNet; knowledge graph; word embeddings}, |
|
url = {http://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14972} |
|
} |
|
``` |
