StevenLimcorn/indo-roberta-indonli

Indo-roberta-indonli

Indo-roberta-indonli is natural language inference classifier based on Indo-roberta model. It was trained on the trained on IndoNLI dataset. The model used was Indo-roberta and was transfer-learned to a natural inference classifier model. The model are tested using the validation, test_layer and test_expert dataset given in the github repository. The results are shown below.

Result

Dataset	Accuracy	F1	Precision	Recall
Test Lay	0.74329	0.74075	0.74283	0.74133
Test Expert	0.6115	0.60543	0.63924	0.61742

Model

The model was trained on with 5 epochs, batch size 16, learning rate 2e-5 and weight decay 0.01. Achieved different metrics as shown below.

Epoch	Training Loss	Validation Loss	Accuracy	F1	Precision	Recall
1	0.942500	0.658559	0.737369	0.735552	0.735488	0.736679
2	0.649200	0.645290	0.761493	0.759593	0.762784	0.759642
3	0.437100	0.667163	0.766045	0.763979	0.765740	0.763792
4	0.282000	0.786683	0.764679	0.761802	0.762011	0.761684
5	0.193500	0.925717	0.765134	0.763127	0.763560	0.763489

How to Use

As NLI Classifier

from transformers import pipeline
pretrained_name = "StevenLimcorn/indonesian-roberta-indonli"
nlp = pipeline(
    "zero-shot-classification",
    model=pretrained_name,
    tokenizer=pretrained_name
)
nlp("Amir Sjarifoeddin Harahap lahir di Kota Medan, Sumatera Utara, 27 April 1907. Ia meninggal di Surakarta, Jawa Tengah, pada 19 Desember 1948 dalam usia 41 tahun. </s></s> Amir Sjarifoeddin Harahap masih hidup.")

Disclaimer

Do consider the biases which come from both the pre-trained RoBERTa model and the INDONLI dataset that may be carried over into the results of this model.

Author

Indonesian RoBERTa Base IndoNLI was trained and evaluated by Steven Limcorn. All computation and development are done on Google Colaboratory using their free GPU access.

Reference

The dataset we used is by IndoNLI.

@inproceedings{indonli,
    title = "IndoNLI: A Natural Language Inference Dataset for Indonesian",
    author = "Mahendra, Rahmad and Aji, Alham Fikri and Louvan, Samuel and Rahman, Fahrurrozi and Vania, Clara",
    booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2021",
    publisher = "Association for Computational Linguistics",
}