README.md

---
language:
- id
tags:
- Twitter
license: apache-2.0
datasets:
- Twitter 2021
widget:
- text: "guweehh udh ga' paham lg sm [MASK]"
---

# IndoBERTweet 🐦 

## 1. Paper
Fajri Koto, Jey Han Lau, and Timothy Baldwin. [_IndoBERTweet: A Pretrained Language Model for Indonesian Twitter
with Effective Domain-Specific Vocabulary Initialization_](https://arxiv.org/pdf/2109.04607.pdf). 
In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (**EMNLP 2021**), Dominican Republic (virtual).

## 2. About

[IndoBERTweet](https://github.com/indolem/IndoBERTweet) is the first large-scale pretrained model for Indonesian Twitter
that is trained by extending a monolingually trained Indonesian BERT model with additive domain-specific vocabulary.

In this paper, we show that initializing domain-specific vocabulary with average-pooling of BERT subword embeddings is more efficient than pretraining from scratch, and more effective than initializing based on word2vec projections.

## 3. Pretraining Data

We crawl Indonesian tweets over a 1-year period using the official Twitter API, from December 2019 to December 2020, with 60 keywords covering 4 main topics: economy, health, education, and government. We obtain in total of **409M word tokens**, two times larger than the training data used to pretrain [IndoBERT](https://aclanthology.org/2020.coling-main.66.pdf). Due to Twitter policy, this pretraining data will not be released to public.

## 4. How to use

Load model and tokenizer (tested with transformers==3.5.1)
```python
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("indolem/indobertweet-base-uncased")
model = AutoModel.from_pretrained("indolem/indobertweet-base-uncased")
```
**Preprocessing Steps:**
* lower-case all words 
* converting user mentions and URLs into @USER and HTTPURL, respectively
* translating emoticons into text using the [emoji package](https://pypi.org/project/emoji/).

## 5. Results over 7 Indonesian Twitter Datasets

<table>
  <col>
  <colgroup span="2"></colgroup>
  <colgroup span="2"></colgroup>
  <tr>
    <th rowspan="2">Models</td>
    <th colspan="2" scope="colgroup">Sentiment</th>
    <th colspan="1" scope="colgroup">Emotion</th>
    <th colspan="2" scope="colgroup">Hate Speech</th>
    <th colspan="2" scope="colgroup">NER</th>
    <th rowspan="2" scope="colgroup">Average</th>
  </tr>
  <tr>
    <th scope="col">IndoLEM</th>
    <th scope="col">SmSA</th>
    <th scope="col">EmoT</th>
    <th scope="col">HS1</th>
    <th scope="col">HS2</th>
    <th scope="col">Formal</th>
    <th scope="col">Informal</th>
  </tr>
  <tr>
    <td scope="row">mBERT</td>
    <td>76.6</td>
    <td>84.7</td>
    <td>67.5</td>
    <td>85.1</td>
    <td>75.1</td>
    <td>85.2</td>
    <td>83.2</td>
    <td>79.6</td>
  </tr>
  <tr>
    <td scope="row">malayBERT</td>
    <td>82.0</td>
    <td>84.1</td>
    <td>74.2</td>
    <td>85.0</td>
    <td>81.9</td>
    <td>81.9</td>
    <td>81.3</td>
    <td>81.5</td>
  </tr>
  <tr>
    <td scope="row">IndoBERT (Willie, et al., 2020)</td>
    <td>84.1</td>
    <td>88.7</td>
    <td>73.3</td>
    <td>86.8</td>
    <td>80.4</td>
    <td>86.3</td>
    <td>84.3</td>
    <td>83.4</td>
  </tr>
  <tr>
    <td scope="row">IndoBERT (Koto, et al., 2020)</td>
    <td>84.1</td>
    <td>87.9</td>
    <td>71.0</td>
    <td>86.4</td>
    <td>79.3</td>
    <td>88.0</td>
  <td><b>86.9</b></td>
    <td>83.4</td>
  </tr>
  <tr>
    <td scope="row">IndoBERTweet (1M steps from scratch)</td>
    <td>86.2</td>
    <td>90.4</td>
    <td>76.0</td>
  <td><b>88.8</b></td>
  <td><b>87.5</b></td>
  <td><b>88.1</b></td>
    <td>85.4</td>
    <td>86.1</td>
  </tr>
  <tr>
    <td scope="row">IndoBERT + Voc adaptation + 200k steps</td>
  <td><b>86.6</b></td>
  <td><b>92.7</b></td>
  <td><b>79.0</b></td>
    <td>88.4</td>
    <td>84.0</td>
    <td>87.7</td>
  <td><b>86.9</b></td>
  <td><b>86.5</b></td>
  </tr>
</table>

## Citation
If you use our work, please cite:

```bibtex
@inproceedings{koto2021indobertweet,
  title={IndoBERTweet: A Pretrained Language Model for Indonesian Twitter with Effective Domain-Specific Vocabulary Initialization},
  author={Fajri Koto and Jey Han Lau and Timothy Baldwin},
  booktitle={Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP 2021)},
  year={2021}
}
```