Update README.md
Browse files
README.md
CHANGED
@@ -65,7 +65,7 @@ with torch.no_grad():
|
|
65 |
|
66 |
Before applying BPE to the pre-training corpus of English Tweets, we tokenized these Tweets using `TweetTokenizer` from the NLTK toolkit and used the `emoji` package to translate emotion icons into text strings (here, each icon is referred to as a word token). We also normalized the Tweets by converting user mentions and web/url links into special tokens `@USER` and `HTTPURL`, respectively. Thus it is recommended to also apply the same pre-processing step for BERTweet-based downstream applications w.r.t. the raw input Tweets.
|
67 |
|
68 |
-
For
|
69 |
|
70 |
- Installation: `pip3 install nltk emoji`
|
71 |
|
|
|
65 |
|
66 |
Before applying BPE to the pre-training corpus of English Tweets, we tokenized these Tweets using `TweetTokenizer` from the NLTK toolkit and used the `emoji` package to translate emotion icons into text strings (here, each icon is referred to as a word token). We also normalized the Tweets by converting user mentions and web/url links into special tokens `@USER` and `HTTPURL`, respectively. Thus it is recommended to also apply the same pre-processing step for BERTweet-based downstream applications w.r.t. the raw input Tweets.
|
67 |
|
68 |
+
For `vinai/bertweet-large`, given the raw input Tweets, to obtain the same pre-processing output, users could employ our [TweetNormalizer](https://github.com/VinAIResearch/BERTweet/blob/master/TweetNormalizer.py) module.
|
69 |
|
70 |
- Installation: `pip3 install nltk emoji`
|
71 |
|