|
--- |
|
language: en |
|
tags: |
|
- exbert |
|
license: apache-2.0 |
|
datasets: |
|
- Confidential |
|
--- |
|
# BERT base model (uncased) |
|
|
|
Pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in |
|
[this paper](https://arxiv.org/abs/1810.04805) and first released in |
|
[this repository](https://github.com/google-research/bert). This model is uncased: it does not make a difference |
|
between english and English. |
|
|
|
## Model description |
|
|
|
BERT is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. This means it |
|
was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of |
|
publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely, it |
|
was pretrained with two objectives: |
|
|
|
- Masked language modeling (MLM): taking a sentence, the model randomly masks 15% of the words in the input then run |
|
the entire masked sentence through the model and has to predict the masked words. This is different from traditional |
|
recurrent neural networks (RNNs) that usually see the words one after the other, or from autoregressive models like |
|
GPT which internally mask the future tokens. It allows the model to learn a bidirectional representation of the |
|
sentence. |
|
- Next sentence prediction (NSP): the models concatenates two masked sentences as inputs during pretraining. Sometimes |
|
they correspond to sentences that were next to each other in the original text, sometimes not. The model then has to |
|
predict if the two sentences were following each other or not. |
|
|
|
This way, the model learns an inner representation of the English language that can then be used to extract features |
|
useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard |
|
classifier using the features produced by the BERT model as inputs. |
|
|
|
## Model description [xaqren/sentiment_analysis] |
|
|
|
This is a fine-tuned downstream version of the bert-base-uncased model for sentiment analysis, this model is not intended for |
|
further downstream fine-tuning for any other tasks. This model is trained on a classified dataset for text-classification. |
