Update README.md

README.md

@@ -2,164 +2,149 @@
 license: cc
 language:
 - pt
-pipeline_tag: text-classification
 tags:
 - Hate Speech
 - kNOwHATE
 widget:
-- text: "as pessoas tem que perceber que ser 'panasca' não é deixar de ser homem, é deixar de ser humano kkk"
+- text: >-
+    Os [MASK] são todos uns animais, deviam voltar para a sua terra.
 ---
+---
+<img align="left" width="140" height="140" src="https://ilga-portugal.pt/files/uploads/2023/06/logo_HATE_cores_page-0001-1024x539.jpg">
+<p style="text-align: center;">&nbsp;&nbsp;&nbsp;&nbsp;This is the model card for HateBERTimbau. 
+  You may be interested in some of the other models from the <a href="https://huggingface.co/knowhate">kNOwHATE project</a>.
+</p>
 
-# Model Card for Model ID
-
-<!-- Provide a quick summary of what the model is/does. -->
-
-This modelcard aims to be a base template for new models. It has been generated using [this raw template](https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/templates/modelcard_template.md?plain=1).
-
-## Model Details
-
-### Model Description
-
-<!-- Provide a longer summary of what this model is. -->
-
-
-
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-
-### Model Sources [optional]
-
-<!-- Provide the basic links for the model. -->
-
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-
-## Uses
-
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
-### Direct Use
-
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
-
-### Downstream Use [optional]
-
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-
-[More Information Needed]
-
-### Out-of-Scope Use
-
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
-
-## Bias, Risks, and Limitations
-
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-
-[More Information Needed]
-
-### Recommendations
-
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-
-## How to Get Started with the Model
-
-Use the code below to get started with the model.
-
-[More Information Needed]
-
-## Training Details
-
-### Training Data
-
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
-
-### Training Procedure
-
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
-
-
-#### Training Hyperparameters
+---
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+# HateBERTimbau
 
+**HateBERTimbau** is a foundation, large language model for European **Portuguese** from **Portugal** for Hate Speech content.
 
-## Evaluation
+It is an **encoder** of the BERT family, based on the neural architecture Transformer and 
+developed over the [BERTimbau](https://huggingface.co/neuralmind/bert-large-portuguese-cased) model, retrained on a dataset of 229,103 tweets specifically focused on potential hate speech.
 
-<!-- This section describes the evaluation protocols and provides the results. -->
+## Model Description
 
-### Testing Data, Factors & Metrics
+- **Developed by:** [kNOwHATE: kNOwing online HATE speech: knowledge + awareness = TacklingHate](https://knowhate.eu)
+- **Funded by:** [European Union](https://ec.europa.eu/info/funding-tenders/opportunities/portal/screen/opportunities/topic-details/cerv-2021-equal)
+- **Model type:** Transformer-based model retrained for Hate Speech in Portuguese social media text
+- **Language:** Portuguese
+- **Retrained from model:** [neuralmind/bert-large-portuguese-cased](https://huggingface.co/neuralmind/bert-large-portuguese-cased)
 
-#### Testing Data
+Several models were developed by fine-tuning Base HateBERTimbau for Hate Speech detection present in the table bellow:
 
-<!-- This should link to a Dataset Card if possible. -->
+| HateBERTimbau's Family of Models                                                                             | 
+|---------------------------------------------------------------------------------------------------------|
+| [**HateBERTimbau YouTube**](https://huggingface.co/knowhate/HateBERTimbau-youtube)        |
+| [**HateBERTimbau Twitter**](https://huggingface.co/knowhate/HateBERTimbau-twitter)        |
+| [**HateBERTimbau YouTube+Twitter**](https://huggingface.co/knowhate/HateBERTimbau-yt-tt)|
 
-[More Information Needed]
+# Uses
 
-#### Factors
+You can use this model directly with a pipeline for masked language modeling:
 
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+```python
+from transformers import pipeline
+unmasker = pipeline('fill-mask', model='knowhate/HateBERTimbau')
 
-[More Information Needed]
+unmasker("Os [MASK] são todos uns animais, deviam voltar para a sua terra.")
 
-#### Metrics
+[{'score': 0.6771652698516846,
+  'token': 12714,
+  'token_str': 'africanos',
+  'sequence': 'Os africanos são todos uns animais, deviam voltar para a sua terra.'},
+ {'score': 0.08679857850074768,
+  'token': 15389,
+  'token_str': 'homossexuais',
+  'sequence': 'Os homossexuais são todos uns animais, deviam voltar para a sua terra.'},
+ {'score': 0.03806231543421745,
+  'token': 4966,
+  'token_str': 'portugueses',
+  'sequence': 'Os portugueses são todos uns animais, deviam voltar para a sua terra.'},
+ {'score': 0.035253893584012985,
+  'token': 16773,
+  'token_str': 'Portugueses',
+  'sequence': 'Os Portugueses são todos uns animais, deviam voltar para a sua terra.'},
+ {'score': 0.023521048948168755,
+  'token': 8618,
+  'token_str': 'brancos',
+  'sequence': 'Os brancos são todos uns animais, deviam voltar para a sua terra.'}]
 
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+```
 
-[More Information Needed]
+Or this model can be used by fine-tuning it for a specific task/dataset:
 
-### Results
+```python
+from transformers import AutoTokenizer, AutoModelForSequenceClassification, TrainingArguments, Trainer
+from datasets import load_dataset
 
-[More Information Needed]
+tokenizer = AutoTokenizer.from_pretrained("knowhate/HateBERTimbau")
+model = AutoModelForSequenceClassification.from_pretrained("knowhate/HateBERTimbau")
+dataset = load_dataset("knowhate/youtube-train")
 
-#### Summary
+def tokenize_function(examples):
+    return tokenizer(examples["sentence1"], examples["sentence2"], padding="max_length", truncation=True)
 
+tokenized_datasets = dataset.map(tokenize_function, batched=True)
 
+training_args = TrainingArguments(output_dir="hatebertimbau", evaluation_strategy="epoch")
+trainer = Trainer(
+    model=model,
+    args=training_args,
+    train_dataset=tokenized_datasets["train"],
+    eval_dataset=tokenized_datasets["validation"],
+)
 
-## Technical Specifications [optional]
+trainer.train()
 
-### Model Architecture and Objective
+```
 
-[More Information Needed]
+# Training
 
-### Compute Infrastructure
+## Data
 
-[More Information Needed]
+229,103 tweets associated with offensive content were used to retrain the base model.
 
-#### Hardware
+## Training Hyperparameters
 
-[More Information Needed]
+- Batch Size: 4 samples
+- Epochs: 100
+- Learning Rate: 5e-5 with Adam optimizer
+- Maximum Sequence Length: 512 sentence pieces
 
-#### Software
+# Testing
 
-[More Information Needed]
+## Data
 
-## Citation [optional]
+We used two different datasets for testing, one for YouTube comments [here](https://huggingface.co/datasets/knowhate/youtube-test) and another for Tweets [here](https://huggingface.co/datasets/knowhate/twitter-test).
 
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+## Hate Speech Classification Results (with no fine-tuning)
 
-**BibTeX:**
+| Dataset         | Precision  | Recall    | F1-score     |
+|:----------------|:-----------|:----------|:-------------|
+| **YouTube**     | 0.928      | 0.108     | **0.193**    | 
+| **Twitter**     | 0.686      | 0.211     | **0.323**    |
 
-[More Information Needed]
+# BibTeX Citation
 
-**APA:**
+``` latex
+@mastersthesis{Matos-Automatic-Hate-Speech-Detection-in-Portuguese-Social-Media-Text,
+title = {{Automatic Hate Speech Detection in Portuguese Social Media Text}},
+author = {Matos, Bernardo Cunha},
+month = nov,
+year = {2022},
+abstract = {{Online Hate Speech (HS) has been growing dramatically on social media and its uncontrolled spread has motivated researchers to develop a diversity of methods for its automated detection. However, the detection of online HS in Portuguese still merits further research. To fill this gap, we explored different models that proved to be successful in the literature to address this task. In particular, we have explored models that use the BERT architecture. Beyond testing single-task models we also explored multitask models that use the information on other related categories to learn HS. To better capture the semantics of this type of texts, we developed HateBERTimbau, a retrained version of BERTimbau more directed to social media language including potential HS targeting African descent, Roma, and LGBTQI+ communities. The performed experiments were based on CO-HATE and FIGHT, corpora of social media messages posted by the Portuguese online community that were labelled regarding the presence of HS among other categories.
+The results achieved show the importance of considering the annotator's agreement on the data used to develop HS detection models. Comparing different subsets of data used for the training of the models it was shown that, in general, a higher agreement on the data leads to better results.
+HATEBERTimbau consistently outperformed BERTimbau on both datasets confirming that further pre-training of BERTimbau was a successful strategy to obtain a language model more suitable for online HS detection in Portuguese. 
+The implementation of target-specific models, and multitask learning have shown potential in obtaining better results.}},
+language = {eng},
+copyright = {embargoed-access},
+}
+```
 
-[More Information Needed]
+# Acknowledgements
+
+This work was funded in part by the European Union under Grant CERV-2021-EQUAL (101049306).
+However the views and opinions expressed are those of the author(s) only and do not necessarily reflect those of the European Union or Knowhate Project.
+Neither the European Union nor the Knowhate Project can be held responsible.