license: mit
NB Linguistic Quality Regressor
Introduction
This model is designed to rate the quality of Norwegian training corpora based on linguistic quality. It predicts a continuous score (float from 0 to 5), assessing the linguistic quality of Norwegian texts. The model is inspired by the classifiers used in the FineWeb project and is trained mainly on Norwegian content.
Model Architecture
It is trained on top of the nb-bert-base model and utilizes code from CosmoPedia.
Training Data
The dataset used for training is derived from GlotCC and has been annotated using Gemini 1.5 Flash.
Purpose
The performance of large language models (LLMs) heavily depends on the quality and size of their pretraining datasets. This regressor aims to assess and enhance the linguistic quality of Norwegian textual data, contributing to better-performing Norwegian LLMs.
This model is part of a pair; the other is the NB Education Quality Regressor, which focuses on educational content.
Using the Model
For convenience we also provide the run_regressor_bert.py script. This is also based on run_edu_bert.py from Cosmopedia. You can modify this script to annotate HuggingFace datasets directly. Cosmopedia also provides slurm-scripts here. We have not included these since we have had the opportunity to test them.
Training and Evaluation Procedure
The following command where used for training. Please note that train_regressor_bert.py has a few minor changes to the original train_edu_bert.py:
python train_regressor_bert.py --base_model_name="NbAiLab/nb-bert-base" --dataset_name="user/linguistic-annotations" --target_column="score" --checkpoint_dir="/home/user/checkpoints/scandinavian_bert/"
The following script where used for evaluation.
python eval_regressor_bert.py --checkpoint_dir="/home/user/checkpoints/scandinavian_bert/final/" --dataset_name="user/linguistic-annotations"
Classification Report
| Class | Precision | Recall | F1-score | Support |
|---|---|---|---|---|
| 0 | 0.84 | 0.60 | 0.70 | 12209 |
| 1 | 0.70 | 0.72 | 0.71 | 24316 |
| 2 | 0.41 | 0.49 | 0.44 | 10499 |
| 3 | 0.38 | 0.51 | 0.43 | 5833 |
| 4 | 0.10 | 0.24 | 0.14 | 1342 |
| 5 | 0.87 | 0.39 | 0.54 | 5656 |
Overall Metrics
| Metric | Value |
|---|---|
| Accuracy | 0.59 |
| Macro Avg | |
| - Precision | 0.55 |
| - Recall | 0.49 |
| - F1-score | 0.50 |
| Weighted Avg | |
| - Precision | 0.65 |
| - Recall | 0.59 |
| - F1-score | 0.61 |
| Support | 59855 |
Confusion Matrix
| Predicted 0 | Predicted 1 | Predicted 2 | Predicted 3 | Predicted 4 | Predicted 5 | |
|---|---|---|---|---|---|---|
| Actual 0 | 7318 | 4278 | 529 | 63 | 19 | 2 |
| Actual 1 | 1364 | 17602 | 4414 | 785 | 135 | 16 |
| Actual 2 | 38 | 2615 | 5130 | 2289 | 369 | 58 |
| Actual 3 | 10 | 333 | 1726 | 2952 | 664 | 148 |
| Actual 4 | 3 | 83 | 350 | 476 | 324 | 106 |
| Actual 5 | 6 | 98 | 479 | 1205 | 1639 | 2229 |
Evaluation Metrics
| Metric | Value |
|---|---|
| Eval Loss | 0.673861563205719 |
| Eval Precision | 0.5502142676492386 |
| Eval Recall | 0.49225148166352145 |
| Eval F1 Macro | 0.49616318856882935 |
| Eval Accuracy | 0.5940188789574806 |
| Eval Runtime | 285.9726 |
| Eval Samples per Second | 209.303 |
| Eval Steps per Second | 3.273 |
| Epoch | 19.96 |
Training Runtime
| Metric | Value |
|---|---|
| Train Runtime | 105056.8322 |
| Train Samples per Second | 102.552 |
| Train Steps per Second | 1.603 |
| Train Loss | 0.6785072675819606 |
| Epoch | 20.0 |
Run Summary
| Metric | Value |
|---|---|
| Eval Accuracy | 0.59402 |
| Eval F1 Macro | 0.49616 |
| Eval Loss | 0.67386 |
| Eval Precision | 0.55021 |
| Eval Recall | 0.49225 |
| Eval Runtime | 285.9726 |
| Eval Samples per Second | 209.303 |
| Eval Steps per Second | 3.273 |
| Total FLOPs | 2.8346790572921083e+18 |
| Train Epoch | 20.0 |
| Train Global Step | 168360 |
| Train Grad Norm | 2.77268 |
| Train Learning Rate | 0.0 |
| Train Loss | 0.6201 |
| Train Loss (Final) | 0.67851 |
| Train Runtime | 105056.8322 |
| Train Samples per Second | 102.552 |
| Train Steps per Second | 1.603 |
Citing & Authors
The model was trained and documentation written by Per Egil Kummervold.