---
license: mit
language:
- ar
metrics:
- accuracy
pipeline_tag: summarization
library_name: PyTorch
tags:
- PyTorch
- Arabic
- Abstractive-Summarization
- 174M
- Scratch
- Base
---
# Arab Bart
Implemented the [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
](https://arxiv.org/abs/1910.13461) paper from scratch using `PyTorch` for an abstractive summarization task in Arabic.

>[!IMPORTANT]
> The model inferenc is not ready, i mean you can't loading it directly from the `Transformers` library.
>
> As soon as possible i will create an inference API, and integrate the model with the Transformers library.
>

## Goal

Reproduce the BART model from scratch to understand its architecture in depth, using the minimum available resources.

## Size

The model size: `174M parameters`.

## Task
Abstractive Summarization in Arabic. 

## Data
The dataset used is the [XL-Sum(Arabic Subset)](https://github.com/csebuetnlp/xl-sum?tab=readme-ov-file#:~:text=Arabic,Download) dataset. I chose this dataset because it's well-suited for our task. Additionally, it's written in pure Arabic, which makes it the best choice. The original source: [BBC Arabic](https://www.bbc.com/arabic).

- Features (columns):
  - text: the full text (source sequences).
  - summary: the summary of the text (target sequences).
    
- Size:
  - train: `32,473 rows`.
  - validation: `4689 rows`.
  - test: `4689 rows`.

## Results

| Epoch | Loss(train) | Loss(validation) | Epoch Time (hours) | Training Time (hours) |  Device  |
|:-----:|:-----------:|:----------------:|:------------------:|:---------------------:|:--------:|
|   1   |    10.03    |       9.72       |        0.23        |          1.1          | 1 x L4OS |
|   2   |    9.61     |       9.44       |        0.22        |          1.1          | 1 x L4OS |
|   3   |    9.36     |       9.22       |        0.22        |          1.1          | 1 x L4OS |
|   4   |    9.16     |       9.05       |        0.22        |          1.1          | 1 x L4OS |
|   5   |    9.01     |       8.92       |        0.22        |          1.1          | 1 x L4OS |


## License

This model is licensed under the `MIT` License.