library_name: peft
base_model: core42/jais-13b
license: mit
datasets:
- arbml/Ashaar_dataset
language:
- ar
metrics:
- perplexity
- bertscore
Model Card for Model ID
Fine-tuned using QLoRA for poem generation task.
Model Description
We utilize Ashaar dataset and fine-tune the model to generate poems.
The input to the model is structred as follows:
'''
### Instruction: Generate a poem based on the following title, and the given era:
### Input: {Title of a poem + poet era}
### Response: {Poem verses}
'''
- Developed by: Abdelrahman ’Boda’ Sadallah, Anastasiia Demidova, Daria Kotova
- Model type: Causal LM
- Language(s) (NLP): Arabic
- Finetuned from model [optional]: core42/jais-13b
Model Sources
Uses
The model is the result of our AI project. If you intend to use it, please, refer to the repo.
Recommendations
For improving stories generation, you can play parameters: temeperature, top_p/top_k, repetition_penalty, etc.
Training Details
Training Data
Link to the dataset on huggungface: https://huggingface.co/datasets/arbml/ashaar.
Evaluation
Testing Data, Factors & Metrics
Test split of the same dataset.
Metrics
We are using perplexity and BERTScore.
Results
Perplexity: 48.3125
BERTScore: 59.33
Training procedure
The following bitsandbytes quantization config was used during training:
- quant_method: bitsandbytes
- load_in_8bit: False
- load_in_4bit: True
- llm_int8_threshold: 6.0
- llm_int8_skip_modules: None
- llm_int8_enable_fp32_cpu_offload: False
- llm_int8_has_fp16_weight: False
- bnb_4bit_quant_type: fp4
- bnb_4bit_use_double_quant: False
- bnb_4bit_compute_dtype: float32
Framework versions
- PEFT 0.6.0.dev0