|
--- |
|
license: other |
|
pipeline_tag: text-generation |
|
datasets: |
|
- cosimoiaia/Loquace-102k |
|
language: |
|
- it |
|
--- |
|
# Model Card for Model ID |
|
|
|
<!-- Provide a quick summary of what the model is/does. --> |
|
An open-source LLaMa language model of 13b parameters fine-tuned to follow instructions in italian. |
|
|
|
|
|
### Model Description |
|
|
|
This model is an open-source LLM of 13b parameters based on [OpenLLaMA](https://github.com/openlm-research/open_llama), an open-source replica of Meta AI's LLaMA. |
|
The model was fine-tuned in order to follow instructions, as proposed in [Alpaca](https://github.com/tatsu-lab/stanford_alpaca), |
|
but using [LoRA](https://arxiv.org/pdf/2106.09685.pdf) technique and a bigger dataset of instruction/answers in italian, [cosimoiaia/Loquace-102k](https://huggingface.co/datasets/cosimoiaia/Loquace-102k/viewer/cosimoiaia--Loquace-102k). |
|
|
|
This repository contains the model merged with the LoRA adapters obtained in the fine-tuning procedure. |
|
|
|
|
|
- **Developed by:** Stefano Scotta (stefano.scotta@rai.it) |
|
- **Model type:** LLM fine-tuned to follow instructions |
|
- **Language(s) (NLP):** Italian |
|
- **License:** [More Information Needed] |
|
- **Finetuned from model:** [openlm-research/open_llama_13b](https://huggingface.co/openlm-research/open_llama_13b) |
|
|
|
|
|
## Uses |
|
|
|
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. --> |
|
|
|
### Direct Use |
|
|
|
<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. --> |
|
|
|
[More Information Needed] |
|
|
|
### Downstream Use [optional] |
|
|
|
<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app --> |
|
|
|
[More Information Needed] |
|
|
|
### Out-of-Scope Use |
|
|
|
<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. --> |
|
|
|
[More Information Needed] |
|
|
|
## Bias, Risks, and Limitations |
|
|
|
<!-- This section is meant to convey both technical and sociotechnical limitations. --> |
|
|
|
[More Information Needed] |
|
|
|
### Recommendations |
|
|
|
<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. --> |
|
|
|
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations. |
|
|
|
## How to Get Started with the Model |
|
|
|
Use the code below to get started with the model. |
|
|
|
[More Information Needed] |
|
|
|
## Training Details |
|
|
|
### Training Data |
|
|
|
<!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. --> |
|
The model was fine-tinuned on [cosimoiaia/Loquace-102k](https://huggingface.co/datasets/cosimoiaia/Loquace-102k/viewer/cosimoiaia--Loquace-102k), a dataset of 102k question/answer pairs in italian. |
|
|
|
|
|
### Training Procedure |
|
|
|
<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. --> |
|
The fine-tuning procedure was done using [LoRA](https://arxiv.org/pdf/2106.09685.pdf) approach following closely what done for fine-tuning models like [Alpaca-LoRA](https://github.com/tloen/alpaca-lora). |
|
|
|
|
|
#### Training Hyperparameters |
|
|
|
**Training setting:** |
|
- train epochs=3, |
|
|
|
- learning_rate=3e-4, |
|
|
|
- optimizer="adamw_hf" |
|
|
|
- mixed precision training: float16 |
|
|
|
**LoRA configuration:** |
|
- r= 8 |
|
|
|
- lora_alpha=16 |
|
|
|
- target_modules=["q_proj","v_proj"] |
|
|
|
- lora_dropout=0.05 |
|
|
|
- bias="none" |
|
|
|
- task_type=TaskType.CAUSAL_LM |
|
|
|
|
|
|
|
[More Information Needed] |
|
|
|
## Environmental Impact |
|
|
|
<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly --> |
|
|
|
Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700). |
|
|
|
- **Hardware Type:** 1 NVIDIA A100/40Gb |
|
- **Hours used:** 68 |
|
- **Cloud Provider:** Private Infrastructure |
|
- **Carbon Emitted:** 7.34 kg eq. CO2 |
|
|
|
## Technical Specifications [optional] |
|
|
|
### Model Architecture and Objective |
|
|
|
[More Information Needed] |
|
|
|
### Compute Infrastructure |
|
|
|
[More Information Needed] |
|
|
|
#### Hardware |
|
|
|
[More Information Needed] |
|
|
|
#### Software |
|
|
|
[More Information Needed] |
|
|
|
## Citation [optional] |
|
|
|
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. --> |
|
|
|
**BibTeX:** |
|
|
|
[More Information Needed] |
|
|
|
**APA:** |
|
|
|
[More Information Needed] |
|
|
|
## Glossary [optional] |
|
|
|
<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. --> |
|
|
|
[More Information Needed] |
|
|
|
## More Information [optional] |
|
|
|
[More Information Needed] |
|
|
|
## Model Card Authors [optional] |
|
|
|
Stefano Scotta (stefano.scotta@rai.it) |
|
|
|
## Model Card Contact |
|
|
|
stefano.scotta@rai.it |
