---
license: other
pipeline_tag: text-generation
datasets:
- cosimoiaia/Loquace-102k
language:
- it
---
# Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->
An open-source LLaMa language model of 13b parameters fine-tuned to follow instructions in italian.


### Model Description

This model is an open-source LLM of 13b parameters based on [OpenLLaMA](https://github.com/openlm-research/open_llama), an open-source replica of Meta AI's LLaMA.
The model was fine-tuned in order to follow instructions, as proposed in [Alpaca](https://github.com/tatsu-lab/stanford_alpaca), 
but using [LoRA](https://arxiv.org/pdf/2106.09685.pdf) technique and a bigger dataset of instruction/answers in italian, [cosimoiaia/Loquace-102k](https://huggingface.co/datasets/cosimoiaia/Loquace-102k/viewer/cosimoiaia--Loquace-102k).

This repository contains the model merged with the LoRA adapters obtained in the fine-tuning procedure.


- **Developed by:** Stefano Scotta (stefano.scotta@rai.it)
- **Model type:** LLM fine-tuned to follow instructions
- **Language(s) (NLP):** Italian
- **License:** [More Information Needed]
- **Finetuned from model:** [openlm-research/open_llama_13b](https://huggingface.co/openlm-research/open_llama_13b)


## Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

### Direct Use

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->

[More Information Needed]

### Downstream Use [optional]

<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->

[More Information Needed]

### Out-of-Scope Use

<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->

[More Information Needed]

## Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

[More Information Needed]

### Recommendations

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.

## How to Get Started with the Model

Use the code below to get started with the model.

[More Information Needed]

## Training Details

### Training Data

<!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
The model was fine-tinuned on [cosimoiaia/Loquace-102k](https://huggingface.co/datasets/cosimoiaia/Loquace-102k/viewer/cosimoiaia--Loquace-102k), a dataset of 102k question/answer pairs in italian.


### Training Procedure 

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
The fine-tuning procedure was done using [LoRA](https://arxiv.org/pdf/2106.09685.pdf) approach following closely what done for fine-tuning models like [Alpaca-LoRA](https://github.com/tloen/alpaca-lora).


#### Training Hyperparameters

 **Training setting:**
- train epochs=3,

- learning_rate=3e-4,

- optimizer="adamw_hf"

- mixed precision training: float16 

 **LoRA configuration:**
-  r= 8
  
- lora_alpha=16
  
- target_modules=["q_proj","v_proj"]

- lora_dropout=0.05
  
- bias="none"
  
- task_type=TaskType.CAUSAL_LM
  


[More Information Needed]

## Environmental Impact

<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->

Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).

- **Hardware Type:** 1 NVIDIA A100/40Gb
- **Hours used:** 68
- **Cloud Provider:** Private Infrastructure
- **Carbon Emitted:** 7.34 kg eq. CO2

## Technical Specifications [optional]

### Model Architecture and Objective

[More Information Needed]

### Compute Infrastructure

[More Information Needed]

#### Hardware

[More Information Needed]

#### Software

[More Information Needed]

## Citation [optional]

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->

**BibTeX:**

[More Information Needed]

**APA:**

[More Information Needed]

## Glossary [optional]

<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->

[More Information Needed]

## More Information [optional]

[More Information Needed]

## Model Card Authors [optional]

Stefano Scotta (stefano.scotta@rai.it)

## Model Card Contact

stefano.scotta@rai.it