README.md

---
library_name: transformers
license: cc-by-nc-sa-4.0
language:
- fr
base_model: OpenLLM-France/Claire-7B-0.1
---

# Model Card for Claire-7B-FR-Instruct

<!-- Provide a quick summary of what the model is/does. -->



## Model Details

This is the instruction-finetuned model based on [OpenLLM-France/Claire-7B-0.1](https://huggingface.co/OpenLLM-France/Claire-7B-0.1), using the [Vigogne dataset](https://github.com/bofenghuang/vigogne).
Note: This is not a chat model. The finetuning was done on instruction-following data, and the model should be used with the template as shown in "How to Get Started with the Model".

- **Developed by:** LINAGORA with the support of OpenLLM-France
- **Language(s) (NLP):** French
- **License:** CC-BY-NC-SA 4.0
- **Finetuned from model: [OpenLLM-France/Claire-7B-0.1](https://huggingface.co/OpenLLM-France/Claire-7B-0.1)


## Uses

The base model, [Claire-7B-0.1](https://huggingface.co/OpenLLM-France/Claire-7B-0.1), results from continuing the pretraining of [Falcon-7B](https://huggingface.co/tiiuae/falcon-7b) on French conversation transcripts and theater plays. The idea was to attune the base model to features of spontaneous conversation so that it could be more efficiently fine-tuned for downstream tasks requiring understanding of spoken conversation.

This instruction-finetuned model serves as a first level of fine-tuning for such tasks. It is designed to provide detailed responses to user instructions. It can be used for generating natural language responses, content creation, answering queries, and other instruction-based tasks.


## Bias, Risks, and Limitations

This model may reflect biases present in the data it was trained on, potentially leading to unintended or biased responses. 

## How to Get Started with the Model

Use the code below to get started with the model.

```python
import transformers
import torch

model_name = "OpenLLM-France/Claire-7B-FR-Instruct-0.1"

tokenizer = transformers.AutoTokenizer.from_pretrained(model_name)
model = transformers.AutoModelForCausalLM.from_pretrained(model_name,
    device_map="auto",
    torch_dtype=torch.bfloat16,
    load_in_4bit=True                          # For efficient inference, if supported by the GPU card
)

pipeline = transformers.pipeline("text-generation", model=model, tokenizer=tokenizer)
generation_kwargs = dict(
    num_return_sequences=1,                    # Number of variants to generate.
    return_full_text= False,                   # Do not include the prompt in the generated text.
    max_new_tokens=200,                        # Maximum length for the output text.
    do_sample=True, top_k=10, temperature=1.0, # Sampling parameters.
    pad_token_id=tokenizer.eos_token_id,       # Just to avoid a harmless warning.
)

prompt = "Utilisateur: {}\n\nAssistant: ".format(
    "Qui était le président Français en 1995 ?"
)

completions = pipeline(prompt, **generation_kwargs)
for completion in completions:
    print(prompt + " […]" + completion['generated_text'])
```

## Training Details

### Training Data

The model was finetuned on the [Vigogne dataset](https://github.com/bofenghuang/vigogne), which is a cleaned version of the [Alpaca dataset](https://huggingface.co/datasets/tatsu-lab/alpaca), translated by `gpt-3.5-turbo`.

### Training Procedure

The model was finetuned using LoRA.

#### Training Hyperparameters

```
lora_rank: 8
lora_alpha: 16
lora_dropout: 0.05
lora_bias: none
learning_rate: 0.0001
lora_target_modules: ['query_key_value', 'dense_h_to_4h', 'dense_4h_to_h', 'dense'] │
lora_task_type: CAUSAL_LM
num_train_epochs: 1
```