Kichtral-7B-v0.3: a Mistral-7B Casual LM for French Rap Lyrics

Overview

Kichtral-7B-v0.3 is a Casual Language Model fine-tuned from the Mistral 7B model on french rap lyrics. The training dataset consists of cleaned French verses, with no repetitions, from songs that have at least 10k streams on Spotify. This dataset contains a total of 36M tokens.

This model aims to understand and generate french rap lyrics, making it a valuable tool for research in french slang and music lyrics generation.

Model Details

Kichtral-7B-v0.3 is based on the Mistral 7B v0.3 architecture and has been fine-tuned with the following hyperparameters:

Parameter	Value
Epochs	1
LoRA Rank	64
LoRA Alpha	128
LoRA Dropout	0.1
Learning Rate	1e-4
Learning Scheduler	Cosine

Versions

The model was trained using AWS SageMaker on a single ml.g5.2xlarge instance during 15 hours with the following software versions:

Requirement	Version
Transformers	4.28
PyTorch	2.0
Python	3.10

Installation

Install the required Python libraries:

pip install transformers

Loading the Model

from transformers import AutoTokenizer, AutoModelForCausalLM

# Load the tokenizer and model
tokenizer = AutoTokenizer.from_pretrained("rapminerz/Kichtral-7B-v0.3")
model = AutoModelForCausalLM.from_pretrained("rapminerz/Kichtral-7B-v0.3")

Using the Model

def generate_lyrics(prompt):
    inputs = tokenizer(prompt, return_tensors="pt")
    outputs = model.generate(
        inputs["input_ids"], 
        max_length=300, 
        num_return_sequences=1,
        top_k=10,
        top_p=0.95,
        temperature=1.0,
        repetition_penalty=1.2
    )
    return tokenizer.decode(outputs[0], skip_special_tokens=True)

generate_lyrics("Okay ça fait")
"""
Okay ça fait un moment que tu m'appelles
Sans t'écouter, j'ai dû me tailler
Jusqu'à présent, je sais pas qui t'es mais je peux pas t'oublier
Tu m'as laissé des images dans l'crâne
Quand je repense à ce soir-là
"""

generate_lyrics("Je viens de là où")
"""
Je viens de là où ça tire
Je fais la loi je suis pas le roi
Et je sais que tu penses à moi quand t'as besoin d'aide
Quand y a trop d'ennemis autour de toi qui se mêlent
"""

Purpose and Disclaimer

This model is designed for academic and research purposes only. It is not intended for commercial use. The creators of this model do not endorse or promote any specific views or opinions that may be represented in the dataset.

Please mention @RapMinerz if you use our models

Improvements

This model doesn't totally capture rhymes, another method should be needed to prompt for example rhymes and topics

Contact

For any questions or issues, please contact the repository owner, RapMinerz, at rapminerz.contact@gmail.com.