--- base_model: meta-llama/meta-llama-3.1-8b-instruct language: - am - ha - ig - rw - st - sn - so - sw - xh - yo - zu - en - fr license: apache-2.0 tags: - text-generation-inference - transformers - unsloth - llama - trl datasets: - masakhane/african-ultrachat --- # Llama-3.1-8B Instruct African Ultrachat - **Developed by:** vutuka - **License:** apache-2.0 - **Finetuned from model :** meta-llama/meta-llama-3.1-8b-instruct - **Max Content Length :** `8192` - **Max Steps :** `800` - **Training Time :** `02h-22min-08s` - **Setup :** - `1 x RTX A6000` - `16 vCPU` - `58 GB RAM` - `150 GB Storage` - **Fine Tuned Language :** - `Amharic` - `Hausa` - `Igbo` - `Kinyarwanda` - `Southern Sotho` - `Shona` - `Somali` - `Swahili` - `Xhosa` - `Yoruba` - `Zulu` - `English` - `French` ## Introducing Llama 3.1-8B Instruct Fine-Tuned on the Masakhane African UltraChat Dataset We are excited to announce the fine-tuned version of the Llama 3.1-8B Instruct model, which has been trained on the Masakhane African UltraChat dataset. This fine-tuning leverages the robust architecture of the Llama 3.1 model, designed for high-performance multilingual tasks and long context processing, to enhance its capabilities in understanding and generating responses in African languages. #### Model Overview **Llama 3.1-8B Instruct** is part of the Llama 3 family, developed by Meta. It features an optimized transformer architecture and supports multiple languages, including English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai. This model variant is particularly suited for instruction-tuned tasks, making it ideal for dialogue and assistant-like applications. #### Training and Fine-Tuning The model was fine-tuned using the Masakhane African UltraChat dataset, which is a diverse and extensive collection of conversational data aimed at promoting and enhancing NLP capabilities for African languages. The fine-tuning process involved supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF), ensuring the model aligns well with human preferences for helpfulness and safety. ```py trainer = SFTTrainer( model = model, tokenizer = tokenizer, train_dataset = shuffled_dataset, dataset_text_field = "text", max_seq_length = max_seq_length, dataset_num_proc = 2, packing = False, # Can make training 5x faster for short sequences. args = TrainingArguments( per_device_train_batch_size = 2, gradient_accumulation_steps = 4, warmup_steps = 5, max_steps = 800, do_eval=True, learning_rate = 3e-4, log_level="debug", #fp16 = not is_bfloat16_supported(), bf16 = True, logging_steps = 10, optim = "adamw_8bit", weight_decay = 0.01, lr_scheduler_type = "linear", seed = 3407, output_dir = "outputs", report_to='wandb', warmup_ratio=0.3, ), ) ``` #### Performance and Capabilities The fine-tuned Llama 3.1-8B model demonstrates improved performance in understanding and generating text in African languages, providing accurate and contextually appropriate responses. It is designed to handle various conversational tasks, from casual dialogue to more complex inquiries, making it a valuable tool for applications targeting African language users. #### Key Features - **Multilingual Support**: Enhanced capabilities in multiple languages, including African languages. - **Long Context Handling**: Supports up to 128k tokens, making it suitable for long-form conversations. - **Instruction-Tuned**: Optimized for generating accurate and helpful responses based on user instructions. - **High Performance**: Utilizes advanced techniques like Grouped-Query Attention (GQA) for improved scalability and efficiency. ## Tokenizer & Chat Format ```py from unsloth.chat_templates import get_chat_template tokenizer = get_chat_template( tokenizer, chat_template = "llama-3", # Supports zephyr, chatml, mistral, llama, alpaca, vicuna, vicuna_old, unsloth mapping={ "role": "role", "content": "content", "user": "", "assistant": "", } ) def formatting_prompts_func(examples): convos = examples["messages"] texts = [tokenizer.apply_chat_template(convo, tokenize = False, add_generation_prompt = False) for convo in convos] return { "text" : texts, } pass ``` ## Inference with Unsloth ```py def chat_llama3_african_ultrachat(message: str, context: str): FastLanguageModel.for_inference(model) # Enable native 2x faster inference messages = [ {"role": "system", "content": context}, {"role": "user", "content": message}, ] inputs = tokenizer.apply_chat_template( messages, tokenize = True, add_generation_prompt = True, # Must add for generation return_tensors = "pt", ).to("cuda") from transformers import TextStreamer text_streamer = TextStreamer(tokenizer) #_ = model.generate(input_ids = inputs, streamer = text_streamer, max_new_tokens = 1024, use_cache = True) output = model.generate(input_ids = inputs, max_new_tokens = 1024, use_cache = True) generated_text = tokenizer.decode(output[0], skip_special_tokens=True) # Extract the assistant's message user_marker = "user" assistant_marker = "assistant" response_start = generated_text.find(assistant_marker) + len(assistant_marker) response_end = generated_text.find(user_marker, response_start) if response_end == -1: response = generated_text[response_start:].strip() else: response = generated_text[response_start:response_end].strip() return response ``` ```py chat_llama3_african_ultrachat( message="Habari !", context="Wewe ni wakala wa mtandaoni anayesaidia ambaye hujibu maswali kwa upole na heshima." ) ``` ```txt <|begin_of_text|><|start_header_id|>user<|end_header_id|> Wewe ni wakala wa mtandaoni anayesaidia ambaye hujibu maswali kwa upole na heshima.<|eot_id|><|start_header_id|>user<|end_header_id|> Habari!<|eot_id|><|start_header_id|>assistant<|end_header_id|> Habari yako? Je, unatafuta ushauri au maelezo kuhusu jambo maalum? Ni furaha yangu kusaidia.<|eot_id|> ``` ## Inference with Unsloth Chat (`new`) - Run our code in a `T4` and try the model. ```sh #@title ↙️ Press ▶ to start 🦥 Unsloth Studio Chat for Gemma-2 2b Instruct # Unsloth Studio # Copyright (C) 2024-present the Unsloth AI team. All rights reserved. # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU Affero General Public License as published # by the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU Affero General Public License for more details. # You should have received a copy of the GNU Affero General Public License # along with this program. If not, see . !git clone https://github.com/unslothai/studio > /dev/null 2>&1 with open("studio/unsloth_studio/chat.py", "r") as chat_module: code = chat_module.read().replace( 'MODEL_NAME = "vutuka/Llama-3.1-8B-Instruct-African-Ultrachat"', 'MODEL_NAME = "unsloth/gemma-2-2b-it-bnb-4bit"', ) exec(code) ``` - Change the `chat.py` ```py # Unsloth Studio # Copyright (C) 2024-present the Unsloth AI team. All rights reserved. # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU Affero General Public License as published # by the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU Affero General Public License for more details. # You should have received a copy of the GNU Affero General Public License # along with this program. If not, see . from IPython.display import clear_output import subprocess import os os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1" os.environ["GRADIO_ANALYTICS_ENABLED"] = "False" MODEL_NAME = "vutuka/Llama-3.1-8B-Instruct-African-Ultrachat" print("Installing packages for 🦥 Unsloth Studio ... Please wait 1 minute ...") install_first = [ "pip", "install", "huggingface_hub[hf_transfer]", ] install_first = subprocess.Popen(install_first) install_first.wait() install_second = [ "pip", "install", "gradio", "unsloth[colab-new]@git+https://github.com/unslothai/unsloth.git", ] install_second = subprocess.Popen(install_second) from huggingface_hub import snapshot_download import warnings warnings.filterwarnings(action = "ignore", category = UserWarning, module = "torch") warnings.filterwarnings(action = "ignore", category = UserWarning, module = "huggingface_hub") warnings.filterwarnings(action = "ignore", category = FutureWarning, module = "huggingface_hub") warnings.filterwarnings(action = "ignore", category = RuntimeWarning, module = "subprocess") warnings.filterwarnings(action = "ignore", category = UserWarning, module = "transformers") warnings.filterwarnings(action = "ignore", category = FutureWarning, module = "accelerate") warnings.filterwarnings(action = "ignore", category = RuntimeWarning, module = "multiprocessing") warnings.filterwarnings(action = "ignore", category = RuntimeWarning, module = "multiprocess") from huggingface_hub.utils import disable_progress_bars disable_progress_bars() snapshot_download(repo_id = MODEL_NAME, repo_type = "model") install_second.wait() install_dependencies = [ "pip", "install", "--no-deps", "xformers<0.0.27", "trl<0.9.0", "peft", "accelerate", "bitsandbytes", ] install_dependencies = subprocess.Popen(install_dependencies) install_dependencies.wait() clear_output() from contextlib import redirect_stdout import io import logging logging.getLogger("transformers.utils.hub").setLevel(logging.CRITICAL+1) print("Loading model ... Please wait 1 more minute! ...") with redirect_stdout(io.StringIO()): from unsloth import FastLanguageModel import torch model, tokenizer = FastLanguageModel.from_pretrained( model_name = MODEL_NAME, max_seq_length = None, dtype = None, load_in_4bit = True, ) FastLanguageModel.for_inference(model) pass clear_output() import gradio gradio.strings.en["SHARE_LINK_DISPLAY"] = "" from transformers import TextIteratorStreamer, StoppingCriteria, StoppingCriteriaList from threading import Thread class StopOnTokens(StoppingCriteria): def __init__(self, stop_token_ids): self.stop_token_ids = tuple(set(stop_token_ids)) pass def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs) -> bool: return input_ids[0][-1].item() in self.stop_token_ids pass pass def async_process_chatbot(message, history): eos_token = tokenizer.eos_token stop_on_tokens = StopOnTokens([eos_token,]) text_streamer = TextIteratorStreamer(tokenizer, skip_prompt = True) # From https://www.gradio.app/guides/creating-a-chatbot-fast history_transformer_format = history + [[message, ""]] messages = [] for item in history_transformer_format: messages.append({"role": "user", "content": item[0]}) messages.append({"role": "assistant", "content": item[1]}) pass # Remove last assistant and instead use add_generation_prompt messages.pop(-1) input_ids = tokenizer.apply_chat_template( messages, add_generation_prompt = True, return_tensors = "pt", ).to("cuda", non_blocking = True) # Add stopping criteria - will not output EOS / EOT generation_kwargs = dict( input_ids = input_ids, streamer = text_streamer, max_new_tokens = 1024, stopping_criteria = StoppingCriteriaList([stop_on_tokens,]), temperature = 0.7, do_sample = True, ) thread = Thread(target = model.generate, kwargs = generation_kwargs) thread.start() # Yield will save the output to history! generated_text = "" for new_text in text_streamer: if new_text.endswith(eos_token): new_text = new_text[:len(new_text) - len(eos_token)] generated_text += new_text yield generated_text pass pass studio_theme = gradio.themes.Soft( primary_hue = "teal", ) scene = gradio.ChatInterface( async_process_chatbot, chatbot = gradio.Chatbot( height = 325, label = "Unsloth Studio Chat", ), textbox = gradio.Textbox( placeholder = "Message Unsloth Chat", container = False, ), title = None, theme = studio_theme, examples = None, cache_examples = False, retry_btn = None, undo_btn = "Remove Previous Message", clear_btn = "Restart Entire Chat", ) scene.launch(quiet = True) ``` This llama model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library. [](https://github.com/unslothai/unsloth)