--- base_model: meta-llama/Meta-Llama-3-8B license: llama3 tags: - axolotl - generated_from_trainer model-index: - name: Egyptian-Arabic-Translator-Llama-3-8B results: [] --- [Built with Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl)
See axolotl config axolotl version: `0.4.1` ```yaml base_model: meta-llama/Meta-Llama-3-8B model_type: LlamaForCausalLM tokenizer_type: AutoTokenizer load_in_8bit: true load_in_4bit: false strict: false datasets: - path: translation-dataset-v3-train.hf type: alpaca train_on_split: train test_datasets: - path: translation-dataset-v3-test.hf type: alpaca split: train dataset_prepared_path: ./last_run_prepared output_dir: ./llama_3_translator hub_model_id: ahmedsamirio/llama_3_translator_v3 sequence_len: 2048 sample_packing: true pad_to_sequence_len: true eval_sample_packing: false adapter: lora lora_r: 32 lora_alpha: 16 lora_dropout: 0.05 lora_target_linear: true lora_fan_in_fan_out: lora_target_modules: - gate_proj - down_proj - up_proj - q_proj - v_proj - k_proj - o_proj wandb_project: en_eg_translator wandb_entity: ahmedsamirio wandb_name: llama_3_en_eg_translator_v3 gradient_accumulation_steps: 4 micro_batch_size: 2 num_epochs: 2 optimizer: paged_adamw_32bit lr_scheduler: cosine learning_rate: 2e-5 train_on_inputs: false group_by_length: false bf16: auto fp16: tf32: false gradient_checkpointing: true early_stopping_patience: resume_from_checkpoint: local_rank: logging_steps: 1 xformers_attention: flash_attention: true warmup_steps: 10 evals_per_epoch: 10 eval_table_size: eval_max_new_tokens: 128 saves_per_epoch: 1 debug: deepspeed: weight_decay: 0.0 fsdp: fsdp_config: special_tokens: pad_token: <|end_of_text|> ```

[Visualize in Weights & Biases](https://wandb.ai/ahmedsamirio/en_eg_translator/runs/hwzxxt0r) # Egyptian Arabic Translator Llama-3 8B This model is a fine-tuned version of [meta-llama/Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B) on the [ahmedsamirio/oasst2-9k-translation](https://huggingface.co/datasets/ahmedsamirio/oasst2-9k-translation) dataset. ## Model description This model is an attempt to create a small translation model from English to Egyptian Arabic. ## Intended uses & limitations - Translating instruction finetuning and text generation datasets ## Inference code ```python from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline tokenizer = AutoTokenizer.from_pretrained("ahmedsamirio/Egyptian-Arabic-Translator-Llama-3-8B") model = AutoModelForCausalLM.from_pretrained("ahmedsamirio/Egyptian-Arabic-Translator-Llama-3-8B") pipe = pipeline(task='text-generation', model=model, tokenizer=tokenizer) en_template = """<|begin_of_text|>Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request. ### Instruction: Translate the following text to English. ### Input: {text} ### Response: """ ar_template = """<|begin_of_text|>Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request. ### Instruction: Translate the following text to Arabic. ### Input: {text} ### Response: """ eg_template = """<|begin_of_text|>Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request. ### Instruction: Translate the following text to Egyptian Arabic. ### Input: {text} ### Response: """ text = """Some habits are known as "keystone habits," and these influence the formation of other habits. \ For example, identifying as the type of person who takes care of their body and is in the habit of exercising regularly, \ can also influence eating better and using credit cards less. In business, \ safety can be a keystone habit that influences other habits that result in greater productivity.[17]""" ar_text = pipe(ar_template.format(text=text), max_new_tokens=256, do_sample=True, temperature=0.3, top_p=0.5) eg_text = pipe(eg_template.format(text=ar_text), max_new_tokens=256, do_sample=True, temperature=0.3, top_p=0.5) print("Original Text:" text) print("\nArabic Translation:", ar_text) print("\nEgyptian Arabic Translation:", eg_text) ``` ## Training and evaluation data [ahmedsamirio/oasst2-9k-translation](https://huggingface.co/datasets/ahmedsamirio/oasst2-9k-translation) ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 2e-05 - train_batch_size: 2 - eval_batch_size: 2 - seed: 42 - gradient_accumulation_steps: 4 - total_train_batch_size: 8 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: cosine - lr_scheduler_warmup_steps: 10 - num_epochs: 2 ### Training results | Training Loss | Epoch | Step | Validation Loss | |:-------------:|:------:|:----:|:---------------:| | 0.9661 | 0.0008 | 1 | 1.3816 | | 0.5611 | 0.1002 | 123 | 0.9894 | | 0.6739 | 0.2004 | 246 | 0.8820 | | 0.5168 | 0.3006 | 369 | 0.8229 | | 0.5582 | 0.4008 | 492 | 0.7931 | | 0.552 | 0.5010 | 615 | 0.7814 | | 0.5129 | 0.6012 | 738 | 0.7591 | | 0.5887 | 0.7014 | 861 | 0.7444 | | 0.6359 | 0.8016 | 984 | 0.7293 | | 0.613 | 0.9018 | 1107 | 0.7179 | | 0.5671 | 1.0020 | 1230 | 0.7126 | | 0.4956 | 1.0847 | 1353 | 0.7034 | | 0.5055 | 1.1849 | 1476 | 0.6980 | | 0.4863 | 1.2851 | 1599 | 0.6877 | | 0.4538 | 1.3853 | 1722 | 0.6845 | | 0.4362 | 1.4855 | 1845 | 0.6803 | | 0.4291 | 1.5857 | 1968 | 0.6834 | | 0.6208 | 1.6859 | 2091 | 0.6830 | | 0.582 | 1.7862 | 2214 | 0.6781 | | 0.5001 | 1.8864 | 2337 | 0.6798 | ### Framework versions - PEFT 0.11.1 - Transformers 4.42.3 - Pytorch 2.1.2+cu118 - Datasets 2.19.1 - Tokenizers 0.19.1