Quantization made by Richard Erkhov. [Github](https://github.com/RichardErkhov) [Discord](https://discord.gg/pvy7H8DZMG) [Request more models](https://github.com/RichardErkhov/quant_request) llama-3-8b-gpt-4o-ru1.0 - GGUF - Model creator: https://huggingface.co/ruslandev/ - Original model: https://huggingface.co/ruslandev/llama-3-8b-gpt-4o-ru1.0/ | Name | Quant method | Size | | ---- | ---- | ---- | | [llama-3-8b-gpt-4o-ru1.0.Q2_K.gguf](https://huggingface.co/RichardErkhov/ruslandev_-_llama-3-8b-gpt-4o-ru1.0-gguf/blob/main/llama-3-8b-gpt-4o-ru1.0.Q2_K.gguf) | Q2_K | 2.96GB | | [llama-3-8b-gpt-4o-ru1.0.IQ3_XS.gguf](https://huggingface.co/RichardErkhov/ruslandev_-_llama-3-8b-gpt-4o-ru1.0-gguf/blob/main/llama-3-8b-gpt-4o-ru1.0.IQ3_XS.gguf) | IQ3_XS | 3.28GB | | [llama-3-8b-gpt-4o-ru1.0.IQ3_S.gguf](https://huggingface.co/RichardErkhov/ruslandev_-_llama-3-8b-gpt-4o-ru1.0-gguf/blob/main/llama-3-8b-gpt-4o-ru1.0.IQ3_S.gguf) | IQ3_S | 3.43GB | | [llama-3-8b-gpt-4o-ru1.0.Q3_K_S.gguf](https://huggingface.co/RichardErkhov/ruslandev_-_llama-3-8b-gpt-4o-ru1.0-gguf/blob/main/llama-3-8b-gpt-4o-ru1.0.Q3_K_S.gguf) | Q3_K_S | 3.41GB | | [llama-3-8b-gpt-4o-ru1.0.IQ3_M.gguf](https://huggingface.co/RichardErkhov/ruslandev_-_llama-3-8b-gpt-4o-ru1.0-gguf/blob/main/llama-3-8b-gpt-4o-ru1.0.IQ3_M.gguf) | IQ3_M | 3.52GB | | [llama-3-8b-gpt-4o-ru1.0.Q3_K.gguf](https://huggingface.co/RichardErkhov/ruslandev_-_llama-3-8b-gpt-4o-ru1.0-gguf/blob/main/llama-3-8b-gpt-4o-ru1.0.Q3_K.gguf) | Q3_K | 3.74GB | | [llama-3-8b-gpt-4o-ru1.0.Q3_K_M.gguf](https://huggingface.co/RichardErkhov/ruslandev_-_llama-3-8b-gpt-4o-ru1.0-gguf/blob/main/llama-3-8b-gpt-4o-ru1.0.Q3_K_M.gguf) | Q3_K_M | 3.74GB | | [llama-3-8b-gpt-4o-ru1.0.Q3_K_L.gguf](https://huggingface.co/RichardErkhov/ruslandev_-_llama-3-8b-gpt-4o-ru1.0-gguf/blob/main/llama-3-8b-gpt-4o-ru1.0.Q3_K_L.gguf) | Q3_K_L | 4.03GB | | [llama-3-8b-gpt-4o-ru1.0.IQ4_XS.gguf](https://huggingface.co/RichardErkhov/ruslandev_-_llama-3-8b-gpt-4o-ru1.0-gguf/blob/main/llama-3-8b-gpt-4o-ru1.0.IQ4_XS.gguf) | IQ4_XS | 4.18GB | | [llama-3-8b-gpt-4o-ru1.0.Q4_0.gguf](https://huggingface.co/RichardErkhov/ruslandev_-_llama-3-8b-gpt-4o-ru1.0-gguf/blob/main/llama-3-8b-gpt-4o-ru1.0.Q4_0.gguf) | Q4_0 | 4.34GB | | [llama-3-8b-gpt-4o-ru1.0.IQ4_NL.gguf](https://huggingface.co/RichardErkhov/ruslandev_-_llama-3-8b-gpt-4o-ru1.0-gguf/blob/main/llama-3-8b-gpt-4o-ru1.0.IQ4_NL.gguf) | IQ4_NL | 4.38GB | | [llama-3-8b-gpt-4o-ru1.0.Q4_K_S.gguf](https://huggingface.co/RichardErkhov/ruslandev_-_llama-3-8b-gpt-4o-ru1.0-gguf/blob/main/llama-3-8b-gpt-4o-ru1.0.Q4_K_S.gguf) | Q4_K_S | 4.37GB | | [llama-3-8b-gpt-4o-ru1.0.Q4_K.gguf](https://huggingface.co/RichardErkhov/ruslandev_-_llama-3-8b-gpt-4o-ru1.0-gguf/blob/main/llama-3-8b-gpt-4o-ru1.0.Q4_K.gguf) | Q4_K | 4.58GB | | [llama-3-8b-gpt-4o-ru1.0.Q4_K_M.gguf](https://huggingface.co/RichardErkhov/ruslandev_-_llama-3-8b-gpt-4o-ru1.0-gguf/blob/main/llama-3-8b-gpt-4o-ru1.0.Q4_K_M.gguf) | Q4_K_M | 4.58GB | | [llama-3-8b-gpt-4o-ru1.0.Q4_1.gguf](https://huggingface.co/RichardErkhov/ruslandev_-_llama-3-8b-gpt-4o-ru1.0-gguf/blob/main/llama-3-8b-gpt-4o-ru1.0.Q4_1.gguf) | Q4_1 | 4.78GB | | [llama-3-8b-gpt-4o-ru1.0.Q5_0.gguf](https://huggingface.co/RichardErkhov/ruslandev_-_llama-3-8b-gpt-4o-ru1.0-gguf/blob/main/llama-3-8b-gpt-4o-ru1.0.Q5_0.gguf) | Q5_0 | 5.21GB | | [llama-3-8b-gpt-4o-ru1.0.Q5_K_S.gguf](https://huggingface.co/RichardErkhov/ruslandev_-_llama-3-8b-gpt-4o-ru1.0-gguf/blob/main/llama-3-8b-gpt-4o-ru1.0.Q5_K_S.gguf) | Q5_K_S | 5.21GB | | [llama-3-8b-gpt-4o-ru1.0.Q5_K.gguf](https://huggingface.co/RichardErkhov/ruslandev_-_llama-3-8b-gpt-4o-ru1.0-gguf/blob/main/llama-3-8b-gpt-4o-ru1.0.Q5_K.gguf) | Q5_K | 5.34GB | | [llama-3-8b-gpt-4o-ru1.0.Q5_K_M.gguf](https://huggingface.co/RichardErkhov/ruslandev_-_llama-3-8b-gpt-4o-ru1.0-gguf/blob/main/llama-3-8b-gpt-4o-ru1.0.Q5_K_M.gguf) | Q5_K_M | 5.34GB | | [llama-3-8b-gpt-4o-ru1.0.Q5_1.gguf](https://huggingface.co/RichardErkhov/ruslandev_-_llama-3-8b-gpt-4o-ru1.0-gguf/blob/main/llama-3-8b-gpt-4o-ru1.0.Q5_1.gguf) | Q5_1 | 5.65GB | | [llama-3-8b-gpt-4o-ru1.0.Q6_K.gguf](https://huggingface.co/RichardErkhov/ruslandev_-_llama-3-8b-gpt-4o-ru1.0-gguf/blob/main/llama-3-8b-gpt-4o-ru1.0.Q6_K.gguf) | Q6_K | 6.14GB | | [llama-3-8b-gpt-4o-ru1.0.Q8_0.gguf](https://huggingface.co/RichardErkhov/ruslandev_-_llama-3-8b-gpt-4o-ru1.0-gguf/blob/main/llama-3-8b-gpt-4o-ru1.0.Q8_0.gguf) | Q8_0 | 7.95GB | Original model description: --- license: llama3 base_model: meta-llama/Meta-Llama-3-8B-Instruct tags: - generated_from_trainer model-index: - name: >- home/ubuntu/llm_training/axolotl/llama3-8b-gpt-4o-ru/output_llama3_8b_gpt_4o_ru results: [] datasets: - ruslandev/tagengo-rus-gpt-4o --- # Llama-3 8B GPT-4o-RU1.0 [[Dataset]](https://huggingface.co/datasets/ruslandev/tagengo-rus-gpt-4o) This model is a fine-tuned version of [meta-llama/Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct). The idea behind this model is to train on a dataset derived from a smaller subset of the [tagengo-gpt4](https://huggingface.co/datasets/lightblue/tagengo-gpt4), but with improved data quality. I tried to achieve higher data quality by prompting GPT-4o, the latest OpenAI's LLM with better multilingual capabilities. The training objective is primarily focused on the Russian language (80% of the training examples). After training for 1 epoch on 2 NVIDIA A100 the model shows promising results on the MT-Bench evaluation benchmark, surpassing GPT-3.5-turbo and being on par with [Suzume](https://huggingface.co/lightblue/suzume-llama-3-8B-multilingual) in Russian language scores, even though the latter is trained on 8x bigger and more diverse dataset. ## How to use The easiest way to use this model on your own computer is to use the GGUF version of this model ([ruslandev/llama-3-8b-gpt-4o-ru1.0-gguf](https://huggingface.co/ruslandev/llama-3-8b-gpt-4o-ru1.0-gguf)) using a program such as [llama.cpp](https://github.com/ggerganov/llama.cpp). If you want to use this model directly with the Huggingface Transformers stack, I recommend using my framework [gptchain](https://github.com/RuslanPeresy/gptchain). ``` git clone https://github.com/RuslanPeresy/gptchain.git cd gptchain pip install -r requirements-train.txt python gptchain.py chat -m ruslandev/llama-3-8b-gpt-4o-ru1.0 \ --chatml true \ -q '[{"from": "human", "value": "Из чего состоит нейронная сеть?"}]' ``` ## Evaluation scores I achieved the following scores on Ru/En MT-Bench: | |meta-llama/Meta-Llama-3-8B-Instruct | ruslandev/llama-3-8b-gpt-4o-ru1.0 | lightblue/suzume-llama-3-8B-multilingual | Nexusflow/Starling-LM-7B-beta | gpt-3.5-turbo | |:----------:|:----------------------------------:|:---------------------------------:|:----------------------------------------:|:-----------------------------:|:-------------:| | Russian 🇷🇺 | NaN | 8.12 | 8.19 | 8.06 | 7.94 | | English 🇺🇸 | 7.98 | 8.01 | 7.73 | 7.92 | 8.26 | ## Training procedure [Built with Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl)
See axolotl config axolotl version: `0.4.1` ```yaml base_model: meta-llama/Meta-Llama-3-8B-Instruct model_type: LlamaForCausalLM tokenizer_type: AutoTokenizer # PreTrainedTokenizerFast load_in_8bit: false load_in_4bit: false strict: false datasets: - path: ruslandev/tagengo-rus-gpt-4o type: sharegpt conversation: llama-3 dataset_prepared_path: /home/ubuntu/llm_training/axolotl/llama3-8b-gpt-4o-ru/prepared_tagengo_rus val_set_size: 0.01 output_dir: /home/ubuntu/llm_training/axolotl/llama3-8b-gpt-4o-ru/output_llama3_8b_gpt_4o_ru sequence_len: 8192 sample_packing: true pad_to_sequence_len: true eval_sample_packing: false use_wandb: false #wandb_project: axolotl #wandb_entity: wandb_entity #wandb_name: llama_3_8b_gpt_4o_ru gradient_accumulation_steps: 2 micro_batch_size: 2 num_epochs: 1 optimizer: paged_adamw_8bit lr_scheduler: cosine learning_rate: 1e-5 train_on_inputs: false group_by_length: false bf16: auto fp16: tf32: false gradient_checkpointing: true gradient_checkpointing_kwargs: use_reentrant: false early_stopping_patience: resume_from_checkpoint: logging_steps: 1 xformers_attention: flash_attention: true warmup_steps: 10 evals_per_epoch: 5 eval_table_size: saves_per_epoch: 1 debug: deepspeed: /home/ubuntu/axolotl/deepspeed_configs/zero2.json weight_decay: 0.0 special_tokens: pad_token: <|end_of_text|> ```

### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 1e-05 - train_batch_size: 2 - eval_batch_size: 2 - seed: 42 - distributed_type: multi-GPU - num_devices: 2 - gradient_accumulation_steps: 2 - total_train_batch_size: 8 - total_eval_batch_size: 4 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: cosine - lr_scheduler_warmup_steps: 10 - num_epochs: 1 ### Training results | Training Loss | Epoch | Step | Validation Loss | |:-------------:|:-----:|:----:|:---------------:| | 1.1347 | 0.016 | 1 | 1.1086 | | 0.916 | 0.208 | 13 | 0.8883 | | 0.8494 | 0.416 | 26 | 0.8072 | | 0.8657 | 0.624 | 39 | 0.7814 | | 0.8077 | 0.832 | 52 | 0.7702 | ### Framework versions - Transformers 4.41.1 - Pytorch 2.2.2+cu121 - Datasets 2.19.1 - Tokenizers 0.19.1