--- language: - en license: apache-2.0 tags: - text-generation-inference - transformers - unsloth - llama - trl base_model: unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit --- # Uploaded model - **Developed by:** Asuncom - **License:** apache-2.0 - **Finetuned from model :** unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit This llama model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library. [

](https://github.com/unslothai/unsloth) ```python !pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git" ``` ```python !pip install --upgrade pip ``` ```python !pip install --no-deps "xformers<0.0.26" "trl<0.9.0" peft accelerate bitsandbytes ``` ```python from unsloth import FastLanguageModel import torch max_seq_length = 2048 # Choose any! We auto support RoPE Scaling internally! dtype = None # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+ load_in_4bit = True # Use 4bit quantization to reduce memory usage. Can be False. # 4bit pre quantized models we support for 4x faster downloading + no OOMs. fourbit_models = [ "unsloth/Meta-Llama-3.1-8B-bnb-4bit", # Llama-3.1 15 trillion tokens model 2x faster! "unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit", "unsloth/Meta-Llama-3.1-70B-bnb-4bit", "unsloth/Meta-Llama-3.1-405B-bnb-4bit", # We also uploaded 4bit for 405b! "unsloth/Mistral-Nemo-Base-2407-bnb-4bit", # New Mistral 12b 2x faster! "unsloth/Mistral-Nemo-Instruct-2407-bnb-4bit", "unsloth/mistral-7b-v0.3-bnb-4bit", # Mistral v3 2x faster! "unsloth/mistral-7b-instruct-v0.3-bnb-4bit", "unsloth/Phi-3-mini-4k-instruct", # Phi-3 2x faster!d "unsloth/Phi-3-medium-4k-instruct", "unsloth/gemma-2-9b-bnb-4bit", "unsloth/gemma-2-27b-bnb-4bit", # Gemma 2x faster! ] # More models at https://huggingface.co/unsloth model, tokenizer = FastLanguageModel.from_pretrained( model_name = "unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit", max_seq_length = max_seq_length, dtype = dtype, load_in_4bit = load_in_4bit, # token = "hf_...", # use one if using gated models like meta-llama/Llama-2-7b-hf ) ``` ```python # ======================================================== # Test before training # ======================================================== alpaca_prompt = """Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request. ### Instruction: {} ### Input: {} ### Response: {}""" FastLanguageModel.for_inference(model) # Enable native 2x faster inference inputs = tokenizer( [ alpaca_prompt.format( "请把现代汉语翻译成古文", # instruction "其品行廉正，所以至死也不放松对自己的要求。", # input "", # output - leave this blank for generation! ) ], return_tensors = "pt").to("cuda") from transformers import TextStreamer text_streamer = TextStreamer(tokenizer) _ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 128) ``` ```python model = FastLanguageModel.get_peft_model( model, r = 16, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128 target_modules = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj",], lora_alpha = 16, lora_dropout = 0, # Supports any, but = 0 is optimized bias = "none", # Supports any, but = "none" is optimized # [NEW] "unsloth" uses 30% less VRAM, fits 2x larger batch sizes! use_gradient_checkpointing = "unsloth", # True or "unsloth" for very long context random_state = 3407, use_rslora = False, # We support rank stabilized LoRA loftq_config = None, # And LoftQ ) ``` ```python alpaca_prompt = """Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request. ### Instruction: {} ### Input: {} ### Response: {}""" EOS_TOKEN = tokenizer.eos_token # Must add EOS_TOKEN def formatting_prompts_func(examples): instructions = examples["instruction"] inputs = examples["input"] outputs = examples["output"] texts = [] for instruction, input, output in zip(instructions, inputs, outputs): # Must add EOS_TOKEN, otherwise your generation will go on forever! text = alpaca_prompt.format(instruction, input, output) + EOS_TOKEN texts.append(text) return { "text" : texts, } pass from datasets import load_dataset dataset = load_dataset("Asuncom/shiji-qishiliezhuan", split = "train") dataset = dataset.map(formatting_prompts_func, batched = True,) ``` ```python from trl import SFTTrainer from transformers import TrainingArguments from unsloth import is_bfloat16_supported trainer = SFTTrainer( model = model, tokenizer = tokenizer, train_dataset = dataset, dataset_text_field = "text", max_seq_length = max_seq_length, dataset_num_proc = 2, packing = False, # Can make training 5x faster for short sequences. args = TrainingArguments( per_device_train_batch_size = 2, gradient_accumulation_steps = 4, warmup_steps = 5, # num_train_epochs = 1, # Set this for 1 full training run. max_steps = 100, learning_rate = 2e-4, fp16 = not is_bfloat16_supported(), bf16 = is_bfloat16_supported(), logging_steps = 1, optim = "adamw_8bit", weight_decay = 0.01, lr_scheduler_type = "linear", seed = 3407, output_dir = "outputs", ), ) ``` ```python #@title Show current memory stats gpu_stats = torch.cuda.get_device_properties(0) start_gpu_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3) max_memory = round(gpu_stats.total_memory / 1024 / 1024 / 1024, 3) print(f"GPU = {gpu_stats.name}. Max memory = {max_memory} GB.") print(f"{start_gpu_memory} GB of memory reserved.") ``` ```python import wandb # 初始化一个离线模式的W&B运行 wandb.init(mode="offline", project="asuncom", entity="asuncom") ``` ```python trainer_stats = trainer.train() ``` ```python #@title Show final memory and time stats used_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3) used_memory_for_lora = round(used_memory - start_gpu_memory, 3) used_percentage = round(used_memory /max_memory*100, 3) lora_percentage = round(used_memory_for_lora/max_memory*100, 3) print(f"{trainer_stats.metrics['train_runtime']} seconds used for training.") print(f"{round(trainer_stats.metrics['train_runtime']/60, 2)} minutes used for training.") print(f"Peak reserved memory = {used_memory} GB.") print(f"Peak reserved memory for training = {used_memory_for_lora} GB.") print(f"Peak reserved memory % of max memory = {used_percentage} %.") print(f"Peak reserved memory for training % of max memory = {lora_percentage} %.") ``` ```python # alpaca_prompt = Copied from above FastLanguageModel.for_inference(model) # Enable native 2x faster inference inputs = tokenizer( [ alpaca_prompt.format( "请把现代汉语翻译成古文", # instruction "其品行廉正，所以至死也不放松对自己的要求。", # input "", # output - leave this blank for generation! ) ], return_tensors = "pt").to("cuda") from transformers import TextStreamer text_streamer = TextStreamer(tokenizer) _ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 128) ``` ```python model.save_pretrained("lora_model") # Local saving tokenizer.save_pretrained("lora_model") model.push_to_hub("Asuncom/Llama-3.1-8B-bnb-4bit-shiji", token = "hf_huggingface的密钥NeKb") # Online saving tokenizer.push_to_hub("Asuncom/Llama-3.1-8B-bnb-4bit-shiji", token = "hf_huggingface的密钥saving ``` ```python if False: from unsloth import FastLanguageModel model, tokenizer = FastLanguageModel.from_pretrained( model_name = "lora_model", # YOUR MODEL YOU USED FOR TRAINING max_seq_length = max_seq_length, dtype = dtype, load_in_4bit = load_in_4bit, ) FastLanguageModel.for_inference(model) # Enable native 2x faster inference # alpaca_prompt = You MUST copy from above! inputs = tokenizer( [ alpaca_prompt.format( "What is a famous tall tower in Paris?", # instruction "", # input "", # output - leave this blank for generation! ) ], return_tensors = "pt").to("cuda") from transformers import TextStreamer text_streamer = TextStreamer(tokenizer) _ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 128) ``` ```python # Merge to 16bit if False: model.save_pretrained_merged("model", tokenizer, save_method = "merged_16bit",) if False: model.push_to_hub_merged("Asuncom/Llama-3.1-8B-bnb-4bit-shiji", tokenizer, save_method = "merged_16bit", token = "hf_huggingface的密钥NeKb") # Merge to 4bit if False: model.save_pretrained_merged("model", tokenizer, save_method = "merged_4bit",) if False: model.push_to_hub_merged("Asuncom/Llama-3.1-8B-bnb-4bit-shiji", tokenizer, save_method = "merged_4bit", token = "hf_huggingface的密钥oRA adapters if False: model.save_pretrained_merged("model", tokenizer, save_method = "lora",) if False: model.push_to_hub_merged("Asuncom/Llama-3.1-8B-bnb-4bit-shiji", tokenizer, save_method = "lora", token = "hf_huggingface的密钥 ``` ```python # Save to 8bit Q8_0 if False: model.save_pretrained_gguf("model", tokenizer,) # Remember to go to https://huggingface.co/settings/tokens for a token! # And change hf to your username! if False: model.push_to_hub_gguf("Asuncom/Llama-3.1-8B-bnb-4bit-shiji", tokenizer, token = "") # Save to 16bit GGUF if False: model.save_pretrained_gguf("model", tokenizer, quantization_method = "f16") if False: model.push_to_hub_gguf("Asuncom/Llama-3.1-8B-bnb-4bit-shiji", tokenizer, quantization_method = "f16", token = "") # Save to q4_k_m GGUF if False: model.save_pretrained_gguf("model", tokenizer, quantization_method = "q4_k_m") if True: model.push_to_hub_gguf("Asuncom/Llama-3.1-8B-bnb-4bit-shiji", tokenizer, quantization_method = "q4_k_m", token = "hf_xxxxx") # Save to multiple GGUF options - much faster if you want multiple! if False: model.push_to_hub_gguf( "Asuncom/Llama-3.1-8B-bnb-4bit-shiji", # Change hf to your username! tokenizer, quantization_method = ["q4_k_m", "q8_0", "q5_k_m",], token = "hf_huggingface的密钥NeKb", # Get a token at https://huggingface.co/settings/tokens ) ``` ```python model.push_to_hub_gguf( "Asuncom/Llama-3.1-8B-bnb-4bit-shiji", # Change hf to your username! tokenizer, quantization_method = ["q4_k_m", "q8_0", "q5_k_m",], token = "hf_huggingface的密钥NeKb", # Get a token at https://huggingface.co/settings/tokens ) ``` ``` [ 279/ 292] blk.30.attn_output.weight - [ 4096, 4096, 1, 1], type = f16, converting to q5_K .. size = 32.00 MiB -> 11.00 MiB [ 280/ 292] blk.30.attn_q.weight - [ 4096, 4096, 1, 1], type = f16, converting to q5_K .. size = 32.00 MiB -> 11.00 MiB [ 281/ 292] blk.30.attn_v.weight - [ 4096, 1024, 1, 1], type = f16, converting to q6_K .. size = 8.00 MiB -> 3.28 MiB [ 282/ 292] blk.31.ffn_gate.weight - [ 4096, 14336, 1, 1], type = f16, converting to q5_K .. size = 112.00 MiB -> 38.50 MiB [ 283/ 292] blk.31.ffn_up.weight - [ 4096, 14336, 1, 1], type = f16, converting to q5_K .. size = 112.00 MiB -> 38.50 MiB [ 284/ 292] blk.31.attn_k.weight - [ 4096, 1024, 1, 1], type = f16, converting to q5_K .. size = 8.00 MiB -> 2.75 MiB [ 285/ 292] blk.31.attn_output.weight - [ 4096, 4096, 1, 1], type = f16, converting to q5_K .. size = 32.00 MiB -> 11.00 MiB [ 286/ 292] blk.31.attn_q.weight - [ 4096, 4096, 1, 1], type = f16, converting to q5_K .. size = 32.00 MiB -> 11.00 MiB [ 287/ 292] blk.31.attn_v.weight - [ 4096, 1024, 1, 1], type = f16, converting to q6_K .. size = 8.00 MiB -> 3.28 MiB [ 288/ 292] output.weight - [ 4096, 128256, 1, 1], type = f16, converting to q6_K .. size = 1002.00 MiB -> 410.98 MiB [ 289/ 292] blk.31.attn_norm.weight - [ 4096, 1, 1, 1], type = f32, size = 0.016 MB [ 290/ 292] blk.31.ffn_down.weight - [14336, 4096, 1, 1], type = f16, converting to q6_K .. size = 112.00 MiB -> 45.94 MiB [ 291/ 292] blk.31.ffn_norm.weight - [ 4096, 1, 1, 1], type = f32, size = 0.016 MB [ 292/ 292] output_norm.weight - [ 4096, 1, 1, 1], type = f32, size = 0.016 MB llama_model_quantize_internal: model size = 15317.02 MB llama_model_quantize_internal: quant size = 5459.93 MB main: quantize time = 147401.53 ms main: total time = 147401.53 ms Unsloth: Conversion completed! Output location: ./Asuncom/Llama-3.1-8B-bnb-4bit-shiji/unsloth.Q5_K_M.gguf Unsloth: Uploading GGUF to Huggingface Hub... unsloth.F16.gguf: 100%|██████████| 16.1G/16.1G [26:20<00:00, 10.2MB/s] Saved GGUF to https://huggingface.co/Asuncom/Llama-3.1-8B-bnb-4bit-shiji Unsloth: Uploading GGUF to Huggingface Hub... unsloth.Q4_K_M.gguf: 100%|██████████| 4.92G/4.92G [08:05<00:00, 10.1MB/s] Saved GGUF to https://huggingface.co/Asuncom/Llama-3.1-8B-bnb-4bit-shiji Unsloth: Uploading GGUF to Huggingface Hub... unsloth.Q8_0.gguf: 100%|██████████| 8.54G/8.54G [13:48<00:00, 10.3MB/s] Saved GGUF to https://huggingface.co/Asuncom/Llama-3.1-8B-bnb-4bit-shiji Unsloth: Uploading GGUF to Huggingface Hub... unsloth.Q5_K_M.gguf: 100%|██████████| 5.73G/5.73G [09:24<00:00, 10.2MB/s] Saved GGUF to https://huggingface.co/Asuncom/Llama-3.1-8B-bnb-4bit-shipython ```