Phase-Technologies/gpt2-5min-internet-lora-adapter

GPT2-Tiny-Internet: LoRA Fine-tuned for Casual Text Generation

Model Description

This model is a GPT-2 variant fine-tuned using the LoRA (Low-Rank Adaptation) technique to generate ultra-casual, internet-style text. It's designed to mimic common online language, incorporating emojis and informal expressions, ideal for social media or chatbot applications requiring a conversational tone.

Base Model

The fine-tuning was performed on lvwerra/gpt2-imdb, a GPT-2 model that has already been pre-trained on Reddit-style text, making it an excellent starting point for generating internet-like discourse.

Fine-tuning Dataset

• Dataset: mteb/tweet_sentiment_extraction
• Size: 25,000 examples (sampled from the original dataset)

The dataset consists of tweets, which inherently provide a good source of casual and short-form text suitable for this fine-tuning objective. The texts were preprocessed by converting to lowercase and appending " 😭😂🔥

" to encourage the desired output style.

Training

This model was fine-tuned efficiently using the PEFT library with LoRA adapters.

LoRA Configuration

• r: 16 (LoRA attention dimension)
• lora_alpha: 32 (Scaling factor for LoRA)
• target_modules: ["c_attn", "c_proj"] (Modules to apply LoRA to)
• lora_dropout: 0.05 (Dropout probability for LoRA layers)
• bias: "none" (No bias in LoRA layers)
• task_type: "CAUSAL_LM" (Task type for Causal Language Modeling)

Training Arguments

The model was trained for a very short duration (approximately 5-8 minutes on a free T4 GPU) to demonstrate rapid fine-tuning.

• output_dir: ./gpt2-tiny-internet
• per_device_train_batch_size: 16
• gradient_accumulation_steps: 2
• num_train_epochs: 1 (Single epoch was sufficient due to LoRA)
• learning_rate: 2e-4
• fp16: True (Mixed-precision training for speed)
• logging_steps: 20
• save_steps: 500
• save_total_limit: 1
• warmup_steps: 30
• max_steps: 350 (Total training steps)

Inference

To use this fine-tuned LoRA adapter for text generation, you need to load the base model and then merge the LoRA weights.

 import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline
from peft import PeftModel

# Define the Hugging Face repository ID
model_repo_id = "Phase-Technologies/gpt2-5min-internet-lora-adapter"
base_model_name = "lvwerra/gpt2-imdb"

# Load the base model
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    torch_dtype=torch.float16, # Use dtype for better compatibility
    device_map="auto"
)

# Load the tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_repo_id)
tokenizer.pad_token = tokenizer.eos_token

# Load the LoRA adapter onto the base model
model = PeftModel.from_pretrained(base_model, model_repo_id)

# Optionally, merge the LoRA weights into the base model for easier deployment
# and then save it as a standalone model if desired (e.g., for GGUF conversion)
# model = model.merge_and_unload()

# Create a text generation pipeline
gen = pipeline("text-generation", model=model, tokenizer=tokenizer) # Removed device=0

# Example usage
# Note: The model is trained to append '😭😂🔥' and newlines, so expect that in output.
# You might need to adjust parameters like max_new_tokens, temperature, top_p
# and num_return_sequences to get varied and coherent outputs.

print("\n--- Generating text ---\n")
prompts = [
    "bro what the hell",
    "tell me about cats",
    "i just woke up and"
]

for prompt in prompts:
    print(f"\nPrompt: {prompt}")
    for out in gen(prompt, max_new_tokens=60, do_sample=True, temperature=0.9, top_p=0.95, num_return_sequences=1):
        # We often take only the first line because the training data has newlines
        print("\u2192", out["generated_text"].split("\n")[0].strip())