benhaotang/mistral-small-physics-finetuned-bnb-4bit

Mistral Physics Fine-tuned Model

This model is a fine-tuned version of mistralai/Mistral-Small-Instruct-2409 via Unsloth on kejian/arxiv-physics-debug-v0. Mostly for concept proofing, don't trust it for real physics (I mean, even Claude 3.5 can be wrong on graduate physics plenty of times, let alone a 22B model, but this hould perform a lot better than benhaotang/llama3.2-1B-physics-finetuned)!

Sorry for not having F16 version, there is no way to fit everything into VRAM or RAM at the same time in my current configuration.

Model description

• Base model: mistralai/Mistral-Small-Instruct-2409
• Training data: kejian/arxiv-physics-debug-v0
• Fine-tuning type: LoRA
• Use case: Physics domain questions
• Training Arguments:
  • Learning Rate: 2e-5
  • Epochs: 3
  • Gradient Accumulation Steps: 8
• Training setup: Colab with Nvidia L4

Usage

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch
bnb_config = BitsAndBytesConfig(
    load_in_8bit=False,
    llm_int8_enable_fp32_cpu_offload=True
)
model = AutoModelForCausalLM.from_pretrained(
    "benhaotang/mistral-small-physics-finetuned-bnb-4bit",
    device_map="auto",
    torch_dtype=torch.float16,
    offload_folder="offload_folder",
    quantization_config=bnb_config
)
tokenizer = AutoTokenizer.from_pretrained("benhaotang/mistral-small-physics-finetuned-bnb-4bit")

# Example usage
text = "Give me a short intodcution to renormalization group(RG) flow in physcis?\n"
inputs = tokenizer(text, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_length=2048)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Example output:

The Renormalization Group (RG) flow is a fundamental concept in physics, particularly in the study of phase transitions and critical phenomena. It describes how the behavior of a system changes with scale, such as the size of the system or the length scale of fluctuations. Here's a brief introduction:

1. Scale Dependence: Many physical systems exhibit properties that depend on the scale at which they are observed. For example, a fluid might appear smooth on a large scale but exhibit turbulent behavior on a small scale.

2. RG Transformation: The RG flow involves a transformation that systematically integrates out short-distance fluctuations to obtain an effective theory that describes the system at longer distances. This is often done by progressively eliminating high-momentum modes in the system.

3. Fixed Points and Universality: The RG flow can lead to fixed points, which are scale-invariant solutions. Systems that flow to the same fixed point under RG transformation exhibit universal behavior, meaning their large-scale properties are the same regardless of the details of the system at small scales.

4. Relevant and Irrelevant Operators: In the vicinity of a fixed point, operators can be classified as relevant (grow under RG flow), irrelevant (shrink), or marginal (remain constant). Relevant operators drive the system away from the fixed point, while irrelevant ones become negligible at large scales.

5. Applications: RG flow is crucial in understanding critical phenomena, such as phase transitions in statistical mechanics, and has applications in condensed matter physics, quantum field theory, and even in areas like biology and computer science.

In essence, RG flow helps us understand how the microscopic details of a system influence its macroscopic behavior, and how universal properties emerge from complex systems.

Training

Step	Training Loss	Validation Loss
50	2.407400	1.798349
100	1.452000	1.765856
150	1.161300	1.716366
200	1.223700	1.704631
250	1.135900	1.683653
300	1.371900	1.677721
350	1.208500	1.657915
400	1.303400	1.657678
450	1.233700	1.642972
500	1.081900	1.653393
550	1.117700	1.645338
600	1.109500	1.651868
650	1.190100	1.689853
700	1.000000	1.663633
750	1.020100	1.647308
800	1.033400	1.675173
850	1.082300	1.652737
900	1.074000	1.665859
950	0.975300	1.661394
1000	0.955000	1.672116
1050	1.017000	1.656730
1100	0.941500	1.652197
1150	1.003100	1.657381
1200	0.891100	1.662021
1250	0.931000	1.662401
1300	0.932800	1.662421
1350	1.042000	1.665535